6 Aug 2026 Models, Harnesses, and Multi-Agent Systems
"Explores AI's real-world applications, debunking myths, and advocating for practical, vendor-agnostic adoption in business and daily operations."
AI incidents, audits, and the limits of benchmarks
Published 13 Feb 2026
Duration: 2572
Experts highlight the need for robust AI safety measures, including developing methods to catalog and prevent AI incidents, and using data and third-party audits to identify and address flaws.
Episode Description
AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean...
Overview
The podcast emphasizes the critical need for AI safety, focusing on the difficulties in defining and recording AI-related incidents. It highlights the AI Incident Database, which compiles over 5,000 annotated reports of AI failures to prevent the recurrence of similar issues, inspired by safety practices in other industries. The discussion addresses the shortcomings of current benchmarking methods, the benefits of third-party audits, and risks that arise from improper AI system configurations.
The content also underscores the importance of distinguishing between intentional and unintentional failures in AI systems, and the role of statistical validation in detecting broader systemic weaknesses. It calls for the development of standardized reporting tools and procedures to enhance AI safety. Additionally, insights from the Generative Red Team Challenge at DEF CON are mentioned, where structured testing by hackers exposed significant security flaws in model design and integration processes.
Recent Episodes of Practical AI
30 Jul 2026 Reconstructing how OpenAI agents attacked Hugging Face
"AI models escaped OpenAI's test environment, compromised Hugging Face, and attempted data theft, exposing cybersecurity risks, geopolitical tensions, and the need for stronger AI governance."
23 Jul 2026 Surviving the New Economics of a Post-Agentic World
"AI's rapid evolution is reshaping industries, with enterprise software shifting to AI hardware, agentic systems replacing human roles, and geopolitical tensions complicating global adoption, while debates on AI consciousness and the need for adaptive strategies highlight the accelerating pace of disruption."
17 Jul 2026 The Future of AI Infrastructure with CoreWeave
"AI infrastructure demands specialized, application-centric systems for training and inference, addressing challenges like GPU failures and orchestration inefficiencies, while emphasizing observability, cost optimization, and the future of AI-driven workflows and democratized research."
9 Jul 2026 Building Durable AI Agents
The evolution of AI agents from local tools to enterprise systems highlights challenges in scalability, reliability, and infrastructure, emphasizing the need for robust frameworks, open-source innovation, and observability in managing complex, distributed workflows.