More Practical AI episodes

AI incidents, audits, and the limits of benchmarks thumbnail

AI incidents, audits, and the limits of benchmarks

Published 13 Feb 2026

Duration: 2572

Experts highlight the need for robust AI safety measures, including developing methods to catalog and prevent AI incidents, and using data and third-party audits to identify and address flaws.

Episode Description

AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean...

Overview

The podcast emphasizes the critical need for AI safety, focusing on the difficulties in defining and recording AI-related incidents. It highlights the AI Incident Database, which compiles over 5,000 annotated reports of AI failures to prevent the recurrence of similar issues, inspired by safety practices in other industries. The discussion addresses the shortcomings of current benchmarking methods, the benefits of third-party audits, and risks that arise from improper AI system configurations.

The content also underscores the importance of distinguishing between intentional and unintentional failures in AI systems, and the role of statistical validation in detecting broader systemic weaknesses. It calls for the development of standardized reporting tools and procedures to enhance AI safety. Additionally, insights from the Generative Red Team Challenge at DEF CON are mentioned, where structured testing by hackers exposed significant security flaws in model design and integration processes.

Recent Episodes of Practical AI

29 Oct 2026 Abundant Intelligence at the Midwest AI Summit

"Examines practical AI applications, challenges, and best practices across industries, emphasizing real-time data, guardrails, human oversight, and iterative design for effective, user-aligned systems."

8 Oct 2026 Narrative intelligence and the human advantage

"Explores AI's role in storytelling, career adaptability, and personal growth, emphasizing narrative intelligence, overcoming imposter syndrome, and leveraging AI for creativity and resilience."

1 Oct 2026 Open models and the future of Physical AI with NVIDIA

"Open models in AI drive innovation, customization, and offline computing, with NVIDIA championing their use in hardware, robotics, and multi-agent systems while promoting open-source collaboration for broader accessibility."

24 Sept 2026 From AGENTS.md to Enterprise Deployment

"Explores AI deployment in regulated enterprise environments, emphasizing scalable, secure, and compliant solutions, governance evolution, agent vs. traditional software differences, latency challenges, security risks like agent swarms, and future AI advancements."

17 Sept 2026 How to get discovered in AI search

"Explores AI-driven search's impact on digital marketing, shifting from SEO to strategies like AEO and GEO, and challenges in adapting to AI's unique retrieval methods, brand visibility, and evolving content needs."

More Practical AI episodes