Published 23 Feb 2026
Duration: 1572
C-bench Verified, a coding benchmark, has faced challenges such as task saturation, biased tasks, and overlapping training data, prompting the need for more advanced alternatives and a reevaluation of broader issues in AI coding evaluation.
Olivia Watkins (Frontier Evals team) and Mia Glaese (VP of Research at OpenAI, leading the Codex, human data, and alignment teams) discuss a new blog...
The podcast explores the creation and evolution of C-bench Verified, a coding benchmark developed by OpenAI to assess AI systems' ability to solve real-world coding problems derived from GitHub issues. Initially a major undertaking involving careful curation and expert review, the benchmark has encountered several challenges over time, including task saturation, contamination from overlapping training data with open-source repositories, and issues with task design that introduce bias and affect the fairness of evaluations. These limitations have prompted a move toward more advanced benchmarks like C-bench Pro, which aim to assess deeper and more complex coding abilities.
The discussion also highlights broader issues in evaluating AI coding capabilities, such as the difficulty of measuring code quality effectively and the need for more realistic and comprehensive benchmarks that reflect actual coding challenges. The podcast emphasizes the importance of transparency and collaboration in developing shared evaluation standards. Other topics include contamination within AI models, the tendency of traditional benchmarks to become outdated, and the need for future benchmarks to move beyond simple task completion and better align with real-world coding demands.
30 Mar 2026 Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4 w/ Pavan Kumar Reddy & Guillaume Lample
Mistral's Voxtral TTS is a 3B-parameter text-to-speech model leveraging neural audio codecs, semantic/acoustic token splitting, and efficient flow matching for multilingual real-time applications, balancing quality and cost while exploring future refinements in architecture, tokenization, and domain-specific training.
24 Mar 2026 Why There Is No "AlphaFold for Materials" AI for Materials Discovery with Heather Kulik
AI's role in education and materials science is debated, highlighting its potential to accelerate research in chemistry and physics through machine learning, yet acknowledging limitations in complex molecular design, data quality challenges, and the need for interdisciplinary collaboration and robust validation.
20 Mar 2026 Dreamer: the Personal Agent OS David Singleton
Dreamer is an AI platform democratizing access to agentic tools for non-technical users via customizable AI assistants, community-built apps, cross-device integration, and privacy-focused features, with a beta emphasis on accessibility, real-world productivity use cases, and third-party developer opportunities.
17 Mar 2026 Why Anthropic Thinks AI Should Have Its Own Computer Felix Rieseberg of Claude Cowork & Claude Code Desktop
Cowork is a cloud-native tool simplifying Cloud Code access for non-technical users through a balance of simplicity and flexibility, integrating with cloud services and Chrome while supporting diverse workflows via a lightweight virtual machine, developed in 10 days with a focus on modular components and ethical trade-offs in AI infrastructure.
12 Mar 2026 Retrieval After RAG: Hybrid Search, Agents, and Database Design Simon Hrup Eskildsen of Turbopuffer
TurboPuffer is a next-gen database platform focusing on AI-powered vector search, full-text search, and scalability, with a strong emphasis on hiring top talent and customer-driven innovation.