![[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka thumbnail](https://substackcdn.com/feed/podcast/1084089/post/189277598/ca7468da5614a246d2906ee8926f6de7.jpg)
Published 26 Feb 2026
Duration: 3137
A podcast discusses the application of distillation in machine learning, particularly in training smaller language models, and raises concerns about the potential for malicious use and the need for improved detection and regulation.
Swyx joined SAIL! Thank you SAIL Media, Prof. Tom Yeh, 8Lee, Hamid Bagheri, c9n, and many others for tuning into SAIL Live #6 with Nathan Lambert and...
The podcast covers the concept of distillation in machine learning, focusing on its application in training smaller language models using the outputs of larger ones. This process can involve using logits from traditional models or synthetic data generated by large language models (LLMs), enabling the creation of more efficient, deployable models. Major companies like DeepSeq and Google are highlighted as using distillation to reduce the size and computational demands of their models while maintaining performance.
The discussion also addresses ethical and legal challenges, particularly around "distributed distillation attacks," where third-party AI labs may use outputs from competing models to train their own systems. This raises concerns about AI geopolitics and the enforcement of terms of service. The podcast explores detection methods for such attacks, such as identifying unusual usage patterns, but notes the difficulty in distinguishing between legitimate training practices and malicious intent. Additionally, the episode touches on the broader implications of distillation for AI innovation and regulation, while examining the role of benchmarks like Sweet Bench in evaluating model performance, despite their known limitations in accurately assessing LLM capabilities across varied domains.
30 Mar 2026 Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4 w/ Pavan Kumar Reddy & Guillaume Lample
Mistral's Voxtral TTS is a 3B-parameter text-to-speech model leveraging neural audio codecs, semantic/acoustic token splitting, and efficient flow matching for multilingual real-time applications, balancing quality and cost while exploring future refinements in architecture, tokenization, and domain-specific training.
24 Mar 2026 Why There Is No "AlphaFold for Materials" AI for Materials Discovery with Heather Kulik
AI's role in education and materials science is debated, highlighting its potential to accelerate research in chemistry and physics through machine learning, yet acknowledging limitations in complex molecular design, data quality challenges, and the need for interdisciplinary collaboration and robust validation.
20 Mar 2026 Dreamer: the Personal Agent OS David Singleton
Dreamer is an AI platform democratizing access to agentic tools for non-technical users via customizable AI assistants, community-built apps, cross-device integration, and privacy-focused features, with a beta emphasis on accessibility, real-world productivity use cases, and third-party developer opportunities.
17 Mar 2026 Why Anthropic Thinks AI Should Have Its Own Computer Felix Rieseberg of Claude Cowork & Claude Code Desktop
Cowork is a cloud-native tool simplifying Cloud Code access for non-technical users through a balance of simplicity and flexibility, integrating with cloud services and Chrome while supporting diverse workflows via a lightweight virtual machine, developed in 10 days with a focus on modular components and ethical trade-offs in AI infrastructure.
12 Mar 2026 Retrieval After RAG: Hybrid Search, Agents, and Database Design Simon Hrup Eskildsen of Turbopuffer
TurboPuffer is a next-gen database platform focusing on AI-powered vector search, full-text search, and scalability, with a strong emphasis on hiring top talent and customer-driven innovation.