Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models, and Building the Inference Cloud

The podcast discusses the rapid scaling of Base 10, which has achieved 30x growth in 24 months with projected annual revenue exceeding $1 billion, driven by rising demand for AI inference across industries. It emphasizes the complexities of AI inference as a lagging market compared to training, noting advancements in open-source models and post-training techniques like reinforcement learning, which enable companies to customize and own their inference processes. The distinction between application-specific AI (e.g., healthcare workflows, customer support systems) and generic "frontier" models is highlighted, with examples of startups like Bridge and Abridge demonstrating the need for domain-specific solutions. The market is dominated by a "long tail" of models and enterprises, but AI-native startups are growing rapidly, pushing infrastructure providers to balance serving both startups and enterprise-scale needs.

Key challenges include compute constraints, supplier difficulties, and the need for strategic infrastructure investments, such as deploying 90 global compute clusters and developing a unified runtime fabric. The discussion underscores the importance of inference and post-training capabilities as core competencies, with post-training optimization and feedback loops critical for refining models. Geopolitical concerns around open-source models, particularly Chinese models, are acknowledged, but the focus remains on leveraging cost-effective options like DeepSeek while prioritizing model performance. The podcast also touches on the evolving role of compute as a strategic asset, the diversification of chip technology, and the need for runtime innovations to address scaling, security, and performance bottlenecks in large language models. Future trends point to a shift toward AI as a service, with AI-driven "units of cognition" reshaping industries and consumer expectations.