More MLOps.community episodes

Serving LLMs in Production: Performance, Cost & Scale // CAST AI Roundtable thumbnail

Serving LLMs in Production: Performance, Cost & Scale // CAST AI Roundtable

Published 19 Feb 2026

Duration: 01:05:55

AI model deployment requires careful planning of infrastructure and scalability to ensure smooth transition from experimental to production stages, considering factors like cost, performance, and control.

Episode Description

Roundtable CAST AI episode: Serving LLMs in Production: Performance, Cost & Scale.Join the Community:https://go.mlops.community/YTJoinInGet the newsle...

Overview

The conversation focuses on the difficulties of moving AI and machine learning models from experimental stages into production, emphasizing the importance of infrastructure planning and scalability. Teams often prioritize solving specific problems or proving concepts without considering the complexities of long-term deployment. As AI adoption expands, there's a growing need to shift from experimentation to scaling, which requires robust MLOps practices. The discussion examines different deployment models, such as APIs, managed GPU services, and self-hosting, each with varying trade-offs in cost, performance, and control. Self-hosting provides the most control and flexibility but demands extensive infrastructure setup, including Kubernetes, GPU orchestration, and auto-scaling, presenting significant complexity.

The choice of infrastructure is influenced by the type of workload, like generative, summarization, or chat-like tasks, which have distinct performance and cost requirements. The conversation highlights key performance metricssuch as time to first token, inter-token latency, and goodputas critical for optimizing model serving. Techniques like model quantization, kernel optimizations, and separating pre-fill and decode phases are discussed as ways to improve efficiency. Overall, the discussion stresses the need to align deployment strategies with specific use cases and user expectations to achieve effective and efficient AI model serving.

Recent Episodes of MLOps.community

9 Nov 2026 My AI Agent Bought Me A Car

Hermes agent uses reinforcement learning and trace-to-environment synthesis to enable continual self-improvement, balancing token efficiency with inference-time steering for better agentic performance.

2 Nov 2026 How Stanford Teaches AI Coding Now

AI agent development, verification loops, and go-to-market automation are reshaping software engineering and sales strategies.

29 Oct 2026 From Slop Code to Software Factories

"Tandem biking in Amsterdam's chaotic cycling culture led to a critique of AI-assisted coding's flaws, particularly its tendency to produce unmaintainable, poorly structured code, raising concerns about long-term software reliability."

22 Oct 2026 Token Maxxing Is Over and Here Is What Comes Next

"AI in software development offers benefits but faces challenges like nonsensical outputs and over-reliance; smaller, controlled models like Sonnet improve adoption, while measuring ROI and human-AI collaboration are key to success."

More MLOps.community episodes