More MLOps.community episodes

Fixing GPU Starvation in Large-Scale Distributed Training thumbnail

Fixing GPU Starvation in Large-Scale Distributed Training

Published 3 Apr 2026

Duration: 00:52:48

Optimizing ML workflows requires addressing data bottlenecks through caching, efficient structuring, and hardware-aware strategies to reduce remote data calls, minimize GPU-CPU overhead, and prioritize infrastructure over model tuning, while managing trade-offs between training efficiency and serving latency.

Episode Description

Kashish Mittal is a Staff Software Engineer at Uber, working on large-scale distributed systems and core backend infrastructure.Fixing GPU Starvation...

Overview

The podcast discusses critical challenges in data handling and infrastructure efficiency within machine learning (ML) workflows. Key focus areas include optimizing data caching by storing datasets locally on GPUs/CPU during the first epoch to avoid redundant remote calls, which reduces latency and resource overhead. It highlights inefficiencies in reading Parquet files, such as filtering data after full reads, as a major bottleneck in ML pipelines. Emphasis is placed on maximizing GPU utilization (>80%) due to their cost and scarcity, stressing the importance of data pipeline design over model architecture for scalability. The discussion extends to software engineering shifts driven by hardware advancements, where infrastructure constraintslike data pipelinesoften hinder progress more than model optimizations. Balancing GPU efficiency with rapid iteration is framed as essential to avoid resource waste that slows development.

Industry-wide challenges include suboptimal data practices, such as inefficient GPU data transfers, and the risk of future economic trade-offs as cloud compute becomes more expensive. Case studies, such as Googles YouTube models underutilizing A100 GPUs, underscore the need for data restructuring (e.g., batch processing, RAM-based loading) and hardware-aware optimization strategies like flattening tensors. Misconceptions about low performance being attributed to models rather than infrastructure are addressed, with universal bottlenecks identified across CPUs, GPUs, and TPUs. Practical solutions, such as caching data in NumPy format to bypass translation overhead and using per-worker queues for deterministic training, are highlighted. The conversation also touches on broader trade-offs between training and serving, the role of hybrid CPU/GPU approaches, and the importance of reproducibility in parallel systems.

The podcast concludes with insights into emerging trends, such as the evolving use of AI agents for coding and workflow automation, and the necessity of clear documentation and critical thinking frameworks to improve human-AI collaboration. Challenges in debugging, documentation parsing, and the balance between speed and accuracy in AI responses are acknowledged. Overall, the discussion emphasizes that addressing data bottlenecks, aligning infrastructure with hardware capabilities, and fostering efficient practices are pivotal to advancing ML performance and scalability.

Recent Episodes of MLOps.community

9 Nov 2026 My AI Agent Bought Me A Car

Hermes agent uses reinforcement learning and trace-to-environment synthesis to enable continual self-improvement, balancing token efficiency with inference-time steering for better agentic performance.

2 Nov 2026 How Stanford Teaches AI Coding Now

AI agent development, verification loops, and go-to-market automation are reshaping software engineering and sales strategies.

29 Oct 2026 From Slop Code to Software Factories

"Tandem biking in Amsterdam's chaotic cycling culture led to a critique of AI-assisted coding's flaws, particularly its tendency to produce unmaintainable, poorly structured code, raising concerns about long-term software reliability."

22 Oct 2026 Token Maxxing Is Over and Here Is What Comes Next

"AI in software development offers benefits but faces challenges like nonsensical outputs and over-reliance; smaller, controlled models like Sonnet improve adoption, while measuring ROI and human-AI collaboration are key to success."

More MLOps.community episodes