Why AI Infrastructure must evolve for Agent Experience Akshat Bubna, Modal CTO

Modo was founded to address limitations in Kubernetes for bursty, compute-heavy workloads, with a focus on building a better runtime for use cases like ETL, job queues, and machine learning inference. Over time, the platform expanded from data pipelines to support GPU-intensive tasks such as classical ML, computer vision, and large model inference, even adding GPU capabilities before the surge in generative AI. A core emphasis has been improving developer experience by minimizing YAML configuration through code co-location and decorators, which has evolved into optimizing for agent experience (AX) as AI agents become more prevalent. This shift includes enabling self-revisioning runtimes that allow dynamic updates without manual infrastructure management.

The platform supports elastic scaling for highly variable workloads, such as reinforcement learning, batch processing, and language model inference, with capabilities to rapidly scale to thousands of GPUs across multiple regions. Innovations like GPU snapshotting and speculative decoding - where a smaller draft model predicts tokens for faster generation - have enhanced performance and cold-start resilience. The system differentiates itself through software optimization rather than hardware ownership, operating across 17 cloud providers with a reliability layer that insulates users from infrastructure failures. It also provides advanced networking features like IPv6 overlays and RDMA support, enabling efficient distributed training and multi-node sandbox communication.

Modo enables a wide range of AI and non-AI workloads, including robotics, computational biology, and video generation, by offering flexible primitives such as persistent storage, custom model deployment, and secure, networked sandboxes. The platform emphasizes co-location of compute resources, fine-grained networking controls, and support for specialized hardware to meet diverse application needs. As agent-based systems grow, the infrastructure is adapting to support production-grade requirements like file persistence, permission models, and secure isolation. The rise of coding agents has also sparked interest in CI/CD optimization, where snapshot-and-restore capabilities can drastically reduce build times. Overall, the platform continues to evolve based on user-driven innovation and emerging patterns in AI workload management.

• What if you redesigned your API-first product to prioritize agent experience over developer experience?

  • Move: Replace YAML-based configuration with code-decorated infrastructure (e.g., @gpu, @autoscale) and embed observability into CLI tools.
  • Why Now?: AI agents are consuming services directly - poor runtime UX creates friction in autonomous workflows, and early adopters are already hacking around config overhead.
  • Expected Upside: Increase adoption by agent-native startups who value self-revising code and dynamic scaling, reducing integration time from days to minutes.

• What if you built a speculative decoding layer for your customers' LLM inference workloads using open-source primitives like DFlash?

  • Move: Implement a block-based speculator that runs a smaller draft model to pre-generate tokens, then verify in bulk on the target model - integrate as a drop-in endpoint feature.
  • Why Now?: Memory bandwidth limits are capping inference speed, and kernel-level optimizations alone yield diminishing returns - speculative decoding offers 2x - 4x speedups today.
  • Expected Upside: Deliver best-in-class latency at lower cost, turning your inference service into a performance anchor for production-grade agent pipelines.

• What if you repurposed your bursty GPU infrastructure to serve CI/CD workloads for AI coding agents?

  • Move: Offer snapshot-enabled, GPU-accelerated CI sandboxes with persistent dependency caches and fast cold starts for agent-driven code generation and testing.
  • Why Now?: Coding agents waste >60% of CI time on re-preparing environments - your runtime already supports fast scaling and state preservation via snapshotting.
  • Expected Upside: Capture emerging demand from AI-native dev teams; differentiate from traditional CI platforms by optimizing for agent efficiency, not just human developers.

• Design your software to co-locate infrastructure configuration with code using decorators or similar patterns to reduce YAML overhead and improve maintainability.
• Prioritize elastic scaling and bursty workload support in your platform, especially for GPU-heavy tasks, to serve real-world AI/ML and inference use cases.
• Build observability directly into your CLI and dashboards so both developers and agents can investigate system behavior without leaving the workflow.
• Offer open and transparent endpoints that allow users to deploy models without code changes - but also provide an "eject" option to access underlying code for full control.
• Develop and open-source performance-enhancing tools (e.g., speculative decoding libraries like DFlash) to build credibility and demonstrate technical leadership in model optimization.