Building Durable AI Agents

The podcast explores the evolving landscape of AI agents, emphasizing their transition from local tools to enterprise-grade systems capable of handling complex workflows. Key topics include the challenges of ensuring durability in agents, particularly in cloud environments, where issues like state management, retries, and failure recovery complicate reliability. The discussion highlights the need for reimagining traditional ML pipeline approachesstructured as deterministic DAGswith dynamic, graph-based agent systems that execute decision-making without explicit DAG visualization. ZenMLs new project, Kitoru, is presented as a solution to these challenges, focusing on runtime resilience through state checkpointing, replay capabilities, and flexible execution paths. The podcast also addresses the shift from ML Ops to agent-based systems, where principles of safety and retryability from ML Ops are being re-applied to manage non-deterministic agent workflows.

Infrastructure considerations, such as the "harness" conceptseparating model execution from tool integrationare critical for enabling agents to interact with external systems. The dialogue contrasts proprietary and open-standard harnesses, noting tensions between model-specific integrations for performance and open frameworks that reduce dependency on particular models. Enterprise adoption of agents is framed as a growing trend, requiring scalable architectures, domain-specific infrastructure, and robust platforms to manage multi-agent fleets. Challenges include handling distributed execution, ensuring idempotency in task queues, and mitigating risks in live updates without disrupting workflows. The conversation also touches on broader industry shifts, such as the commoditization of AI models and the rise of open-source tools like Keteru, which aim to support modular, observable agent systems. Finally, the discussion emphasizes the need for holistic system design, iterative optimization, and the development of companion agents to automate troubleshooting and experimentation in complex environments.

• What if you redefined agent durability using state checkpointing?

  • Move: Implement a state checkpointing system that captures agent execution states at critical intervals (e.g., during Kubernetes pod transitions) and stores them in blob storage.
  • Why Now?: Cloud-based agents are increasingly fragile due to unoptimized architectures and distributed failures, making durability a bottleneck for production readiness.
  • Expected Upside: Enables replay, debugging, and recovery of failed agent runs, reducing downtime and improving trust in agent-driven workflows.

• What if you decoupled agent execution from API calls via message queues?

  • Move: Introduce a message broker (e.g., RabbitMQ, Kafka) to handle task distribution between APIs and worker agents, ensuring idempotent task execution and worker resilience.
  • Why Now?: Direct API-to-worker task execution risks failure due to worker unavailability, a common issue in scaling agent fleets for enterprise use cases.
  • Expected Upside: Achieve scalable, fault-tolerant agent orchestration by isolating stateless APIs from stateful workers, aligning with modern microservices patterns.

• What if you built an open-ended workflow orchestration framework for agents?

  • Move: Leverage existing open-source tools like Enamel (used in Kitoru) to define dynamic, retryable agent workflows that adapt to conditional branching and external tool interactions.
  • Why Now?: The commoditization of AI models shifts value to infrastructure, and open frameworks allow solo developers to avoid vendor lock-in while enabling complex agent behaviors.
  • Expected Upside: Create modular, reusable agent pipelines that can evolve with changing requirements, reducing reliance on proprietary solutions and accelerating deployment.

• Implement durable agent workflows with checkpointing and replay capabilities
  Use tools like Kiteru (built on Enamel) to capture state checkpoints during agent execution, store them in external databases, and enable replaying traces for troubleshooting, model-swapping, or experimentation.

• Decouple model harnesses from execution environments
  Design agent systems using a "harness" that maps model outputs (e.g., token sequences) to executable code, ensuring flexibility and reliability. Explore open frameworks like Lengraf or Pydantic AI to avoid model-specific lock-in.

• Adopt cloud-native infrastructure for scalable agent deployments
  Leverage orchestration tools (e.g., Kubernetes, AWS ECS) and task queues (e.g., message brokers) to manage distributed agent workloads, ensuring fault tolerance, idempotency, and efficient resource utilization.

• Invest in internal agent platforms for complex workflows
  For enterprise-scale operations, build or adopt domain-specific agent platforms (similar to MLOps) that integrate observability, state management, and tools like Gitaroo to analyze execution traces and prioritize optimization efforts.

• Prioritize observability and experiment-driven agent refinement
  Integrate checkpointing, logging, and debugging tools to monitor agent behavior in production. Use companion agents or automation scripts to identify issues, run experiments, and iterate on performance without disrupting workflows.