AI Performance Testing: How to Scale Agentic AI with Kandasamy Selvaraj

The podcast discusses the challenges of scaling AI applications, particularly agent-based systems, from demo to enterprise-level production. A key focus is on adapting performance engineering practices to handle the non-deterministic nature of AI, where traditional deterministic models no longer apply. This includes shifting from span-level to outcome-based Service Level Objectives (SLOs), implementing performance testing early in development (shift-left) and in production-like environments (shift-right), and using techniques like synthetic monitoring and stubbing external dependencies to ensure reliability and scalability. Observability tools such as OpenTelemetry and Langfuse are emphasized for tracking token usage, reasoning loops, tool calls, and context bloat that can degrade performance.

Further topics include cost optimization strategies such as prompt caching, model routing based on query complexity, and summarizing inputs to reduce token consumption - critical for managing expenses at scale. The discussion highlights the importance of designing guardrails for agent behavior, including limits on tool calls, reasoning steps, and token usage per trace, to maintain system stability. Developer involvement in performance testing through CI/CD pipelines, pluggable testing frameworks, and real-time feedback mechanisms ensures performance is built into the development lifecycle. The conversation underscores that effective AI performance engineering requires a holistic approach centered on user experience, cost efficiency, and system resilience, moving beyond mere functional correctness to ensure safe, scalable, and economical deployment.

• What if you implemented outcome-based performance gates in your CI/CD pipeline today?

  • Move: Integrate a lightweight performance gate using JMeter or Soystar in your dev/test environment that runs on every commit, measuring trace-level outcomes (e.g., token usage < 5K, steps 4). Use stubbed dependencies to isolate application behavior.
  • Why Now?: AI's non-deterministic nature means performance regressions can silently inflate costs and degrade UX - catching them early prevents compounding issues at scale. Shift-left testing is now feasible with pluggable frameworks and synthetic monitoring.
  • Expected Upside: Reduce AI token costs by up to 50% through early detection of inefficiencies (e.g., prompt bloat, loop overruns), while ensuring consistent user experience before production deployment.

• What if you classified and routed user queries to optimal models based on complexity?

  • Move: Build a classifier (rule-based or ML) that tags incoming queries as simple (e.g., "fetch PDF"), medium, or complex, then route to cheaper/faster models (e.g., Llama 3 8B) or high-power ones (e.g., GPT-4) accordingly. Log token savings per decision.
  • Why Now?: With millions of conversations scaling rapidly, unoptimized model use leads to avoidable 30 - 70% cost overhead. The infrastructure for dynamic routing (API gateways, observability via Langfuse/RIS) is already accessible to solo developers.
  • Expected Upside: Achieve 30%+ reduction in AI inference costs within two weeks, with measurable ROI visible in trace-level SLO dashboards - freeing budget for innovation or scaling.

• What if you enforced strict token capping and caching at the agent level from day one?

  • Move: Implement conversation summarization (reduce 50K 5K tokens) and enable prompt caching (10% cost for cached tokens) in your agent framework. Set hard limits: max 2 tool calls, 4 reasoning steps, 5K tokens per trace.
  • Why Now?: Token leakage and context bloat are silent killers in agentic workflows - early design decisions prevent exponential cost growth when scaling to enterprise loads. Tools like AWS Agent Core Memory and OpenTelemetry make this actionable now.
  • Expected Upside: Prevent retrace storms and platform saturation, cut per-conversation costs by up to 90%, and ensure platform stability even under high concurrency - making your solo-built agent production-ready at scale.

• Implement prompt caching in your AI workflows to reduce token costs by up to 90% on repeated inputs, starting with static system prompts or frequently used context blocks.
• Integrate performance gates in CI/CD pipelines using tools like JMeter or GitHub Actions to automatically block deployments that degrade response times or increase token usage beyond set baselines.
• Adopt outcome-based SLOs at the trace level, such as ensuring 95% of agent conversations complete within 5,000 tokens or 4 reasoning steps, and enforce these via automated monitoring (e.g., Langfuse).
• Apply model routing logic to classify user queries (simple/medium/complex) and route them to cheaper models when possible - e.g., use a lightweight model for document retrieval instead of a high-cost reasoning model.
• Use stubbing for external dependencies during load testing (e.g., mock third-party APIs) to isolate performance bottlenecks in your application and enable safe scalability tests at 10x production load without side effects.