Strands Agents with Clare Liguori

The podcast discusses the development and deployment of AI agents, focusing on the Strands Agents SDK, an open-source framework initially developed at AWS to build reliable, production-ready agents. A core theme is the shift from traditional workflow-based systems to a model-driven approach, which reduces complexity by relying on the inherent reasoning capabilities of large language models (LLMs). This approach simplifies development by limiting design choices to system prompts, tools, and model selection, enabling faster deployment and greater flexibility in handling unpredictable user inputs.

Key challenges in agent development are explored, including ensuring responsible AI behavior, managing determinism, and evaluating performance in production. The discussion highlights techniques like steering hooks - runtime interventions that enforce guardrails without rigid workflows - and the use of smaller LLMs as judges to evaluate agent outputs efficiently. Observability, continuous evaluation, and adaptability are emphasized, with OpenTelemetry traces and feedback loops playing a crucial role in monitoring and improving agent performance over time. The framework supports deployment across multiple environments and model providers, allowing teams to iterate quickly and scale effectively.

• What if you validated agent behavior before production using automated LLM judges?

  • Move: Implement the Strands Evals Kit to test your agent's outputs against a rubric using a lightweight LLM (e.g., GPT-OSS 20B) as a judge, focusing on guardrail compliance (e.g., correct tool call order, staying on script).
  • Why Now?: Evaluation datasets decay quickly as user behavior evolves; catching deviations early with local evals prevents costly rework post-launch.
  • Expected Upside: Reduce off-script behavior by up to 100% (as seen with steering hooks), gain confidence in production readiness, and shorten feedback cycles from weeks to hours.

• What if you replaced rigid workflows with a model-driven agent using steering hooks for critical decisions?

  • Move: Refactor a high-maintenance workflow-based agent into a model-driven one using Strands, inserting steering hooks (e.g., Python functions) to enforce rules like "verify income before mortgage approval" at runtime.
  • Why Now?: Traditional workflows create technical debt when inputs evolve; model-driven design with hooks offers flexibility without sacrificing control.
  • Expected Upside: Cut development time from months to days, reduce brittleness with unexpected queries, and maintain 100% accuracy on critical logic paths.

• What if you continuously evolved your agent's test suite based on real user traces from OpenTelemetry?

  • Move: Pipe live agent traces (via OpenTelemetry) into a feedback loop that identifies failed tool sequences or misunderstood queries, then automatically adds them to your eval dataset in Strands.
  • Why Now?: Pre-launch test sets miss >50% of real-world edge cases (e.g., "How do I connect EC2 to S3?" vs. "What is S3?"); observability data is the richest source of truth.
  • Expected Upside: Improve real-world accuracy over time, detect degradation early, and align agent behavior with actual user needs - turning production traffic into fuel for iteration.

• Use a model-driven architecture instead of rigid workflows to reduce development time and improve adaptability to unpredictable inputs.
• Implement steering hooks to inject runtime determinism (e.g., validate tool call order) without over-engineering complex workflow logic.
• Adopt OpenTelemetry for tracing agent tool calls and responses to enable effective debugging and performance monitoring in production.
• Evaluate agent behavior continuously using real-world user traffic and an LLM judge (with a different model than the agent) to detect deviations from guardrails.
• Design agents with modular, independently tested tools and leverage smaller, cost-effective models with just-in-time guidance instead of defaulting to expensive frontier models.