More MLOps.community episodes

My AI Agent Bought Me A Car thumbnail

My AI Agent Bought Me A Car

Published 9 Nov 2026

Duration: 00:35:15

Hermes agent uses reinforcement learning and trace-to-environment synthesis to enable continual self-improvement, balancing token efficiency with inference-time steering for better agentic performance.

Episode Description

Sam Herring's first real test of Hermes Agent was a shopping tool that matched clothes to photos of ancient pottery. A few months later, the same agen...

Overview

The conversation covers Hermes, an agent that began as an internal productivity tool at Noose Research and evolved into a widely used system. Sam describes personal use cases, including tasking Hermes to find clothing matching art pieces and to monitor government auction sites daily, which led to purchasing a three-wheel ticketing car in May. He notes the agent handles "life admin tasks that would require you to kind of be on your computer a lot" without step-by-step instructions, and that designers use it for creative pipelines producing launch videos with composed music.

The discussion shifts to RL training approaches, centering on the "trace to environment" method where agent traces are used to synthesize RL environments for continual learning. Sam explains the failure analysis pipeline: identifying exact mistake points, clustering them by error type, and building synthetic training environments from those clusters. The conversation also covers token efficiency trade-offs, reasoning length diminishing returns, and inference-time steering as complementary approaches to improving agentic model performance without full retraining.

What If

  • What if you built a nightly trace-to-environment loop for your own agent workflows?

    • Move: Record your agent's task traces, run an automated failure analysis to find exact mistake points, cluster errors by type, and generate synthetic training environments from those clusters. Propose a scheduled job that trains a small model on the synthesized environments each night.
    • Why Now? The method is described in published research (Xiaomi's MIMO v2.6 pro paper) and Sam confirms the failure analysis pipeline is "all automated". Cloud RL services make offloading training runs feasible without owning large GPU clusters.
    • Expected Upside: You could propose a self-improving agent that reduces its error rate on recurring tasks over weeks without manual retraining, as Sam describes the loop where "every task that you've had your agent do while you're sleeping it goes and creates RL environments trains itself on them and updates".
  • What if you ran a hot-swap RL cycle on a single local machine for one specific task?

    • Move: Set up a small local model as your agent, have it design an RL environment for a task you struggle with (e.g., terminal management or a specific API), kick off training on cloud compute, download the updated weights, and hot-swap them back into the running agent.
    • Why Now? Sam describes running "loops on my own hardware" with this exact pattern, and cloud RL providers like Thinking Machines' tinker library allow per-use offloading of training loops.
    • Expected Upside: You could propose a targeted skill improvement where the agent becomes measurably better at one narrow task overnight, as Sam frames it: "it can teach itself a skill like if you task Hermes with like I want you to yes create an RL environment to learn how to use the terminal better".
  • What if you applied inference-time activation steering to a smaller on-device model instead of full RL training?

    • Move: Identify a behavioral direction you want (e.g., more pragmatic math solving or more careful documentation lookup), create a steering vector that multiplies into the model's final layer at sampling time, and serve it as a lightweight sidecar alongside your inference engine.
    • Why Now? Sam calls this "a really underexplored area of improving small models" and notes it is "very useful" especially when the model is on-device. It avoids the compute cost of full RL runs.
    • Expected Upside: You could propose a lower-cost path to improving agent behavior on constrained hardware, where the same base model produces different task-appropriate outputs depending on which steering vector is active, as Sam analogizes: "you insert the key for one behavior and all of a sudden now you've" unlocked a different performance profile.

Takeaway

  • Task your agent with a daily monitoring script for a specific recurring search (government auctions, job listings, price drops) and act on results when they appear, rather than checking manually each day.
  • Build a failure analysis pipeline for your agent: log every task trace, flag the exact step where the agent erred, cluster errors by type, and use those clusters to design targeted synthetic training environments.
  • Let your agent write and refine its own skills after repeated execution of the same task, rather than manually codifying each workflow; convergence to a stable skill happens automatically with repeated use.
  • Experiment with lower reasoning settings for routine or well-understood tasks to cut token spend, reserving high or extra-high reasoning only for genuinely novel or complex problems.
  • Explore inference-time steering vectors as a lightweight complement to training: create a small activation vector that nudges your model toward a desired behavior (pragmatism, caution, documentation lookup) at sampling time without retraining.

Recent Episodes of MLOps.community

2 Nov 2026 How Stanford Teaches AI Coding Now

AI agent development, verification loops, and go-to-market automation are reshaping software engineering and sales strategies.

29 Oct 2026 From Slop Code to Software Factories

"Tandem biking in Amsterdam's chaotic cycling culture led to a critique of AI-assisted coding's flaws, particularly its tendency to produce unmaintainable, poorly structured code, raising concerns about long-term software reliability."

22 Oct 2026 Token Maxxing Is Over and Here Is What Comes Next

"AI in software development offers benefits but faces challenges like nonsensical outputs and over-reliance; smaller, controlled models like Sonnet improve adoption, while measuring ROI and human-AI collaboration are key to success."

15 Oct 2026 Agents Acting Safely on Our Behalf

"Web infrastructure must evolve to support AI agents securely, with protocols like WebMCP enabling safe, deterministic actions while accommodating diverse interaction methods and user control."

More MLOps.community episodes