Sonnet 5 review: I ran 64 generations to find out if it's worth it

The podcast discusses the introduction of Anthropics new Claude Sonnet 5 model, emphasizing its agentic capabilities, improved task performance compared to earlier Sonnet versions, and cost advantages over Opus models. It notes Sonnet 5s slightly lower performance on specific benchmarks (e.g., 69% on Agentic Coding Sweet Bench Pro, 82% on Terminal Bench 2.1) but highlights its ability to handle longer agentic sessions at a reduced cost. The model is positioned as a cost-effective alternative for complex tasks like coding and browser interactions.

A focus is placed on the development of repeatable benchmarks called How I AI Bench, designed to assess models across use cases such as PRD writing, bug fixing, and design tasks. These benchmarks prioritize human-centric evaluation using historical data and avoid AI-as-judge methods, incorporating tasks like transforming notes into PRDs, creating prototypes, and generating cited information. The evaluation methodology combines automated scoring with manual vibe checks to balance objective metrics with subjective preferences, though discrepancies between human and AI judgments are noted.

The podcast evaluates multiple models, including Sonnet 5, Opus, Gemini 3 Pro, and GPT 5.5, across tasks like prototyping and coding. Results show Gemini 3 Pro and Sonnet 5/GPT 5.5 leading in some areas, while Sonnet 4.6 and Opus show mixed performance. Challenges include subjective biases (e.g., preference for Sonnet 4.6s voice) and inconsistent model reliability. The discussion also outlines plans to refine benchmarks, update rankings with new models, and develop a weighted index combining human and technical metrics to standardize AI evaluation. Limitations include ongoing debates about effectively measuring agentic capabilities and reconciling subjective taste with quantitative performance.

• What if you replace Opus-based agentic workflows with Claude Sonnet 5 for cost-effective task execution?

  • Move: Adopt Sonnet 5 for agentic tasks (e.g., browser automation, code-based searches) that previously required Opus, leveraging its lower cost ($2M input tokens vs. Opuss higher pricing).
  • Why Now?: Sonnet 5 achieves Opus-level performance on benchmarks like Terminal Bench 2.1 (82%) and supports longer sessions at a fraction of the cost, making it viable for solo operators with budget constraints.
  • Expected Upside: Reduce infrastructure costs by 5070% while maintaining comparable task completion rates, freeing resources for other priorities like feature development or client acquisition.

• What if you build a custom benchmark suite using Claude to evaluate LLMs for your niche software use cases?

  • Move: Develop a task-based benchmark (e.g., PRD generation, bug-hunting workflows) using structured inputs, historical data, and blind scoring via Claude or Codex.
  • Why Now?: The text highlights the limitations of vague "vibe checks" and emphasizes standardized evaluation, which is critical for comparing models like Sonnet 5 and GPT 5.5 for your specific needs.
  • Expected Upside: Identify models tailored to your business (e.g., Sonnet 5 for coding, Gemini 3 Pro for PRDs), improving decision-making and reducing reliance on subjective AI reviews.

• What if you deploy Hyper Agent to automate repetitive tasks like email management and CRM updates while testing agentic models?

  • Move: Integrate Hyper Agent to create autonomous agents for tasks like competitor monitoring or ad creation, then use Sonnet 5 to refine their logic and output consistency.
  • Why Now?: Hyper Agents ease of use and no local infrastructure requirements make it ideal for solo developers, while Sonnet 5s agentic improvements (longer sessions at lower cost) can enhance agent reliability.
  • Expected Upside: Automate 4060% of administrative work, enabling faster project delivery and allowing you to focus on high-value tasks like product innovation or client consulting.

• Adopt Sonnet 5 for cost-effective agentic tasks: Use Sonnet 5 for complex coding or browser automation workflows where performance is slightly behind Opus but at a significantly lower cost ($2 input, $10 output per million tokens). Prioritize tasks where Opus-level results are not strictly required.
• Build custom benchmarks with human-centric tools: Create repeatable, task-specific benchmarks using tools like Claude or Codex to evaluate models (e.g., PRD writing, bug fixing). Focus on consistency, time-based metrics, and relevance to your product needs.
• Leverage hybrid scoring for model evaluation: Combine structured rubric scoring with manual "vibe checks" (e.g., 15-point gut-feel scale) to balance objective task performance with subjective quality (e.g., clarity, functionality, or aesthetic). Store feedback in structured formats like JSON for analysis.
• Optimize model selection by task type: Use GPT 5.5 for PRD writing and Sonnet 6 for prototyping, as they showed stronger performance in these areas. Avoid over-relying on Opus 4.8 for lightweight tasks due to higher costs and mixed functional results.
• Implement lightweight, experience-driven scoring: Manually score prototypes or outputs using domain expertise (e.g., product design, engineering) to avoid overcomplicating evaluation. Scale this by grouping similar tasks and using historical data to refine criteria.