How to Move from Prompt Engineering to Harness Engineering in Testing with Matt Wynne

The discussion explores the evolving role of AI and large language models (LLMs) in software development, emphasizing a shift from manual coding to engineering systems that generate code automatically. Concepts such as the "software factory" or "dark factory" are examined, where humans focus on designing and guiding automated systems rather than writing or reading code directly. This transformation is supported by AI's ability to modernize legacy systems - such as decades-old COBOL applications - by understanding, refactoring, and generating equivalent code in modern languages, with automated validation ensuring correctness, security, and maintainability.

A key theme is the redefinition of development practices like Test-Driven Development (TDD) and Behavior-Driven Development (BDD), where structured natural language (e.g., Given-When-Then) plays a critical role in aligning AI with intended behavior. The conversation highlights the importance of trust in AI-generated outputs, advocating for systems that validate code through automated checks, cross-model review, and guardrails rather than manual inspection. As AI takes over routine tasks, roles are shifting toward setting constraints, improving communication, and curating design heuristics extracted from team feedback. The focus moves from individual productivity to system-level steering, where developers and testers evolve into "generic thinking engineers" who guide AI agents through structured workflows, context engineering, and continuous improvement loops.

• What if you automated validation of AI-generated code using cross-model reviews?

  • Move: Set up a script that routes AI-generated code through three different LLMs (e.g., GPT-4, Claude 3, Gemini) with prompts asking each to identify architectural flaws, security risks, and deviations from team heuristics.
  • Why Now?: As solo developers increasingly rely on AI for coding, manual validation doesn't scale; multi-model review reduces blind spots and aligns with harness engineering principles.
  • Expected Upside: Catch 40 - 60% more edge cases before testing, reduce rework time by half, and create a repeatable quality gate for future AI outputs.

• What if you encoded your team's design heuristics into an AI-readable checklist for code generation?

  • Move: Mine your past 20 pull request comments for recurring feedback (e.g., "Don't mutate input," "Prefer composition"), then structure them into a Markdown-based heuristic guide used as system prompt in your AI coding assistant.
  • Why Now?: AI often replicates legacy or inconsistent patterns; explicitly defining unwritten rules ensures generated code matches your standards without manual cleanup.
  • Expected Upside: Reduce code review cycles by 70%, increase first-draft accuracy, and make your solo workflow mimic a seasoned team's quality control.

• What if you stopped writing tests and instead engineered a system that validates behavior via AI-powered BDD?

  • Move: Convert user stories into Given-When-Then templates, feed them to an LLM to generate testable acceptance criteria and corresponding test code, then run automated checks using a headless browser or API runner.
  • Why Now?: With AI excelling at natural language interpretation, BDD's structure becomes a direct input for test automation - shifting you from writing code to steering systems.
  • Expected Upside: Cut test creation time by 80%, maintain living documentation aligned with functionality, and enable non-coders (clients, yourself in six months) to verify behavior through plain-language scenarios.

• Integrate BDD's Given-When-Then structure into AI prompting workflows to clearly define expected behaviors and improve output accuracy.
• Build automated validation systems (e.g., deterministic checks, test suites) to verify AI-generated code instead of manually reviewing every line.
• Extract and document team-specific design heuristics from past code reviews to guide AI toward consistent, context-aware code generation.
• Use multiple LLMs in parallel (e.g., GPT, Claude, Gemini) to cross-review AI outputs and catch errors that single models might miss.
• Conduct regular retrospectives on AI interactions to refine prompts, improve context engineering, and treat AI like a junior developer needing coaching.