How To Test With Augmented Coding Ben Fellows

The podcast explores the evolving role of QA in software development, emphasizing the integration of AI as a tool to enhance, not replace, human testing expertise. Key topics include the importance of QA in building user trust by addressing not just functional errors but also UX/UI, workflows, and broader customer expectations. The discussion highlights challenges such as reducing production bugs, overcoming QA bottlenecks, scaling teams, and reevaluating QA as an investment rather than a cost center. Augmented coding, using AI as an assistant for tasks like test automation (tools like Playwright and Cursor), is presented as a way to amplify human capabilities while addressing misconceptions about automation alone solving quality issues. The conversation also critiques the overreliance on test automation without addressing upstream process problems, such as inadequate requirements or team collaboration.

The episode delves into practical applications of AI in QA, including generating test cases, improving automation efficiency (e.g., cutting UI automation time from hours to minutes), and shifting QA roles toward code-based engineering and strategic test design. Challenges like flaky tests caused by race conditions or test data inconsistencies are discussed, with solutions emphasizing high-leverage fixes like improving front-end code rather than low-leverage manual adjustments. The podcast underscores the need for QA professionals to develop technical skills, collaborate with developers, and adopt practices like TDD/BDD to avoid bottlenecks. It also addresses the broader implications of AI on career trajectories, urging QA and software engineers to embrace continuous learning, hybrid roles blending testing and development, and the potential for AI to democratize access to advanced coding and testing resources.

• What if you fully integrate AI into your test automation to handle low-leverage tasks?

  • Move: Use AI tools like Playwright and Cursor to automate test case generation, fix flaky locators, and generate synthetic test data via data factories.
  • Why Now?: QA bottlenecks caused by manual fixes (e.g., adjusting test locators or managing test data) eat up time that could be spent on high-leverage tasks like test strategy or architecture.
  • Expected Upside: Reduce test automation setup time from weeks to hours, catch bugs earlier, and shift focus to improving test coverage and architectural decisions (e.g., optimizing the test pyramid).

• What if you transition from manual QA to a code-based engineering role using augmented coding?

  • Move: Master augmented coding by practicing with AI assistants (e.g., Cursor) to write unit/integration tests alongside developers, and learn to analyze codebases for test coverage gaps.
  • Why Now?: QA roles are evolving toward hybrid technical and testing expertise, and AI tools like Playwright, combined with hands-on coding, allow QA engineers to expand their strategic impact.
  • Expected Upside: Position yourself as a "unicorn" role with deep technical and testing knowledge, enabling you to contribute to code reviews, test automation, and product design decisions.

• What if you implement "policy as code" for PR reviews to enforce test automation standards?

  • Move: Create scripted rules (e.g., automated checks for hardcoded locators or adherence to styling guidelines) to run before PR submissions, reducing manual QA review burdens.
  • Why Now?: Organizations misallocate QA resources by not aligning their roles with code-centric responsibilities, and AI-generated code reviews risk being overwhelmed by volume and inconsistencies.
  • Expected Upside: Enforce deterministic testing standards, reduce flaky tests caused by poor code quality, and free up QA time for exploratory testing and test strategy planning.

• Leverage AI-assisted test automation tools (e.g., Playwright, Cursor) to accelerate test creation and reduce manual effort, ensuring comprehensive test coverage (e.g., generating 400 API tests in an afternoon) while focusing on high-leverage tasks like test strategy and architecture.
• Prioritize test strategy and architecture improvements by repurposing time saved from AI automation to optimize test types, analyze the test pyramid, and align automated testing with development workflows (e.g., adding unit/integration tests directly into code).
• Implement "policy as code" checks for test code in PR reviews (e.g., automating rules for hardcoded locators or styling guidelines) to enforce standards and reduce manual review burdens, ensuring test intent (e.g., assertions) is prioritized over syntax.
• Use AI-powered data factories to generate synthetic test data from schemas (e.g., using tools like Faker), streamlining test suite setup and reducing reliance on manual API/UI interactions for data preparation.
• Address QA bottlenecks by focusing on high-leverage fixes (e.g., resolving front-end race conditions, ensuring API data readiness) instead of low-leverage tasks (e.g., fixing flaky locators), and ensure access to codebases for QA to implement AI-assisted test IDs or accessibility attributes.