Liz Fong-Jones: 2x the PRs, 1.5x the Incidents

The podcast discusses the challenges of scaling software development, particularly as the volume of pull requests increases, leading to higher incident rates despite initial improvements in change failure metrics. With human attention for code reviews being a limiting factor, the focus shifts to integrating AI tools like Autobot and Jev to automate parts of the development and review process. These tools aim to reduce review load by classifying pull requests, identifying trivial bugs, and enabling faster, cheaper feedback loops - though human oversight remains essential to maintain quality and accountability.

AI's role in software development is explored not as a standalone solution but as an amplifier of existing organizational behaviors. In high-ownership teams with strong DevOps and CI/CD practices, AI increases speed and efficiency; in dysfunctional environments, it exacerbates problems like inconsistency and "slop bombs." The discussion emphasizes that successful AI integration depends on clear code patterns, observability, testability, and developer ownership across the full lifecycle. Looking ahead, platform engineering is seen as a unifying force, centralizing disciplines like observability, security, and reliability to support developers without requiring deep expertise in each area, while open source faces new sustainability challenges due to the rise of low-quality AI-generated contributions.

• What if you automated triage of your pull requests using AI classification?

  • Move: Implement a lightweight LLM classifier (like Jev) to categorize incoming PRs into "safe to auto-merge" (e.g. docs, version bumps) vs. "requires human review". Start with 5 - 10 rule-based categories and integrate with your CI/CD pipeline.
  • Why Now?: PR volume is rising fast - even for solo devs using AI assistants - and manual review fatigue leads to oversights. Early automation prevents technical debt accumulation before it scales.
  • Expected Upside: Reduce review time by 20 - 30%, free mental bandwidth for complex logic changes, and catch risky patterns (e.g. env var edits) before they deploy.

• What if you shifted from writing tests to teaching AI your testing standards?

  • Move: Document your test quality heuristics (e.g. "no empty assertions", "must include negative case") and fine-tune an open-source model or prompt template to audit new tests in PRs. Use it as a pre-merge gate.
  • Why Now?: AI-generated code often produces "empty calorie" tests; catching them early stops degradation of trust in your suite. As you scale output via AI, validation must scale too.
  • Expected Upside: Maintain high test signal-to-noise ratio, reduce false confidence, and create reusable quality guards that work across future projects.

• What if you treated every incident as a system design flaw - not just a bug fix?

  • Move: After each production issue, run a 30-minute postmortem to update one preventive control: add a telemetry attribute, tighten a schema, or create an automated check in CI. Enforce it via tooling, not process.
  • Why Now?: Incidents compound when root causes aren't systemic fixes. Solo operators can't afford recurring fires - each one erodes velocity.
  • Expected Upside: Break the cycle of repeat outages, build self-documenting systems through observability, and increase deployment confidence even as complexity grows.

• Audit and classify your pull requests using automation to identify which can be safely auto-merged, reducing manual review load by focusing human effort on high-impact changes.
• Implement fast, hermetic CI/CD pipelines with built-in observability and testing to enable safe, rapid feedback and prevent low-quality AI-generated code from reaching production.
• Enforce strict contribution standards for AI-assisted code by requiring verification steps (e.g., quizzes or reproducers) to ensure understanding and maintain quality, especially in open-source or collaborative projects.
• Use AI to automate repetitive development tasks like adding telemetry, attributes, or test scaffolding, but retain human oversight for design decisions and system-level understanding.
• Define clear, consistent code patterns in your codebase before adopting AI tools to avoid amplification of inconsistencies and technical debt through automated pattern replication.