What Engineering Organizations Get Wrong About AI Adoption

The podcast discusses the evolving role of AI in software engineering, emphasizing the need for engineering leaders to measure the return on investment (ROI) of AI spending through meaningful metrics like AI cost per merged pull request, rather than relying on vague or "vibes-based" assessments. It highlights that successful AI adoption goes beyond simply distributing tools - it requires rethinking workflows, maintaining up-to-date configurations, and fostering a culture of continuous learning and experimentation. Companies are shifting from early enthusiasm focused on frontier models toward more strategic, cost-effective use of diverse AI models, with an emphasis on model flexibility, task routing, and long-term efficiency.

AI is transforming engineering workflows by introducing autonomous agents as active participants in coding, code review, and deployment, leading to potential shifts in practices such as aggregating multiple pull requests or moving toward version-level reviews. While AI may handle much of the code production, human engineers remain responsible for outcomes like reliability, maintainability, and problem-solving. The discussion stresses balanced adoption - avoiding overuse of AI where simpler tools suffice - and advocates for empathy in AI tool design to support engineers at all stages of maturity. Personal experimentation with AI, even in non-work contexts like family scheduling, is encouraged as a way to build transferable skills and ownership in an era of rapid change.

• What if you measured your AI spend per shipped feature to optimize ROI?

  • Move: Track all AI usage (API calls, tool subscriptions) over the next 30 days and divide total cost by the number of merged PRs or shipped features. Use this metric weekly to identify inefficiencies.
  • Why Now?: Engineering leaders are being asked to justify AI spending; early solo operators who adopt metrics now will gain credibility and avoid top-down budget cuts later.
  • Expected Upside: You'll uncover whether AI is truly accelerating output or just inflating costs - enabling smarter model selection and spending decisions that improve profitability.

• What if you treated your AI tools like physical equipment requiring routine maintenance?

  • Move: Schedule a bi-weekly "AI tool audit": update agent configurations, refresh context files (e.g., agents.md), verify access permissions, and test against latest model versions.
  • Why Now?: Stale AI setups lead to poor performance and resistance - just like outdated dev environments. Proactive upkeep ensures reliability and compound gains from automation.
  • Expected Upside: Consistent tool performance increases trust in AI outputs, reduces debugging time, and creates a personal workflow advantage that scales with complexity.

• What if you built a personal AI agent to automate a recurring life task as practice for professional use?

  • Move: Pick one repetitive non-work task (e.g., scheduling family events, tracking expenses) and build an automated solution using AI + scripting (e.g., Google Apps Script + LLM API).
  • Why Now?: Real-world AI skills are best developed through hands-on experimentation, especially when workplace constraints limit access to models or data.
  • Expected Upside: You'll develop transferable skills in prompt engineering, workflow design, and agent ownership - making you faster and more effective when applying AI to customer or product problems.

• Track your monthly AI spending and divide it by the number of merged pull requests to calculate AI spend per PR, establishing a baseline metric for efficiency.
• Schedule weekly or biweekly personal demo sessions to showcase small AI experiments - applied to real coding tasks or automation scripts - to reinforce learning and identify practical use cases.
• Regularly audit and update your AI tool configurations (e.g., agent settings, context files like agents.md) at least once per sprint to ensure alignment with current models, access rights, and workflows.
• Experiment with cost-effective, open-weight models instead of defaulting to expensive frontier models; implement a simple routing rule (e.g., use cheaper models for documentation or testing tasks).
• Apply AI to personal automation projects (e.g., scheduling, task assignment via scripts) to build prompt engineering and system design skills in a low-risk environment, then adapt successful patterns to professional work.