The AI Spend Question Nobody Can Answer

The podcast discusses the evolving landscape of software engineering in large-scale environments, focusing on challenges related to scale, tooling, and workflow efficiency. With thousands of engineers managing thousands of repositories - from legacy systems to modern microservices - organizations face bottlenecks not only in code reviews but also in security, compliance, and release pipelines. The rise of AI-driven "agentic" tooling is reshaping development, enabling engineers to manage AI agents as part of their workflows, though this introduces new complexities around cost, latency, and developer experience. Cross-disciplinary differences - such as mobile versus web development - demand tailored verification processes, and there is a growing need for infrastructure that supports secure, scalable execution of AI agents.

A major theme is the shift from traditional productivity metrics like lines of code or commit frequency toward outcome-based measurements that reflect real value delivery. Organizations are building comprehensive event-tracking systems to analyze the software development lifecycle, identifying blockers and inefficiencies through both quantitative data and qualitative feedback like developer surveys. There's a strong emphasis on balancing innovation with reliability, especially as AI adoption accelerates without clear ROI for many enterprises. The discussion also explores the democratization of coding through low-code/no-code tools, enabling non-engineers to build solutions, though this raises concerns about maintenance, governance, and technical debt. Ultimately, the focus remains on creating high-trust, data-informed cultures that prioritize developer experience, operational efficiency, and long-term system reliability.

• What if you built a personal verification agent stack to automate code, security, and compliance checks before every commit?

  • Move: Set up a local agent pipeline using open-source LLMs (e.g., Meta's Code Llama) combined with existing tools (e.g., pre-commit hooks, OPA for policy checks) to automatically review pull requests for code quality, dependency risks, and style compliance. Start with one repository and expand iteratively.
  • Why Now?: AI model costs and runtime are dropping; running smaller agents locally or on low-cost cloud instances (e.g., $0.10/hr VMs) is now feasible for solo developers, avoiding expensive API calls while retaining control.
  • Expected Upside: Reduce 30 - 50% of manual review cycles, catch bugs/security gaps early, and increase deployment confidence - leading to faster iteration and higher-quality output without relying on team infrastructure.

• What if you tracked your own development workflow events to uncover hidden bottlenecks in your solo projects?

  • Move: Instrument your environment (e.g., IDE, Git, CI) to log key events (e.g., first commit, build failure, deployment, ticket transition) into a personal event store (SQLite or DuckDB). Write SQL queries weekly to analyze patterns like cycle time, rework frequency, or test flakiness.
  • Why Now?: Tools like GitHub Actions, VS Code extensions, and open-source telemetry libraries (e.g., OpenTelemetry) make passive tracking lightweight and accessible - even for solo operators.
  • Expected Upside: Identify personal inefficiencies (e.g., slow builds, recurring review delays) and measure the impact of process changes (e.g., adopting AI linting), leading to 20 - 40% faster delivery by focusing on real blockers instead of vanity metrics.

• What if you treated yourself as a platform team, creating reusable AI-powered "skills" for your most frequent tasks?

  • Move: Define repeatable workflows (e.g., deploy script, API client generator, doc updater), then build small AI-driven tools (e.g., prompt-based agents in Python + LangChain) that execute them. Store them in a personal registry with clear inputs/outputs and reuse across projects.
  • Why Now?: With stable model APIs and local inference options (e.g., Llama.cpp on a laptop), solo developers can now run reliable, low-latency agents without enterprise infrastructure.
  • Expected Upside: Cut repetitive work by 50%+, accelerate project onboarding, and create a compounding toolset that scales across clients or products - turning individual effort into reusable capability.

• Implement passive event tracking early (e.g., via Git commits, CI/CD pipelines) to build a foundational dataset for identifying workflow bottlenecks without relying on vanity metrics like lines of code.
• Replace rigid KPIs (e.g., deployment frequency targets) with dynamic, context-aware dashboards that surface real-time bottlenecks such as slow code reviews or flaky tests, enabling targeted improvements.
• Adopt a dual feedback system by combining quantitative event data (e.g., build times, deployment failures) with periodic qualitative surveys (e.g., every 6 months) to balance metrics with human experience and avoid burnout.
• Establish a centralized registry for shared AI tools and skills (e.g., agents for GitLab, AWS) while allowing domain experts to maintain ownership - ensuring consistency, compliance, and reusability across projects.
• Focus AI investment on accelerating customer-facing feature delivery - not activity volume - and measure ROI by tracking shifts in KTLO (Keeping The Lights On) vs. BAU (Business As Usual) workloads.