BONUS: DevCon London: Real Talk on AI ROI, Harnesses & Evals

The AI Native DevCon in London highlighted key discussions around AI integration, agentic coding, and practical applications in development workflows. Attendees explored the convergence of AI with fintech and blockchain to enhance reliability and efficiency, while speakers emphasized the need to avoid vanity metrics like code commits or token usage, focusing instead on business outcomes and user feedback. Central themes included balancing AI hype with tangible outcomes, aligning AI adoption with user needs, and addressing challenges in scaling agentic development without feature overload. Presenters like Chris Beattys stressed the importance of structured workflows over unstructured "vibe coding," advocating for deterministic processes and Einstein-like problem-solving approaches. Workshops on tools like "virtual SRE" for observability and "evals" for skill evaluation underscored practical implementation, alongside critiques of over-reliance on AI without human oversight or understanding of its implications.

Key sessions delved into the evolution of AI tooling, such as improved model capabilities and the role of infrastructure like MCP servers, alongside challenges in knowledge gaps within QA teams regarding AI/ML tools. Industry applications showcased AIs impact, including a 1,000 AML case review in 20 minutes and a 75% reduction in costs for legacy system modernization. Discussions also addressed emerging areas like harness engineering and tokenomics, while cautioning about the affordability and sustainability of AI solutions. The conference underscored the need for education and skill consolidation, particularly for non-technical roles leveraging tools like Autonomy AI, which aims to democratize development workflows through web-based interfaces and automated pull requests. Challenges in agent skills management, workflow inefficiencies, and the importance of change management alongside tool adoption were recurring concerns, emphasizing the necessity of optimizing organizational processes for successful AI integration.

• What if you built a minimal viable product (MVP) for an AI-augmented AML solution targeting small fintech startups?

  • Move: Leverage Autonomy AI to automate AML case analysis using pre-built templates and integrate with blockchain data sources to validate compliance.
  • Why Now?: The text highlights AIs success in reducing AML processing time from 3,0004,000 hours to 20 minutes, and startups are actively seeking efficient tools.
  • Expected Upside: Rapid adoption by niche markets, reduced operational costs, and a scalable product that could attract early-stage fintech investors.

• What if you implemented user feedback loops as your primary success metric instead of code output?

  • Move: Replace feature-centric KPIs (e.g., code commits) with user engagement scores, tracking how often end-users interact with your AI-driven tool.
  • Why Now?: Chris Beattys emphasized prioritizing business outcomes over vanity metrics, and the conference stressed aligning AI adoption with user needs.
  • Expected Upside: Higher user retention, clearer product-market fit, and reduced feature overload by focusing on what matters to users.

• What if you structured your agentic coding workflows using a deterministic process with embedded validation checks?

  • Move: Adopt the "harness engineering" framework (e.g., Teslas agent harness) to define environment hooks, verifiers, and constraints for AI agents.
  • Why Now?: The text criticizes "vibe coding" and underscores the importance of structured planning for agentic workflows. GitHubs potential for agentic readiness also opens new tooling opportunities.
  • Expected Upside: Reduced errors from AI-generated code, faster onboarding for non-technical team members, and greater confidence in deploying AI-augmented workflows.

• Engage with AI tooling and practices at events like AI Native DevCon's Expo Hall to identify practical tools and workflows that can directly enhance your development efficiency and agentic coding capabilities.
• Focus on business outcomes and user feedback rather than vanity metrics (e.g., code commits) by iterating on AI-integrated features based on real user needs and measurable engagement, as emphasized by Chris Beattys' talk.
• Implement structured, deterministic workflows in agentic coding, avoiding "vibe coding" by prioritizing problem analysis and disciplined planning (e.g., Einsteins approach) over unstructured prompt writing.
• Invest in education and skill development in AI/ML tooling (e.g., "evals," harness engineering) and QA practices to address knowledge gaps, ensuring your team (or yourself) can effectively validate and manage agent skills.
• Streamline internal workflows and prioritize change management before adopting AI tools, ensuring process optimization complements technological adoption to achieve measurable improvements in productivity and outcomes.