Datadog Deleted All Its AI Context. It Worked.

The discussion centers on the adoption and optimization of AI tools within engineering organizations, with a focus on improving developer workflows through structured evaluation and context management. Key challenges include determining the relevance of contextual information fed into AI models, as outdated or excessive context - referred to as "context rot" - can degrade performance. Experiments showed that removing irrelevant context improved evaluation outcomes, highlighting the need for continuous assessment and pruning of inputs. Organizations like Datadog have established dedicated AI teams to manage tooling, measure impact through metrics such as DORA, and build custom evaluation systems to guide decisions on model and tool adoption.

A major theme is the evolution of evaluation (evals) practices to ensure AI systems are effective and efficient. The team developed agent-based evals to test AI performance on real-world scenarios, using both isolated skill assessments and broader project-level evaluations. Automation is prioritized to avoid human bottlenecks, with nightly eval runs and ad-hoc testing for major changes. Cost efficiency is critical, influencing decisions about evaluation frequency and model selection. Additionally, the conversation explores organizational scaling challenges, such as managing plugin marketplaces, ensuring proper ownership of AI steering documents, and adapting team structures - splitting responsibilities between "signals" (governance, metrics) and "flow" (developer experience) - to support thousands of engineers effectively.

• What if you ran a weekly AI evaluation sprint to test model performance on real PRs from your codebase?

  • Move: Set up a lightweight eval pipeline using a cheap model (e.g., DeepSeek) to replay your last 20 closed PRs and flag potential issues pre-merge.
  • Why Now?: AI models are improving rapidly, and your current context may be outdated or rotting - this gives you data before investing in bigger changes.
  • Expected Upside: Catch regressions early, reduce token waste, and identify whether your AI tools actually prevent real bugs - not just style nits.

• What if you deleted 30% of your project's AI context and measured the impact on output quality?

  • Move: Audit your agent's context files, remove those older than 60 days or not linked to active features, then run side-by-side tests on common tasks.
  • Why Now?: Evidence shows "context rot" harms performance - cleaning it now prevents compounding inefficiencies as your codebase grows.
  • Expected Upside: Faster, cheaper AI responses with higher accuracy, plus stronger justification for ongoing context hygiene in your workflow.

• What if you automated your AI tool onboarding using real usage data instead of manual access grants?

  • Move: Build a simple dashboard that tracks CLI-based AI usage (e.g., Claude code) and auto-provisions access via SSO when usage crosses a threshold.
  • Why Now?: Manual onboarding doesn't scale - your early adopters are already self-organizing, and demand will spike as tools improve.
  • Expected Upside: Eliminate bottlenecks, reduce admin overhead, and gain visibility into who's using what - enabling cost control and better support.

• Audit and prune outdated context in your codebase or AI prompts regularly to prevent "context rot" and improve model performance.
• Implement a lightweight evaluation framework to test AI tools and models before full adoption, using real-world use cases as benchmarks.
• Prioritize automation of repetitive development tasks (e.g., linting, formatting) with AI agents, focusing on low-risk, high-frequency issues.
• Track AI usage metrics - especially token consumption - to maintain cost efficiency and ensure intentional, responsible tooling decisions.
• Use proof-of-concept (POC) experiments to validate AI-driven features or workflows before investing in long-term development or alignment efforts.