AI Testing from Production Logs: Generate Smarter Regression Tests with Tanvi Mittal

The podcast discusses leveraging AI for log mining and shift-right testing, emphasizing the underutilized potential of production logs as sources for regression test cases. AI tools, like the Tanvis Log Miner, analyze structured logs (e.g., fraud alerts, transaction sequences) to identify event patterns, generate test scenarios in Gherkin format, and prioritize testing based on high-impact features. This approach addresses challenges in traditional testing, particularly for AI-driven systems, which require continuous monitoring due to their non-deterministic behavior. The tool uses Python and transformer models to process logs, ensuring data privacy through anonymization (e.g., hash tokens), and aims to reduce manual log analysis by converting large datasets into structured test cases. However, it currently lacks full integration into CI/CD pipelines and relies on human validation for automation.

Key challenges highlighted include the need for structured, meaningful log data (e.g., account details, payment info) and the limitations of current AI tools in replicating real-world transaction sequences (e.g., simultaneous deposits and third-party payments). Shift-right testing, which uses production insights to inform test design, helps catch bugs that escape pre-deployment testing. The discussion also underscores the importance of collaboration between testers and developers to improve log quality and test coverage. While AI tools like Log Miner enhance efficiency by automating repetitive tasks, they are not meant to replace human judgment. Testers are advised to focus on understanding AIs probabilistic nature and integrating these tools into existing workflows rather than becoming AI/ML experts.

The podcast emphasizes that AI testing requires updated strategies, balancing automation with human oversight, particularly in high-risk domains like finance. Future efforts aim to expand the tools use cases (e.g., security vulnerability detection) and improve CI/CD compatibility. However, adoption hurdles persist, including concerns over open-source AI tools security and the need for domain-specific customization. The conversation also highlights the role of logs in identifying high-risk areas and prioritizing testing within limited timelines, while reinforcing that AI should augmentnot replacetesters expertise in complex, non-deterministic systems.