Scaling AI in QA at Yandex: Measuring the Impact of LLMs Across 1,000+ Engineers

At Yandex, we are used to measuring everything—and the adoption of GenAI in testing is no exception. In this talk, we’ll explore how to evaluate the impact of AI not based on gut feeling but through clear, reproducible metrics. Which indicators truly reflect value? How do you separate real gains from the initial “wow” effect? And how do you build a measurement framework in an organization with 1,000+ QA engineers so that the numbers are representative and the conclusions are sound?

We will examine all of the above through real-world implementations: LLMs for generating checklists and test cases, tools that accelerate E2E test automation, and AI agents conducting regression testing.