The AI in Your Product No Longer Follows the Rules. Now We Have to Test Not Correctness, but Efficiency

There comes a point in the life of a big product when we start to optimize the performance of user scenarios. Instead of asking "How does it work?" we ask ourselves "why?" and “How effective?”

The euphoria from the introduction of the first “smart” algorithm or ML model is aggravated by the dawning realization that all this will have to be tested. It is important for you to make sure that the new logic functions as intended and that there are no regressions. And then it turns out that there is no “correctness” of work, only an incomprehensible “efficiency” and it is not clear how to measure it correctly. What doesn't make things easier is that you can't capture the input.

In the talk we will tell you how Avito and other large companies get out of a difficult situation with the help of a search quality infrastructure. Using the example of the ranking launch history in Avito, we will analyze the main problems and techniques that allow us to partially solve this impossible task.

A report for bold product developers and caring engineers who are ready to transform the product.