Your A/B tests are broken

User statistics analysis and split-testing are complementary to testing in the following way: we test to figure out if we implemented what we wanted to implement, and then we run an online user experiment to see if the users like it.

Unfortunately, the machinery needed to make experiment-based decisions is usually very fragile and frequently gets broken causing the whole system to produce incorrect results. In this talk, we’ll review several common problems that exist in split-testing systems. We’ll also provide ways to check if these problems are present in your system.