Name
Abstract | ||
|---|---|---|
A/B tests are an important aspect of modern applications, particularly on the internet, as they allow businesses to optimize their user or customer experience to maximize usage and ultimately profits. In this paper we review Sweet IM A/B testing system, with focus on statistical aspects and methodology. We describe all parameters of the Sweet IM A/B test environment including randomization mechanism, data collection and analysis. We also provide some interesting examples and case studies from real A/B tests that took place at Sweet IM. Analyses of count data like number of searches or content sent per user are the most significant performance indicators for the majority of Sweet IM tests. Accuracy of such analyses is a key-success factor and can increase the ROI of the tests dramatically. We expand the popular method of the analyses of counts based on Poisson distribution and show its inappropriateness when dealing with over dispersed Poisson. We propose to use Negative Binomial distribution as appropriate solution for over dispersion in search and content count data. We show that the conclusions from analyses of specific A/A and A/B tests run in this application with NB differ from those with an incorrect Poisson assumption. |
Year | DOI | Venue |
|---|---|---|
2011 | 10.1109/ICDMW.2011.19 | ICDM Workshops |
Keywords | Field | DocType |
sweet im a,poisson distribution,sweet im test,b test environment,sweet im,b testing,specific a,content count data,b testing system,real a,proper statistical analysis,b test,hypothesis testing,negative binomial,data analysis,profitability,binomial distribution,hypothesis test,customer satisfaction,negative binomial distribution,internet,count data,statistical analysis,data collection,performance indicator | Data collection,Data mining,Binomial distribution,Overdispersion,Computer science,A/B testing,Count data,Negative binomial distribution,Poisson distribution,Statistics,Statistical hypothesis testing | Conference |
Citations | PageRank | References |
1 | 0.36 | 3 |
Authors | ||
2 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Slava Borodovsky | 1 | 1 | 0.36 |
| Saharon Rosset | 2 | 1087 | 105.33 |