A Discrepancy-Based Design for A/B Testing Experiments

The aim of this paper is to introduce a new design of experiment method for A/B tests in order to balance the covariate information in all treatment groups. A/B tests (or "A/B/n tests") refer to the experiments and the corresponding inference on the treatment effect(s) of a two-level or multi-level controllable experimental factor. The common practice is to use a randomized design and perform hypothesis tests on the estimates. However, such estimation and inference are not always accurate when covariate imbalance exists among the treatment groups. To overcome this issue, we propose a discrepancy-based criterion and show that the design minimizing this criterion significantly improves the accuracy of the treatment effect(s) estimates. The discrepancy-based criterion is model-free and thus makes the estimation of the treatment effect(s) robust to the model assumptions. More importantly, the proposed design is applicable to both continuous and categorical response measurements. We develop two efficient algorithms to construct the designs by optimizing the criterion for both offline and online A/B tests. Through simulation study and a real example, we show that the proposed design approach achieves good covariate balance and accurate estimation.

本文的目的是介绍一种用于A/B测试的新实验设计方法，以便平衡所有处理组中的协变量信息。A/B测试（或“A/B/n测试”）是指对一个两水平或多水平可控实验因素的处理效应进行的实验以及相应的推断。通常的做法是使用随机设计，并对估计值进行假设检验。然而，当处理组之间存在协变量不平衡时，这种估计和推断并不总是准确的。为了克服这个问题，我们提出了一个基于差异的准则，并表明使该准则最小化的设计显著提高了处理效应估计的准确性。基于差异的准则是无模型的，因此使得处理效应的估计对模型假设具有稳健性。更重要的是，所提出的设计适用于连续和分类响应测量。我们开发了两种高效算法，通过针对离线和在线A/B测试优化该准则来构建设计。通过模拟研究和一个实际例子，我们表明所提出的设计方法实现了良好的协变量平衡和准确的估计。