An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Decisions and Modelsintermediate

Experiments and A/B testing

Q: A test on 900 visitors per arm shows conversion of 4.0 per cent in control and 4.6 per cent in treatment. The team reports a 15 per cent lift. The best response is

note that at this sample size the gap sits inside ordinary random variation, so the test needs to run far longer. With rates near four per cent and 900 users per arm, the standard error of the difference is close to nine tenths of a percentage point, so a gap of six tenths is smaller than the noise. Detecting an effect this size needs tens of thousands of users per arm. The relative figure sounds impressive precisely because the base rate is small.

Q: Which finding most undermines the validity of a completed A/B test?

across 20,000 eligible visitors the treatment group received 53 per cent of traffic, and a sample ratio test rejects the configured even split at p below 0.001. A three-point departure from an even split is roughly eight standard errors at this sample size, which is what the sample ratio test is reporting. Chance is not a plausible explanation, so the assignment or the logging is suspect and the two groups may differ systematically. A negative or disappointing result is a finding rather than a defect. The longer run does no damage here because the extension rule was fixed before launch and applied blind. Extending after watching the numbers would be peeking under another name, and that would undermine a fixed-horizon test.

An A/B test is a controlled experiment run on live users. People are assigned at random to version A or version B, and one metric chosen in advance is compared between the groups. Randomisation is what licenses the causal reading, because it balances the two groups in expectation on everything except the change, including traits nobody thought to record. It does not make the realised groups alike, so any two arms still differ by chance, and that leftover variation is what the statistics are there to measure. The hard parts are not the arithmetic. They are choosing the metric before you look, running long enough to detect a difference worth acting on, and resisting the pull to stop the moment the numbers look good.

Why it matters

Two identical shops on the same street, the same passing crowd, the same weather. Change the window display in one and count what happens. If the display is the only systematic difference, what changed in sales is attributable to it. Assigning visitors at random online does the same job, and does it better, because chance splits the crowd on traits you could never have measured. It splits them evenly on average rather than exactly, which is why a small gap between the two arms still has to be weighed against ordinary variation.

Before you read on — recall

A test on 900 visitors per arm shows conversion of 4.0 per cent in control and 4.6 per cent in treatment. The team reports a 15 per cent lift. The best response is

Formulas

Absolute and relative lift

\Delta = p_T - p_C, \qquad \text{lift} = \frac{p_T - p_C}{p_C}

Here

p_C

is the conversion rate in the control group and

p_T

the rate in the treatment group. Moving a rate from four per cent to four and a half is an absolute gain of half a percentage point and a relative lift of about twelve and a half per cent. Reporting only the relative figure flatters small changes on small bases, so show both.

Standard error of a difference in two rates

SE = \sqrt{\frac{p_C(1-p_C)}{n_C} + \frac{p_T(1-p_T)}{n_T}}

The yardstick the observed gap is measured against. With rates near four per cent and 25,000 users in each arm,

SE

is roughly 0.18 percentage points, so a half-point gap is close to three standard errors. A gap that large would be unlikely if the two versions performed identically, which is what a small p-value reports. With 1,000 users per arm,

SE

is about 0.9 points and the very same gap tells you nothing at all.

How many users each group needs

n \approx \frac{16\,\sigma^2}{\Delta^2} \quad \text{per group}

A standard rule of thumb for detecting a difference of size

\Delta

under the usual conventions. For a conversion rate,

\sigma^2 = p(1-p)

. Detecting a lift from four per cent to four and a half needs about

n \approx 16 \times 0.0384 / 0.000025

, which is roughly 24,600 users in each group. Small effects need large samples, and that arithmetic belongs before the test rather than after it.

Worked examples

Scenario

An online retailer tests a new checkout button. After two days the treatment is ahead by a wide margin and the product manager wants to ship it.

Solution

Two days is usually too early for two separate reasons. The sample may still be small enough that the gap sits inside ordinary random variation. The larger problem is peeking. If you check repeatedly and stop the first time the result looks good, you have stacked the deck, because noise crosses the threshold sooner or later even when the two versions are identical. Fix the run length and the metric in advance, or use a method designed for continuous monitoring. Shopping behaviour also varies from one day to the next, so a two-day test misses part of the population.

Scenario

A subscription business tests a redesigned sign-up page. Sign-ups rise nine per cent and the team declares victory.

Solution

Ask what happened after sign-up. A page that promises more than the product delivers lifts the metric being watched and raises cancellations two months later. Pair the primary metric with guardrails, here retention at 60 days and support contacts per new customer. Then check the split itself. A treatment share of 52 per cent looks close to half, but across 40,000 eligible visitors it sits about eight standard errors away from the configured split, which is a sample ratio mismatch and would almost never arise by chance. That points at the assignment or the logging rather than at the design, and until it is explained the result cannot be trusted however large the effect looks. On a few hundred visitors the same 52 per cent would be unremarkable, so the size of the sample decides whether the split is evidence of anything.

Common mistakes

✗A statistically significant result is a result worth shipping. Significance says that a difference at least this large would be unlikely if the two versions performed identically. It is a statement about the data given that assumption, not the probability that the assumption is false, and it says nothing about whether the difference is large enough to pay for the change, so decide the smallest worthwhile effect before the test runs.
✗You can stop a test as soon as it reaches significance. Checking repeatedly and stopping on a good look inflates false positives badly, because random noise wanders across the threshold on its own. Fix the sample size in advance, or use a sequential method built for monitoring.
✗Randomisation is a formality that could be replaced by matching on known traits. Matching balances only what you thought to measure. Random assignment balances the unmeasured traits as well, in expectation rather than exactly, so whatever imbalance remains is chance rather than something systematic. That is the reason the comparison supports a causal claim, and the reason the claim comes with an interval rather than a certainty.
✗A test showing no difference was a waste of traffic. A credible null result tells you not to spend on the change and rules out a story the team believed. That is worth what the test cost.

Revision bullets

•Random assignment balances groups in expectation, which licenses the causal reading
•Choose the metric and the run length before the test starts
•Report absolute and relative change, since relative alone flatters small effects
•Sample size grows with variance and with the inverse square of the effect
•Peeking and stopping early inflates false positives
•Pair the primary metric with guardrails that catch damage elsewhere

Quick check

A test on 900 visitors per arm shows conversion of 4.0 per cent in control and 4.6 per cent in treatment. The team reports a 15 per cent lift. The best response is

Which finding most undermines the validity of a completed A/B test?

Connected topics

Correlation vs Cause Sampling Bias What Is a Decision Supervised vs Unsup.KPI Design Web Mining Analytics Leverage

More in Decisions and Models

Decision Process Models & Abstraction Decision Support

Sources

Kohavi, Tang & Xu (2020)
Kohavi, R., Tang, D., & Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.
Practical treatment of online experiments, covering sample-size arithmetic, guardrail metrics, mismatched traffic splits and the cost of stopping early.
Kohavi et al. (2009)
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. "Controlled experiments on the web: survey and practical guide." Data Mining and Knowledge Discovery, 18(1), 2009.
Survey setting out the mechanics of web experimentation and the recurring ways results get misread.
Fisher (1935)
Fisher, R. A. The Design of Experiments. Oliver & Boyd, 1935.
Origin of randomisation as the device that makes treatment groups comparable on unmeasured as well as measured traits.

How to cite this page

Dr. Phil's Quant Lab. (2026). Experiments and A/B testing. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-experiments-ab-testing