Contact, if you are interested in this website / development / Sponsorship / Advertisement / Partnership →

A/B testing, explained properly

What significance really means, why sample size comes first, and the mistakes that quietly turn experiments into coin flips.

Data.Studio Editorial · Updated November 2026 · 10 min read

What an A/B test actually does

An A/B test randomly assigns users to a control (A) and a variant (B), then compares an outcome such as conversion rate. Randomisation is the magic ingredient. Because the groups differ only by chance and by the change you made, a large enough difference in outcomes can be attributed to the change.

"Large enough" is the hard part. Even two identical pages will show different conversion rates on any given week, purely from random variation. Statistics tells us how big a difference must be before random variation stops being a plausible explanation.

The vocabulary, in plain English

  • Null hypothesis: the default assumption that B performs the same as A.
  • p-value: if the null hypothesis were true, the probability of seeing a difference at least as large as the one observed. A p-value of 0.03 means "a gap this big would appear about 3% of the time by chance alone". It is not "a 97% chance B is better".
  • Significance level (α): the threshold you set in advance, commonly 0.05. If p < α, you reject the null hypothesis.
  • Confidence interval: a range of plausible values for the true difference. It tells you the size of the effect, not just whether one exists.
  • Statistical power: the probability your test detects a real effect of a given size. 80% power means that if the effect is real, you'll catch it four times out of five.
  • Minimum detectable effect (MDE): the smallest real improvement your test is designed to detect at that power.

Sample size comes first

The single most common mistake is starting a test without deciding how long it will run. Sample size depends on three inputs:

  1. Baseline rate. Lower baseline conversion rates need more traffic.
  2. MDE. Smaller effects need far more traffic. Halving the MDE roughly quadruples the required sample.
  3. α and power. Stricter thresholds need more data.

For example, detecting a 10% relative lift on a 5% baseline (5.0% → 5.5%) at α = 0.05 and 80% power needs roughly 31,000 visitors per variant. Detecting a 20% lift needs about a quarter of that. Use the Sample Size Calculator to plan, then run the test until you reach that number. Not sooner, and not later.

Six mistakes that invalidate tests

1. Peeking and stopping early

Checking results daily and stopping as soon as p dips below 0.05 dramatically inflates false positives, because random fluctuations cross the line far more often than 5% of the time over many looks. Fix: commit to a sample size, or use a sequential testing method designed for repeated looks.

2. Ignoring weekly cycles

Behaviour on Monday differs from Saturday. Run tests in whole weeks, at least one and ideally two, even if you hit the sample size sooner.

3. Sample ratio mismatch

If you planned a 50/50 split and got 52/48 on large traffic, something is wrong: a redirect losing users, a bot filter, a caching bug. Results from a broken split can't be trusted. Our A/B Test Calculator checks this automatically.

4. Testing too many metrics

Track twenty metrics at α = 0.05 and you should expect one "significant" result by chance. Pick one primary metric in advance. Treat the rest as guardrails or exploratory.

5. Novelty and primacy effects

Returning users may click a new design simply because it's new, or avoid it because it's unfamiliar. If the effect fades over time, look at new users separately or run longer.

6. Confusing significance with importance

With enough traffic, a 0.1% lift becomes "significant" while being commercially meaningless. Always read the confidence interval and translate it into money or users.

Reading a result

A good readout says: "B increased checkout conversion from 5.0% to 5.8% (+16% relative). The 95% confidence interval for the absolute difference is +0.2 to +1.4 percentage points (p = 0.01). At current traffic, that's between X and Y extra orders per month." It states the effect, the uncertainty and the business impact. Never just "it won".

When not to A/B test

  • When traffic is too low to reach a sensible MDE within a few weeks. Make bolder changes or use qualitative research.
  • When the change is obviously right (fixing a broken button) or legally required.
  • When the outcome takes months to observe. Use leading indicators carefully, or a holdout group.
Try it: plan with the Sample Size Calculator, then analyse with the A/B Test Calculator. It shows the p-value, confidence interval and an SRM check.