A/B Testing Done Right: Statistics, Sample Size and Common Pitfalls

A/B testing sounds simple: show half your users version A, half version B, and pick the winner. In practice, badly run experiments are everywhere: tests stopped the moment they look significant, metrics chosen after the fact, sample sizes too small to detect anything real. The result is confident decisions based on noise.

This guide covers what you need to run experiments you can actually trust.

#Key takeaways

• Start with a hypothesis and a primary metric chosen before the test runs.
• Calculate the sample size up front, based on the smallest effect worth detecting.
• Don't peek and stop early on a fixed-horizon test; it inflates false positives. Use sequential methods if you need early stopping.
• Check sample ratio mismatch, guardrail metrics and novelty effects before celebrating.
• Statistical significance tells you an effect is probably not zero, not that it's large or important.

#Step 1: Write a hypothesis

A good hypothesis has a change, an expected effect and a reason:

Showing annual pricing savings on the plan page will increase paid conversions, because users currently don't notice the discount.

This forces clarity about what you're changing, which metric should move and why. That "why" matters when results come back ambiguous.

#Step 2: Choose metrics

• Primary metric: the single metric that decides the test, such as paid conversion rate. Pick one.
• Secondary metrics: help explain why (plan page clicks, checkout starts).
• Guardrail metrics: things that must not get worse (refund rate, page load time, support tickets, revenue per user).

Pick metrics sensitive enough to move within the test period. "12-month retention" won't work for a two-week test; use a leading indicator that correlates with it (product metrics that matter).

#Step 3: Randomize properly

• Randomize at the right unit: usually the user, not the session or pageview, so each person consistently sees one variant.
• Use deterministic hashing of user ID + experiment ID to assign variants.
• Make sure assignment happens before exposure, and log exposures when users actually see the change.
• Watch for interference: in marketplaces and social products, one user's treatment can affect others. That may require cluster or switchback designs.

#Step 4: Calculate sample size and power

Before running the test, decide:

• Baseline rate (current conversion, say 4%).
• Minimum detectable effect (MDE): the smallest improvement worth acting on, say a 10% relative lift (4% → 4.4%).
• Significance level (α): false positive tolerance, commonly 5%.
• Power (1 − β): probability of detecting a real effect of size MDE, commonly 80%.

For comparing two proportions, a standard approximation per variant is:

textn ≈ (z₁₋α/₂ + z₁₋β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₂ − p₁)²

With p₁ = 0.04, p₂ = 0.044, α = 0.05 (z = 1.96) and 80% power (z = 0.84):

textn ≈ (2.8)² × (0.0384 + 0.0421) / (0.004)² ≈ 7.84 × 0.0805 / 0.000016 ≈ 39,400 per variant

Roughly 39,000 users per variant to reliably detect a 10% relative lift from a 4% baseline. That surprises many teams. Small effects on low baselines need big samples. If you don't have that traffic, test bigger changes, pick a more sensitive metric, or accept a larger MDE.

pythonfrom statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.044, 0.04)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, ratio=1)
print(round(n))   # ≈ 39,000 per group

Also run for whole weeks (at least one or two full weekly cycles) to average out day-of-week effects.

#Step 5: Don't peek (or use methods designed for peeking)

With a fixed-sample test, checking results daily and stopping the first time p < 0.05 dramatically increases your false positive rate. Random fluctuations cross the threshold by chance along the way.

Options:

• Fixed horizon: decide the sample size up front and analyze once at the end.
• Sequential testing: methods designed for continuous monitoring with valid error rates (group sequential designs, always-valid p-values).
• Bayesian approaches: report probabilities and expected loss, but still need decision rules set in advance and awareness of priors.

#Step 6: Analyze correctly

Before looking at the primary metric:

1. Sample ratio mismatch (SRM) check: if you intended a 50/50 split but got 52/48 with large samples, something is broken (bots, assignment bugs, redirect failures). A chi-square test flags it. Don't trust results with SRM.
2. Data quality: exposures logged correctly, no duplicate users, the test ran the planned duration.

Then evaluate:

• Effect size with confidence intervals, not just p-values: "+8% relative lift (95% CI: +2% to +14%)".
• Guardrails: did anything important get worse?
• Segments: pre-registered segments only (new vs returning, mobile vs desktop). Slicing into dozens of segments after the fact guarantees false discoveries.

#Common pitfalls

#Novelty effects and long-term impact

Users sometimes click a new design because it's new (novelty) or resist it because it's unfamiliar (primacy). Plot the treatment effect over time. If it's shrinking or growing, the short-term result may mislead. For big changes, consider holdout groups that stay on the old experience for weeks or months to measure long-term impact.

#When not to A/B test

• Too little traffic to reach adequate power in a reasonable time.
• Obvious fixes: bugs, accessibility issues, broken flows. Just ship them.
• Strategic bets where the effect takes months to appear, or where network effects dominate.
• Ethical concerns: never experiment in ways that could harm users.

In these cases, use qualitative research, staged rollouts, before/after analysis with caution, or quasi-experimental methods (product discovery).

#Building an experimentation culture

• Maintain an experiment log: hypothesis, metrics, sample size, dates, results and decision.
• Share losses and nulls, not just wins. Most experiments don't win, and that's normal and valuable.
• Standardize analysis with a shared platform or templates, so each team isn't reinventing statistics.
• Review for statistical hygiene before launch, not after.

#FAQ

#What does a p-value actually mean?

It's the probability of seeing a result at least as extreme as yours if there were truly no difference. A p-value of 0.03 doesn't mean there's a 97% chance B is better. It means the data would be fairly unusual if A and B were identical.

#How long should an A/B test run?

Until it reaches the precomputed sample size, and for at least one full business cycle (usually one to two weeks minimum) to capture weekday and weekend behavior. Don't end early because results look good.

#Can I test more than two variants?

Yes (A/B/n), but each additional variant needs more traffic, and multiple comparisons increase false positives. Adjust for multiple comparisons, or test fewer, bolder variants.

#Frequentist or Bayesian A/B testing?

Both can work well. Frequentist methods are standard and well understood; Bayesian methods give intuitive probability statements. Either way, set decision rules in advance and avoid unprincipled early stopping. See the statistics every data scientist uses.