Interactive figure · Hypothesis testing

A/B Testing & Sequential Testing

Every A/B test ends in one of two worlds: the change did nothing (null), or it genuinely worked (alternative).

The story

Here we explore an example of an experiment betting on pure upside lift.

Let’s say, hypothetically, you and a friend decide to go on a fishing trip. You’ve done your research, scouted a high-potential spot on the lake, and packed two tools: a standard fishing rod and a large netting setup.

The rod is your default option - reliable, familiar, and hooks a steady stream of fish. But your friend argues that the net is more effective - the rod could miss smaller fish entirely, or it could land a massive haul in one scoop.

To decide which strategy to commit to, you hire a team of local fishermen and split them into two equal groups (sample size per group)

Control: Uses the rod.
Treatment: Uses the net.

Before casting, you define your hypotheses:

Null Hypothesis (H₀): The fishermen with the net perform the same as or worse than those with the rod (average catch per angler)
Alternate Hypothesis (H₁): The net delivers a significant upside lift over the rod in average catch per angler

Now, you throw the net to see if the alternative is high enough to reject the baseline and make the switch.

Power - chance you detect the win, if your feature really is better

—

α - your tolerance for declaring a winner when there's no real effect

—

MDE - smallest effect this design reliably detects (80% power)

—

Set Significance level (α)

Threshold for deciding when an observed result is too unlikely to be random coincidence.

By lowering α, you reduce risks of declaring an effect which is missing

Set Statistical power target

By increasing power, you reduce the risk of missing a true effect

The chance you detect the win, when the feature really is better.

Size per group(n = 20K)

Expanding user group size reduces metric variance that smaller groups might include (due to outliers). This is detected through curves getting skinnier.

The effect: You reliably detect smaller, incremental metrics (shown by lower MDE)

The con: But you now need to run the experiment longer to collect as many users

Users in each arm. More users sharpen both curves.

Variance reduction (VR = 0%)

Keep sample size consistent

Effect sized = 2.50

How far apart the null and alternative curves sit, in standard errors.

The larger distance between curves ⇒ undeniable difference b/w treatment and control groups

Express the same effect on a real metric — d is derived from your inputs, and a noisier metric makes the same lift harder to see.

Baseline (%) —

Experiment timeline

Day of experiment

Assumes a two-week experiment.

Metrics / variants testedk = 1

How many hypothesis tests this experiment really runs — every extra metric or treatment arm is another chance for noise to look significant, if not corrected appropriately.

For ex: Imagine testing 5 different net designs against the standard rod

Treatment : control ratio50 : 50

You usually want your treatment-to-control ratio to be 1:1 because it maximizes statistical power with lowest standard error, and allows you to detect a win with the smallest possible sample size.

Total traffic (2n) is held fixed — the ratio only reallocates it.

Sequential TestingAdvanced

⏳ The cost of peeking It’s day three of the trip and the net crew is comfortably ahead. Do you call it there and hand everyone nets? That is the peeking problem.

Frequentist A/B testing requires running experiments to their full predetermined duration before making a decision. Every interim peek inflates your false positive rate.

However, the opportunity cost: postponed rollout of high-upside features, pro-longed exposure to underperforming ones.

PS: I still believe running full-length tests are beneficial.

⚡ The valid approach Solution: sequential testing / Bayesian framework (with expected loss). These techniques allow for continuous evaluation, which helps in detecting winners early, thereby speeding up experimentation and launch decisions.

Step 1 — The scenario

The feature: a checkout optimization designed to increase Average Order Value (AOV).

Historical stream variance (σ²) from historical baseline AOV data

Tuning parameter (τ²)iTau (τ) represents the expected size of the treatment effect you are looking for — a good rule of thumb is to set it as the MDE. τ = $2 MDE → τ² = 4

Significance level (α) Type I error target

Running Maximum ΛiCalled the Likelihood Ratio, intuitively this means how extreme is the alternate data under the null hypothesis. = 1

Step 2 — The math (optional)

Step 3 — See it run

A fresh AOV experiment, simulated day by day at these parameters. The run stops the day Λ̃ crosses the boundary (or reaches day 14 without crossing).

Day 0 — setup

Null & Alternate Default SetupRead me

Default: At genesis, both curves- normal distributions with σ = 1, 20K samples per arm, 5% significance level, 80% power.

The sample and variance-reduction sliders re-derive σ as
σ = √(20K/n) × √(1−VR)
reshaping both curves at once;

Power and MDE are recomputed from these settings on every change - so each number you see is always the exact consequence of the current design, relative to this default.