A/B Test Calculator

An A/B test compares two conversion rates and asks whether the gap between them is larger than chance would produce. With 480 conversions from 12,000 control visitors (4.0%) and 552 from 12,000 on the variant (4.6%), the variant is 0.6 points ahead, a 15% relative lift. The question is whether a gap that size could plausibly appear between two identical pages.

The standard method is the two-proportion z-test. Under the assumption that the variant makes no difference, both groups share one conversion rate, estimated by pooling them: 1,032 conversions from 24,000 visitors. That gives a standard error for the difference, and the z-score is the observed difference divided by it, here about 2.29. The p-value is the probability of a z at least that extreme if the null were true, read from the normal distribution: about 0.022 two-sided. Below 0.05, so significant at 95%; above 0.01, so not at 99%.

The p-value is widely misread. It is not the probability that the variant is better, and it is not the probability that the result is a fluke. It is the probability of data this extreme on the assumption that there is no effect. And a 95% threshold means exactly what it says: test a change that does nothing and one time in twenty it will come out significant. Test twenty headlines, or check twenty metrics on one test, and one of them is expected to "win" on noise alone.

PEEKING is the most common way tests go wrong. A p-value is valid for one look at a sample size fixed in advance. Check a running test every day and stop the first time it dips below 0.05, and the real false-positive rate is several times the 5% you think you are running at, because a random walk crosses a threshold far more often than it ends below it. Decide the sample size before the test starts and read the result once. Sequential testing methods exist that allow early stopping legitimately, but they use adjusted thresholds; an ordinary p-value checked repeatedly is not one of them.

That is why the sample size is the most useful number on the page. It depends on the baseline rate, the smallest lift you care about, the confidence level and the power (the chance of detecting the lift if it is real, conventionally 80%). Small lifts on low rates need very large samples: halving the lift you want to detect roughly quadruples the visitors required. The table shows how fast it grows. For the defaults, detecting a 15% lift at 80% power needs about 18,000 visitors per variant and the test had 12,000. It reached significance anyway, which happens. But an underpowered test that does reach significance tends to overstate the effect, since only the draws that happened to run high cleared the bar. Expect the lift to shrink after launch.

The power figure for the observed effect is reported because people ask for it, with a warning: it is a direct function of the p-value and tells you nothing the p-value did not. A non-significant test with low observed power is not evidence that the variant has no effect. It is evidence that the test was too small to say.

ONE-SIDED OR TWO-SIDED. A one-sided test only asks whether the variant is better. It reaches significance with a smaller gap, which is why it is tempting, and it is blind to the variant making things worse. Choosing one-sided after seeing which way the result went halves the p-value for free and is a form of peeking. Two-sided is the safer default.

Finally, statistical significance is not practical significance. With enough traffic a 0.02-point difference is significant and worth nothing. The confidence interval is the honest summary: it is the range of true differences consistent with the data, and its low end, multiplied by your traffic and the value of a conversion, is what the change can be counted on to be worth.

The method assumes visitors are independent and randomly assigned, and that the test ran long enough to cover a full business cycle of weekdays and weekends. It does not correct for testing several variants or several metrics at once.