How statistical significance works
An A/B test compares two conversion rates, and both move around from week to week. Significant means the gap is so large that it would rarely appear by chance if both versions were equally good. “Rarely” here means fewer than 5 out of 100 tests.
What the p-value tells you
The p-value answers one question: how often would a gap at least this large appear if A and B were actually equally good? A p-value of 0.03 means it would happen in about 3 out of 100 such tests. Below 0.05, the result counts as significant. The 0.05 threshold is a convention, and some teams set a stricter one before the test starts.
What the p-value does not tell you is how likely it is that B is better, or how large the advantage turns out to be. The American Statistical Association makes the same point in its statement on p-values (Wasserstein and Lazar 2016). To decide whether a switch is worth it, also look at the difference itself and at the ranges around both rates.
The 95% range around each rate
A measured conversion rate is an estimate. The calculator draws a bracket around each rate: the range that contains the true rate with 95% confidence, also called the confidence interval. It is calculated as a Wilson interval. The more visitors, the narrower the bracket.
If the ranges for A and B overlap a lot, the gap is usually not significant. Reading the brackets is not a reliable test on its own, though. Two ranges can overlap slightly and the test can still be significant. The test makes the call; the brackets show how precisely each rate is estimated.
Sample size: why small samples mislead
With few visitors, a handful of random conversions can make one version look far ahead. How many visitors a test needs depends on your current conversion rate and on the smallest difference it should still detect. At a 3% conversion rate, a test needs about 13,896 visitors per version to detect a lift to 3.6%. The same 20% lift needs about 42,642 visitors per version at a 1% conversion rate, and about 8,146 at 5%.
These numbers assume 80% power: if the difference is real, the test finds it in 80 out of 100 cases. The sample size formula for two proportions (Kim 2016) is: visitors per version = (1.96 + 0.84)² × (p₁ × (1 − p₁) + p₂ × (1 − p₂)) / (p₂ − p₁)², where p₁ is your current rate and p₂ the rate you want to detect.
When an underpowered test does come out significant, the measured gap is usually larger than the real one (Gelman and Carlin 2014). That is why the calculator tells you when your sample is smaller than the one the measured difference needs.
Peeking: why you don’t stop at the first significant result
Many testing tools show the p-value while the test runs. It moves around and dips below 0.05 from time to time even when both versions are equally good. If you check every day and stop the test the first time that happens, you will declare a false winner much more often than the 5% allows for.
So set the sample size before you start, using the “Plan a test” tab, and evaluate only once you reach it. Also run the test in full weeks, so every day of the week counts equally.
Sources, retrieved October 13, 2026: NIST/SEMATECH e-Handbook of Statistical Methods, z-test for two proportions and Wilson interval; Kim 2016 for the sample size; Wasserstein and Lazar 2016; Gelman and Carlin 2014.