How to Check If Your A/B Test Is Significant

Your A/B test dashboard shows variant B converting at 4.2% against variant A’s 3.8%, and someone on the team wants to ship B today. It looks like a win. But with a few hundred visitors per variant, that 0.4 percentage-point gap could just as easily be noise — and shipping a “winner” that isn’t one is how teams end up flip-flopping between variants every quarter for no real gain.

The obvious approach — eyeballing the two conversion rates and picking the bigger one — is the annoying part, because it works fine when the gap is huge and the sample is large, and fails silently everywhere else. What you actually need is a quick statistical check, not a gut call.

What A/B test significance actually involves

A two-proportion z-test compares two conversion rates while accounting for how much data backs each one. It asks: given the sample sizes and observed rates, how likely is a gap this large (or larger) to show up by chance alone, if the two variants were actually equally good? That likelihood is the p-value. A common threshold is p < 0.05, meaning there’s less than a 5% chance the observed gap is just noise — but the threshold itself is a judgment call, not a law of nature.

The test doesn’t tell you the winner is better by some fixed amount forever. It tells you whether the data you’ve collected so far is strong enough to trust the direction of the result.

Why people get stuck here

• Small sample sizes. A 10% relative lift on 50 conversions per variant is exciting on a dashboard and statistically meaningless in practice.
• Stopping the test too early. Checking results daily and stopping as soon as you see a “significant” result inflates your false-positive rate — this is sometimes called “peeking.”
• Confusing statistical and practical significance. A result can be statistically significant and still not matter — a confirmed 0.1% lift might not be worth the engineering cost of shipping it.
• Ignoring variance in traffic sources. If variant A got disproportionately more returning users than variant B, the comparison is contaminated before the math even starts.

What a good significance calculator looks like

Uses the right test for the data

Comparing two conversion rates (a proportion, not a continuous metric like revenue per user) calls for a two-proportion z-test specifically — using the wrong test type gives you a wrong p-value even with correct inputs.

Reports uplift alongside significance

A p-value alone doesn’t tell you whether the win is worth shipping. Seeing the relative uplift next to the significance result lets you judge both “is this real” and “is this worth it” at once.

Makes the inputs obvious

Visitors and conversions per variant should be the only required inputs — anything more complicated invites input errors that quietly wreck the result.

Common mistakes to avoid

• Declaring a winner the moment the p-value crosses 0.05 for the first time during an ongoing test, instead of committing to a sample size or duration up front.
• Running the test with a sample size too small to detect anything but a huge effect, then concluding “no difference” from an underpowered test.
• Comparing more than two variants with a test built for two, which inflates the chance of a false positive across all the comparisons.
• Treating a statistically significant result as permanent — seasonality, traffic mix, and audience can shift what “wins” the next month.
• Ignoring the sample size context entirely and only looking at the percentage gap.

How to do it with A/B Test Significance Calculator

Online Tool Store’s A/B Test Significance Calculator runs the two-proportion z-test entirely in your browser.

1. Open the A/B Test Significance Calculator.
2. Enter visitors and conversions for variant A and variant B.
3. Review the calculated conversion rates, uplift, and p-value.
4. Use the significance result to decide whether the gap is real or still needs more data.

Because it computes everything locally, you can re-run the numbers as often as your test data updates without waiting on a dashboard refresh.

Frequently asked questions

What sample size do I need for a reliable A/B test?

It depends on your baseline conversion rate and the minimum lift you care about detecting — smaller expected effects need larger samples. As a rough sanity check, a few hundred conversions per variant is usually a minimum before trusting the result; a few thousand is safer for smaller lifts.

What does a p-value of 0.05 actually mean?

It means that if the two variants truly performed identically, there’d be roughly a 5% chance of seeing a gap this large purely by random variation. It’s not the probability that variant B is better — it’s a measure of how surprising the observed gap would be under the assumption of no real difference.

Can I trust a “significant” result from a test I stopped early?

Be cautious. If you checked the result repeatedly and stopped as soon as it looked significant, the true false-positive rate is higher than the p-value suggests. Deciding your sample size or test duration before starting, and sticking to it, avoids this problem.

Final thought

A p-value doesn’t replace judgment — it just tells you whether the gap you’re looking at is likely to be real or likely to be noise. Combine it with a sensible sample size and a lift that’s actually worth shipping before you call the test done.

Try the free A/B Test Significance Calculator