A/B test significance calculator
Two-proportion z-test. Tells you whether a lift is real or noise.
Inputs
Results
Control CR = Control conversions / Control visitors × 100
Variant CR = Variant conversions / Variant visitors × 100
Relative lift = (Variant CR − Control CR) / Control CR × 100
z = (p₂ − p₁) / √(p̄(1-p̄)(1/n₁ + 1/n₂)), where p̄ is the pooled proportion
p = 2 × (1 − Φ(|z|)), Φ = standard normal CDF
Probability of seeing this lift or larger if the variants were truly identical. Below 0.05 = statistically significant at 95%.
Confidence = (1 − p-value) × 100
How confident you can be the lift is real. 95%+ is the conventional threshold to call the test.
Why this calculator is verified
Two-proportion z-test with pooled standard error. Identical to the chi-squared test of independence at one degree of freedom (χ² = z²) and to Optimizely's, VWO's, and Google Optimize's significance reporting at typical A/B testing scales. The standard normal CDF is computed via the Abramowitz & Stegun §7.1.26 erf approximation (accurate to ~1.5×10⁻⁷). For samples where n×p < 5 in either variant, Fisher's exact test is more rigorous; this calculator handles every typical ecom A/B test (1,000+ visitors per variant) without that caveat. The 95% confidence threshold (p < 0.05) is the conventional null-hypothesis cutoff established by Fisher in 1925.
Worked example
Control 350/10,000, Variant 408/10,000
p₁ = 3.50%, p₂ = 4.08%. Pooled p̄ = 758/20,000 = 3.79%. SE = √(0.0379 × 0.9621 × (1/10,000 + 1/10,000)) ≈ 0.00270. z = (0.0408 − 0.0350) / 0.00270 ≈ 2.146. P-value = 2 × (1 − Φ(2.146)) ≈ 0.0319. Confidence level reached: 96.81%. Above 95%, so the lift is statistically significant. The relative lift is +16.5%, meaningfully more than zero.
Sources for the formula
- Evan Miller, chi-squared significance calculator
Canonical operator reference. χ² and z-test are equivalent at 2x2 contingency tables.
- Wikipedia, two-proportion z-test
Textbook derivation of the z-test statistic this calculator computes.
- Abramowitz & Stegun, Handbook of Mathematical Functions §7.1.26
Source of the erf approximation used to convert z-score to p-value.
- Fisher, R.A. (1925). Statistical Methods for Research Workers
Origin of the p < 0.05 conventional significance threshold this calculator reports against.
Related on Ad-Lab
Want this run on your real account?
Free 15-minute audit. We screen-share your Google Ads + GA4 + Shopify, run every number above against the real data, and leave you the audit doc whether you become a client or not.
Book the call