How it works
This calculator runs a two-proportion z-test to check whether variant B’s conversion rate differs from control A’s by more than random chance would explain. It pools both groups to estimate the standard error under the assumption that there is no real difference.
The p-value is the probability of seeing a difference at least this large if the variants truly performed the same. If it is below 1 − confidence (0.05 at 95%), the result is called statistically significant.
The confidence interval for the difference shows the plausible range of the true effect. An interval that barely excludes zero means the lift may be much smaller than the observed value.
Formula
p̂ = (c_A + c_B) ÷ (n_A + n_B) · z = (p_B − p_A) ÷ √( p̂(1 − p̂)(1/n_A + 1/n_B) ) · p = 2(1 − Φ(|z|)) two-sided
Questions people ask
Should I use a one-sided or two-sided test?
Two-sided is the safer default: it detects a difference in either direction. Use one-sided only if you decided before the test that you only care whether B is better.
How many visitors do I need?
It depends on your baseline rate and the smallest lift you want to detect. Detecting a 10% relative lift on a 5% baseline at 95% confidence and 80% power takes roughly 31,000 visitors per variant.
Does significant mean the change is worth shipping?
Not necessarily. Significance says the difference is probably real, not that it is large. Check the size of the lift and its confidence interval against the cost of the change.