A/B test calculator
Compare two conversion samples and see the difference, uncertainty and statistical evidence.
B minus A
Observed conversion difference
Conversion difference
+0.5 points
Relative lift against A
+20%
Positive values mean B converts above A. Relative lift needs at least one conversion in A.
Evidence that B converts above A at the chosen threshold.
Rates and uncertainty
| Measure | Result |
|---|---|
| A conversion rate | 2.5% |
| B conversion rate | 3% |
| Confidence interval | 0.05 to 0.95 points |
| Approximate p-value | 0.031 |
95% two-sided interval
Interval values show their units.
At least one group has fewer than five conversions or five non-conversions. Treat these approximate results cautiously. A significance verdict is withheld.
Save the inputs and results as a CSV file.
Compare conversion samples
Use unique visitors assigned to A. Each visitor must appear in only one group.
95% confidence uses a 5% significance threshold for one planned analysis.
Choose before seeing results. Two-sided checks either direction. B above A checks only an increase.
Use the same conversion definition and observation window for both groups. Your figures stay in this browser.
Assumes independent visitors and one planned comparison.
This calculator compares two independent binomial samples: each visitor either converts or does not convert. Randomly assign visitors and keep each person in one variant.
Repeat sessions from the same person are not independent visitors. Before-and-after periods can differ in traffic mix, so they do not establish the same causal comparison.
The Agresti-Caffo method adds one conversion and one non-conversion to each group for its approximate test and interval. Displayed conversion rates use the original counts.
The calculation uses the adjusted B rate minus the adjusted A rate, divided by its standard error. The p-value compares that statistic with a normal distribution.
With fewer than five observed conversions or non-conversions in either group, the tool withholds a significance verdict. This screening rule does not guarantee accuracy above the boundary.
Small or extreme samples need methods suited to sparse counts. Adding observations in the formula does not create evidence from actual visitors.
Read significance and effect size
Small effects and interval bounds use basis points (bps), then microbasis points (µbp). One basis point is 0.01 percentage points. One microbasis point is one millionth of a basis point. Relative lift uses the same smaller units below 0.01%. A bounded range excludes zero when the effect is too small to show directly.
The hypothetical example has A at 250 / 10,000 = 2.5% and B at 300 / 10,000 = 3%. The observed difference is +0.5 percentage points, or +20% relative lift.
At 95% confidence with a two-sided test, the adjusted interval runs from 0.05 to 0.95 percentage points. The approximate p-value is 0.031, below the 0.05 threshold.
That supports a conversion difference under the test assumptions. The p-value is not the probability that B is better or that the result happened by chance.
Across repeated valid experiments, a 95% interval method aims to cover the true difference about 95% of the time. This method’s coverage is approximate.
A one-sided test checks only whether B converts above A. Its interval has an estimated lower bound and a fixed upper limit of +100 percentage points.
A commercially useful improvement must cover its costs. Compare the effect range with the extra contribution profit needed to justify the change, including refunds and implementation costs.
Read the guide to conversion rate optimisation to choose a storefront change for your next experiment.
Avoid repeated peeking
Stopping at the first favourable p-value gives random fluctuations repeated chances to cross the threshold. The false-positive rate then exceeds the rate planned for a single analysis.
Set the sample size and stopping rule before starting. Choose the conversion definition and test direction at that point too, then analyse at the planned endpoint.
If you need decisions while the experiment runs, use a sequential testing design that accounts for repeated looks. This calculator does not make that adjustment.
Check operational safety throughout a test, but keep safety monitoring separate from declaring a conversion improvement. Comparing many variants or metrics also needs a multiple-testing plan.