Skip to content

UncategorizedBy 2 min read

Break Even Calculator

%

What the control converts at today.

%

A 10% relative lift on 5% means detecting 5.50%.

How rarely you accept a false positive.

Your chance of catching the effect if it is real.

Test direction

Two-sided detects a drop as well as a lift. Use it unless you have a strong reason not to.

Across all variations combined. Used to estimate run time.

Sample size needed — visitors per variation
Total, both variations
—
Estimated run time
—
Detecting a move from
—

How this is calculated

This uses the standard two-proportion test. With p₁ as the baseline rate, p₂ as the target rate, and p̄ as their average:

n = ( zα·√(2p̄(1−p̄)) + zβ·√(p₁(1−p₁) + p₂(1−p₂)) )² ⁄ (p₂ − p₁)²

The z-values come from the normal distribution: zα from your confidence level (halved if the test is two-sided) and zβ from your power. The result is rounded up, per variation.

Reading the result honestly

This number is a floor, not a finish line. Two things break it in practice: stopping early because the result looks good, which inflates your false-positive rate well past the confidence level you chose; and running past the planned sample while continuing to check, which does the same thing more slowly.

Decide the sample size first, run to it, then look. If you cannot reach the number in a reasonable window, the honest move is to widen the effect you are willing to detect rather than to accept a smaller sample and hope.