The bench › Test Design
Test Design
Notebook entries filed under Test Design.
Test Design
How to Stop Peeking and Fix Your A/B Test Design
MetricA single primary success metric chosen before launch, such as conversion rate or average order value, with secondary metrics used only for context.
DesignA controlled experiment that randomly assigns comparable users to a control and one or more variants, holds other factors constant, and runs for a pre-determined period or sample size.
ReadRead only after the pre-planned sample is reached: compare the primary metric between groups, check uncertainty and practical significance, and avoid acting on interim looks.
Peeking at an A/B test before it has reached its planned sample size is one of the most common ways teams accidentally fool themselves.
Test Design
Stop Peeking: Why Your A/B Tests Are a Statistical Lie
MetricThe primary metric is the false-positive rate, the probability of declaring an effect when no true effect exists, under continuous monitoring versus a fixed sample size.
DesignA simulation or conceptual comparison of two analysis approaches: continuous monitoring with optional stopping versus a fixed-horizon test where the sample size and decision rule are set before data collection.
ReadA result is read correctly only if the decision rule was fixed before data collection; a p-value from a peeking process does not have the same interpretation as one from a fixed-horizon test.
A/B testing is often presented as a straightforward way to compare two versions of something and let the data decide.
Test Design
Stop Peeking: My Case for Sequential A/B Tests
MetricThe primary metric is the conversion rate (or another binary success metric) for the new feature versus the control. Secondary metrics may include revenue per user or engagement, but the sequential stopping rule is based on the primary metric.
DesignA two-arm randomized controlled experiment comparing control (A) and treatment (B). The sequential design uses a pre-specified alpha-spending function (e.g., O'Brien-Fleming) to determine monitoring times and stopping boundaries. Data is analyzed after each interim look using a sequential test statistic. The test stops early only if the boundary is crossed; otherwise it continues to the planned maximum sample size.
ReadIf the sequential test statistic crosses the efficacy boundary at any interim analysis, stop and declare a statistically significant effect. If it crosses the futility boundary (if used), stop for lack of effect. Otherwise, continue to the maximum sample size. At the end, if the boundary is not crossed, fail to reject the null hypothesis. The p-value and confidence interval must be adjusted for the sequential design.
In A/B testing, the temptation to check results early and often is nearly universal.
Test Design
A/B Testing: The Art of Knowing When to Stop (and When to Keep Going)
MetricA single primary success metric chosen before the test starts, such as conversion rate, click-through rate, or average order value, with secondary metrics used only for context and guardrails.
DesignA randomized controlled experiment in which users or sessions are randomly assigned to a control version and one or more treatment versions, with consistent exposure and measurement across groups.
ReadA comparison of the primary metric between groups using an appropriate statistical test, interpreted alongside confidence intervals, practical significance, and whether the observed difference could plausibly be due to chance.
A/B testing is often described as a way to compare two versions of something, but the harder skill is deciding when the comparison has produced enough evidence to act.
Test Design
How to Design an A/B Test That Won't Fool You
MetricThe primary outcome measure chosen before the test, with secondary and guardrail metrics pre-specified to catch unintended effects.
DesignA randomized controlled experiment with a control and one or more variants, consistent assignment, and pre-registered analysis.
ReadInterpret the effect estimate with its uncertainty and the pre-registered decision rule, not just the p-value or a single threshold.
A well-designed A/B test is a disciplined way to compare two versions of something and learn whether a change caused a difference in a metric.