The bench › Case Studies
Case Studies
Notebook entries filed under Case Studies.
Case Studies
Stop Peeking: How Fixed-Horizon A/B Tests Sabotage Your Results
MetricOne primary metric chosen before launch, with secondary and guardrail metrics tracked separately so that the decision rule is unambiguous.
DesignA two-arm randomized controlled experiment (A versus B) with a fixed horizon: a target sample size computed in advance and a single planned analysis at the end.
ReadRead the result only at the planned horizon: compare the primary metric between arms, report the effect with an interval, and treat any interim look as decision-altering rather than confirmatory.
A fixed-horizon test is a simple promise: you decide in advance how much data you will collect, you collect exactly that much, and then you look once.
Case Studies
Why Peeking at Your A/B Test Is Sabotaging Your Results
MetricThe primary metric is the false positive rate (Type I error rate) and the secondary metric is the estimated effect size at the time of stopping.
DesignA simulation study comparing a fixed-horizon test (analyzed only at the planned sample size) with a peeking test (analyzed after every new observation or at arbitrary intervals).
ReadIf the false positive rate exceeds the nominal alpha level, the test is invalid. The observed effect size at an early stop is likely to be inflated and should not be trusted.
Imagine you launch an A/B test and check the results every hour.
Case Studies
Why Your A/B Test Results Are Lying to You: A Case Study Guide
MetricOne primary metric chosen before launch, with secondary and guardrail metrics tracked separately so the decision rests on the primary metric.
DesignA randomized controlled experiment that splits eligible traffic into a control and one or more variants, with assignment, exposure, and analysis rules fixed in advance.
ReadRead the effect estimate together with its uncertainty and the pre-registered decision rule, not the raw difference or a single p-value.
A case study in experimentation is not a story about a clever trick that produced a big win.
Case Studies
Case Studies: Fixed-Horizon vs Sequential A/B Testing – Which Wins?
MetricThe primary metric is the decision outcome: whether the test declares a statistically significant winner, and if so, which variant. Secondary metrics include the false positive rate, the false negative rate, and the average sample size required to reach a decision.
DesignA simulation or thought experiment comparing two analysis approaches on the same stream of simulated or real data: one using a fixed-horizon test with a predetermined sample size and one using a sequential test with valid error control under continuous monitoring.
ReadThe result is read by comparing the decisions and the error rates of the two approaches. A fixed-horizon test that is peeked at repeatedly will have inflated false positive rates, while a properly designed sequential test maintains the nominal error rate. The sequential test may stop earlier on average when there is a large effect, but may require more samples when the effect is small or absent.
In the world of A/B testing, two broad families of methods dominate: fixed-horizon testing and sequential testing.
Case Studies
Stop Peeking at Your A/B Test: A Case-Study Guide to Better Decisions
MetricPrimary conversion rate (e.g., click-through rate or purchase rate), with secondary metrics for guardrails.
DesignA two-arm randomized controlled trial (A/B test) where users are randomly assigned to either the control (A) or treatment (B) experience.
ReadCompare the difference in conversion rates between arms using a statistical test (e.g., two-proportion z-test) and interpret the p-value and confidence interval in context.
In A/B testing, the temptation to check results early and often is strong.
Case Studies
Don't Trust Your A/B Test: Why Most Results Are False Positives
MetricA single, pre-defined primary metric such as conversion rate, click-through rate, or average order value.
DesignA controlled A/B test where users are randomly assigned to either the control (current version) or the treatment (new version).
ReadA statistically significant result indicates that the observed difference is unlikely to have occurred by chance alone, but it does not guarantee practical importance or validity.
A/B testing is a powerful method for making data-driven decisions, but it is not immune to error.