The bench › Statistical Analysis
Statistical Analysis
Notebook entries filed under Statistical Analysis.
Statistical Analysis
Peeking Is Not the Only Sin: The Hidden Risks of A/B Tests
MetricOne primary outcome metric chosen in advance, supported by a small set of secondary and guardrail metrics that capture potential trade-offs.
DesignA randomized controlled experiment that assigns units to treatment and control, fixes the decision rule and analysis plan before launch, and monitors only pre-specified health checks.
ReadRead the effect estimate with its uncertainty interval, the pre-specified decision threshold, and the practical significance; a non-significant result is not proof of no effect, and a significant result is not proof of importance.
A/B testing is often taught as a simple sequence: form a hypothesis, split the traffic, wait for significance, and ship the winner.
Statistical Analysis
Peeking vs. Sequential Testing: Why Your A/B Test Lies
MetricThe primary metric is the false-positive rate (Type I error rate) under continuous or repeated significance testing, with secondary attention to power and confidence interval coverage.
DesignSimulate or analytically compare fixed-horizon testing (one analysis at a predetermined sample size) with naive peeking (frequent unadjusted significance checks) and with sequential methods (alpha spending, group sequential, or always-valid inference) that adjust for multiple looks.
ReadA significant result from a naive peeking procedure should be interpreted with caution because the actual false-positive rate can be much higher than the nominal level; sequential methods provide valid error control if the design is prespecified and followed.
A/B testing rests on a simple promise: if you run an experiment to a predetermined sample size and analyze it once, a statistically significant result at the five percent level means there is only a five percent chance of seeing such an extreme result if the null hypothesis were true.
Statistical Analysis
Why Your A/B Test Results Are Lying: Peeking, Power, and Practical Significance
MetricA pre-defined primary metric, such as conversion rate or average revenue per user, with secondary metrics for context.
DesignA randomized controlled experiment in which users are assigned to control and treatment groups and the primary metric is compared between groups.
ReadRead the result by checking whether the observed difference is statistically significant, whether the confidence interval excludes the minimum effect of interest, and whether the effect is practically meaningful.
A/B tests are often treated as simple scoreboards: run two versions, wait for a winner, and ship it.
Statistical Analysis
Stop Peeking: Why Your A/B Test Results Are a Lie
MetricThe primary metric defined before the test, such as conversion rate or average revenue per user.
DesignA randomized controlled experiment comparing a control version with one or more treatment versions.
ReadStatistical significance and effect size are read only after the pre-planned sample size is reached; interim peeking inflates false positives.
You have probably heard the warning: do not peek at your A/B test results before the test is scheduled to end.
Statistical Analysis
Peeking Is Killing Your A/B Tests: Stop It Now
MetricThe primary metric is the pre-declared success metric for the experiment (for example a conversion rate or a continuous outcome such as revenue per user), evaluated with a single planned analysis rather than repeated interim significance checks.
DesignA two-arm randomized controlled experiment (control versus treatment) with a fixed horizon: the sample size and analysis time are specified before launch, assignment is randomized, and the decision rule is applied once at the end.
ReadRead the result only at the planned analysis: compare the treatment effect and its confidence interval against the pre-declared decision threshold, and treat any interim significant reading as noise rather than evidence.
Peeking at an A/B test means checking whether the result is statistically significant before the experiment has reached its planned sample size, and then deciding to stop early when it looks good.