The bench › Optimization Tools
Optimization Tools
Notebook entries filed under Optimization Tools.
Optimization Tools
Stop Feeling Guilty About Peeking: Why Your A/B Test Tool Is the Real Problem
MetricPrimary conversion metric (e.g., click-through or purchase rate) and the false positive rate (Type I error) under repeated significance testing.
DesignA/B test comparing a control and a variant, with continuous or frequent monitoring, using either fixed-horizon or sequential/always-valid inference.
ReadIf the tool uses proper sequential methods, early stopping can be valid; if it uses fixed-horizon p-values, peeking inflates false positives. Read results with the tool's error guarantees in mind.
You have probably heard the warning: do not peek at your A/B test results before the planned end date, or you will invalidate the test.
Optimization Tools
Your A/B Test Tool Is Probably Lying: Why You Need Sequential Testing
MetricThe primary metric is the false positive rate (Type I error rate) of the decision rule under a true null effect, with secondary attention to the achieved power and the average sample size at the stopping decision.
DesignCompare two analysis policies on the same simulated or replayed experiment data: (a) a fixed-horizon test evaluated once at the planned sample size, and (b) a sequential test with pre-specified monitoring points or an always-valid boundary, stopping early only when the boundary is crossed.
ReadRead the result as a decision rule, not a single p-value: if the sequential boundary is crossed, stop and act; if the maximum sample size is reached without crossing, do not declare an effect. Report the effect estimate with a confidence interval appropriate to the sequential design.
Most teams do not run an A/B test and then look at it once.
Optimization Tools
A/B Tests Keep Lying to You: Stop Peeking and Use Sequential Testing
MetricA pre-defined primary metric such as conversion rate, click-through rate, or revenue per user, with a clear definition and measurement plan.
DesignA randomized controlled experiment (A/B test) where users are assigned to control or treatment, with sequential testing methods that allow valid interim analyses without inflating false positives.
ReadInterpret the effect size, confidence interval, and p-value in light of the sequential design; avoid declaring significance at arbitrary peeks and consider practical significance.
A/B testing is a cornerstone of data-driven optimization, but many practitioners unknowingly undermine their own experiments by repeatedly checking results before the planned sample size is reached.
Optimization Tools
Stop Peeking at Your A/B Test: Why Waiting Is the Best Optimization Tool
MetricPrimary conversion metric (e.g., click-through, sign-up, purchase) with a pre-defined minimum detectable effect and significance threshold.
DesignTwo-arm randomized controlled experiment (A vs. B) with a fixed sample size determined in advance; no interim analyses unless pre-registered with alpha-spending adjustments.
ReadOnly analyze after reaching the planned sample size. If p-value < alpha, reject null; otherwise, fail to reject. Do not stop early based on p-value.
A/B testing is a cornerstone of optimization, but its power is easily undermined by a common habit: peeking at results before the test concludes.
Optimization Tools
Stop Peeking at Your A/B Test and Let It Run
MetricThe primary metric should be a single pre-declared outcome tied to the hypothesis, such as conversion rate, click-through rate, or revenue per visitor, with any secondary metrics labeled as exploratory rather than confirmatory.
DesignA two-variant randomized controlled experiment: visitors are assigned randomly to control (current experience) or treatment (the change), each visitor is counted once, and the test runs until the pre-computed sample size is reached.
ReadRead the result only after the planned sample is reached: compare the treatment and control on the primary metric, check the confidence interval and p-value, and treat the estimated effect as uncertain rather than exact.
Peeking at an A/B test is one of the most common habits in experimentation, and it is also one of the most damaging.