1 Overview
A fixed-horizon test is a simple promise: you decide in advance how much data you will collect, you collect exactly that much, and then you look once. The appeal is that the promise is easy to state and easy to break. Teams break it constantly, usually for understandable reasons. Someone checks the dashboard on day two, sees the variant ahead, and feels a pull to call it early. Someone else checks on day five, sees the variant behind, and starts hunting for explanations. Neither reaction is irrational, but both change the meaning of the final number.
The core problem is that a significance threshold assumes a single look at a fixed amount of data. When you look repeatedly and let each look influence whether you continue, you are effectively running many tests and reporting the most flattering one. The math does not care that the looks felt like casual monitoring. Each peek is a chance to cross the threshold by luck, and the more chances you take, the more likely at least one of them crosses it.
This article walks through how fixed-horizon tests are framed, designed, sized, and read, and why peeking quietly converts a controlled experiment into something closer to an open-ended search. The goal is not to forbid monitoring, but to separate operational monitoring from decision-making, and to make the cost of blurring that line concrete.
2 Framing the question
A useful test starts with a hypothesis that could be wrong. Not a slogan, not a hope, but a specific claim about a mechanism: if we change this element, then this behavior should shift, because this is what we believe drives the outcome. The hypothesis should name the audience, the change, and the expected direction. If the claim cannot fail, it is not a hypothesis, it is a description.
Alongside the hypothesis, you need one primary metric. One. Secondary metrics are fine, and guardrail metrics are essential, but the decision must rest on a single pre-specified measure. When teams leave the primary metric vague, they create room for post-hoc selection: whichever metric moved the most becomes the story. That is not analysis, it is narration. Choosing the primary metric in advance forces the team to agree on what success actually means before anyone has seen the data.
It also helps to state the decision rule up front. What will you do if the primary metric improves, stays flat, or worsens? What effect size would justify shipping, and what would justify abandoning? Writing this down before launch protects against the very human tendency to reinterpret results after the fact. The framing stage is where most of the rigor lives, because it is the only stage where you are not yet tempted by the numbers.
3 Designing the test
The design of a fixed-horizon test is mostly about constraints. You need a clean randomization unit, such as users or sessions, and that unit must be assigned consistently so the same person does not drift between arms. You need a control that genuinely represents the status quo, not a subtly improved version that makes the treatment look weak. You need exposure to be comparable, so differences in traffic sources or timing do not masquerade as treatment effects.
You also need to decide the horizon before launch. The horizon is not a feeling; it is a target sample size or a target duration, and it should be tied to the smallest effect you care about detecting. If the change is expected to produce a small lift, you need more data to distinguish it from noise. If the change is expected to be large, you can detect it sooner, but you still commit to the planned analysis rather than stopping at the first favorable reading.
Operational monitoring is still allowed. You can watch for broken tracking, crashes, or obviously skewed assignment. What you cannot do is let a peek at the primary metric change your plan. If you look at the outcome and then decide to extend, shorten, or stop, you have moved from a fixed-horizon test to a sequential procedure, and the standard thresholds no longer apply. That distinction is the whole game.
4 Sample size and validity
Sample size is where fixed-horizon tests earn their credibility. The required size depends on the baseline rate of the metric, the smallest effect worth detecting, the significance level you are willing to accept, and the power you want. None of these are universal constants. They depend on your setup, your traffic, and your tolerance for error. A test that is adequately powered for one product may be hopelessly underpowered for another, even with the same metric.
Underpowered tests are dangerous because they produce noisy results that invite overinterpretation. When the sample is small, the interval around the estimated effect is wide, and a wide interval can contain both a meaningful gain and a meaningful loss. Reporting only the point estimate hides that ambiguity. Reporting the interval makes it visible, which is exactly why intervals should accompany every headline result.
Validity threats compound the problem. Peeking is the most common, but it is not the only one. Assignment bias, inconsistent exposure, metric definitions that change mid-flight, and external events that affect one arm more than the other all threaten the comparison. The fixed-horizon discipline does not eliminate these threats, but it removes one large, self-inflicted source of error and makes the remaining ones easier to reason about.
5 Running and reading the results
When the planned horizon arrives, analyze the primary metric as pre-specified. Compare the arms, report the effect, and include an interval so readers can see the range of plausible values. If the interval is wide, say so. If the effect is small relative to the noise, say that too. The result is not a verdict about the change in general; it is evidence from one setup, one population, and one period.
Resist the urge to re-analyze until something looks significant. Slicing by device, geography, or time can generate interesting hypotheses, but those slices are exploratory. They are not confirmatory, and treating them as such reintroduces the same inflation that peeking causes, just in a different costume. If a subgroup looks promising, the honest next step is a new test designed to examine that subgroup directly.
Also separate statistical significance from practical importance. A result can be statistically distinguishable from zero and still be too small to matter for the business. Conversely, a result can be practically interesting but statistically uncertain, which is a signal that more data or a better design is needed. The fixed-horizon result gives you a clean input to that judgment, not a substitute for it.
6 What to change next
The most valuable output of a fixed-horizon test is often the next question. If the primary metric moved, ask why. Which mechanism behaved as predicted, and which did not? If the metric did not move, ask whether the hypothesis was wrong, the implementation was weak, or the measurement was too noisy to detect the effect. Each answer points to a different next step.
If the test was underpowered, the next step is usually to increase the sample or to target a larger expected effect. If the test was adequately powered and the effect was null, the next step is to revise the hypothesis rather than rerun the same test hoping for a different outcome. If the effect was positive but the interval was wide, the next step is a confirmation test with a pre-specified horizon and a clear decision rule.
Finally, treat the process itself as something to improve. Document the hypothesis, the primary metric, the sample-size reasoning, and the decision rule before launch. After the test, record what you learned and what you would do differently. Over time, this record becomes more useful than any single result, because it shows where your predictions tend to be right, where they tend to be wrong, and how much data you typically need to tell the difference.
7 Frequently asked questions
Is it ever acceptable to stop a test early? Yes, but only under a pre-specified rule that accounts for multiple looks, such as a formal sequential or group-sequential design. Stopping because the result looks good is not a rule; it is a reaction, and it inflates error rates. If you want the option to stop early, build that option into the design before launch.
Can I check the dashboard without ruining the test? You can monitor operational health, such as traffic volume, error rates, and assignment balance. The danger is looking at the primary metric and letting it influence your decision. If you cannot resist acting on what you see, either avoid the outcome metric until the horizon or adopt a design that formally permits interim analysis.
What if my test is underpowered? You have three honest options: collect more data with a revised plan, target a larger effect that the current sample can detect, or accept that the test cannot answer the question and treat it as exploratory. What you should not do is declare a winner based on a noisy estimate.
Does a significant result mean the change will work everywhere? No. It means the data are unlikely under the null hypothesis in this setup. Generalization depends on how representative the population, period, and implementation are. Treat every result as conditional evidence and plan the next test accordingly.