1 Overview
A case study in experimentation is not a story about a clever trick that produced a big win. It is a story about a question, a design, and a decision. When teams say their A/B test results are lying to them, they usually mean the numbers pointed one way and reality pointed another. The gap almost never comes from the statistics being wrong. It comes from the question being vague, the design being leaky, or the reading of the result being selective.
This guide walks through a generic case study structure you can apply to your own tests. It covers how to frame a hypothesis, how to pick a primary metric, how to design a clean split, how sample size and validity shape what you can conclude, how to read results without fooling yourself, and what to change next. The emphasis throughout is on process rather than on any single outcome, because a well-run test that shows no effect is still valuable information.
Treat every number as conditional. A result depends on the population you sampled, the time window you used, the metric definition you chose, and the size of the sample you collected. Change any of those and the estimate can move. The goal is not to find a result that feels satisfying. The goal is to build a decision you can defend before you see the data.
2 Framing the question
Most misleading tests begin with a question that cannot be answered by the test being run. If the hypothesis is simply that a new design will perform better, there is no defined direction, no defined metric, and no defined threshold for acting. A useful hypothesis names the change, the population, the metric, and the expected direction. It also states what would count as evidence against it, so the team is not left negotiating after the fact.
Suppose the change is a rewritten call to action on a signup page. The hypothesis might be that the rewrite increases the share of eligible visitors who complete signup within a fixed window. That phrasing forces choices. Who is eligible? What counts as completion? Which window? A vague version of the same idea invites metric shopping later, where the team scans dashboards for whichever number moved most.
Framing also means deciding what the test is not trying to answer. Secondary metrics and guardrail metrics matter, but they should not become the primary basis for a decision after launch. If a guardrail metric degrades, that is a real signal worth investigating, but it does not retroactively turn a secondary metric into the primary one. Write the decision rule down before the test starts, including the minimum effect that would justify shipping. That single habit prevents many of the misreadings that make results appear to lie.
3 Designing the test
A clean design assigns units to variants at random, keeps the assignment stable for the duration of the test, and ensures that each unit experiences only one variant. The unit might be a user, a session, a device, or an account. The choice matters because metrics are computed over units, and interference between units can distort the comparison. If two users share a household, a device, or a social feed, one user's treatment can bleed into another's experience.
The control should represent the current experience as it actually exists, not an idealized version of it. Variants should differ in the way the hypothesis describes and not in unrelated ways. If a variant changes copy, layout, and load time at once, the result cannot tell you which change mattered. That is fine if the goal is to test a bundle, but it should be stated as such. Instrumentation should be in place before launch, with exposure events logged at the moment the variant is shown, not at the moment of conversion.
Pre-registration is the discipline that holds the design together. Before launch, record the hypothesis, the primary metric, the analysis window, the exclusion rules, and the decision threshold. This does not require a formal registry. A short written plan shared with the team is enough. The point is to separate the act of designing from the act of interpreting, so that the interpretation is not quietly reshaped to fit whatever the data turned out to show.
4 Sample size and validity
Sample size is not a formality. It determines the smallest effect the test can reliably detect. A test with too few units will produce noisy estimates, and a noisy estimate can look large by chance. A test with far more units than needed can detect effects so small that they are not worth acting on. The right size depends on the baseline rate of the metric, the variability of the metric, the smallest effect worth detecting, and the error rates the team is willing to accept.
Validity threats are the reasons a result might not mean what it appears to mean. Peeking at results repeatedly and stopping when they look good inflates the chance of a false positive. Testing many metrics or many segments and reporting the most exciting one does the same. A sample ratio mismatch, where the observed split between variants departs from the intended split, often signals a logging or assignment problem. Novelty effects can make a new experience look strong early and fade later, while seasonality can make a temporary period unrepresentative.
None of these threats is exotic. They appear in ordinary tests run by careful teams. The defense is procedural: fix the analysis window, limit the number of primary comparisons, check the split before interpreting outcomes, and treat unexpected movements in guardrail metrics as reasons to pause rather than as footnotes. Validity is not a property you add at the end. It is a property you protect from the beginning, and it depends on the setup and the sample size as much as on the statistical method.
5 Running and reading the results
When the test ends, read the result in the order the plan specified. Start with the primary metric and its uncertainty, not with the largest movement on the dashboard. An effect estimate without an interval is easy to overinterpret, because the same estimate can be consistent with a wide range of true effects. The interval tells you how much precision the sample actually bought. If the interval is wide, the honest conclusion is that the test was not informative enough to support a strong claim.
Next, check the mechanics. Was the split close to the intended ratio? Did exposure logging behave as expected? Were there outages, tracking changes, or traffic shifts during the window? A result produced during a broken week is not a result about the change. Then look at secondary and guardrail metrics as context, not as replacements for the primary metric. A guardrail degradation may justify not shipping even when the primary metric improved, and that is a legitimate outcome of the decision rule.
Finally, separate statistical significance from practical importance. A small effect can be statistically distinguishable from zero while being too small to matter for the business, and a large-looking effect can be too uncertain to act on. The question is not whether the number is impressive. The question is whether the evidence is strong enough, and the effect large enough, to justify the change given the costs. Reading results well means resisting the urge to promote whichever comparison tells the best story.
6 What to change next
The output of a test is not only a ship or no-ship decision. It is an updated understanding of the system. If the primary metric moved in the expected direction with adequate precision, the next step is to confirm the mechanism and consider whether the change generalizes beyond the tested population. If the result was inconclusive, the next step is usually to increase the sample, narrow the question, or improve the metric definition rather than to rerun the same test hoping for a different answer.
If the result was negative but the design was sound, that is a finding worth keeping. It narrows the space of plausible improvements and prevents the team from repeating the same idea under a new name. If the result was positive but a guardrail moved, the next step is to investigate the guardrail before scaling. If the split was mismatched or the instrumentation was suspect, the next step is to fix the measurement and treat the test as uninformative rather than as evidence in either direction.
Over time, the most valuable change is often to the experimentation process itself. Shorter, clearer hypotheses, a single primary metric per test, pre-registered decision rules, and routine checks on assignment and logging reduce the frequency of misleading results. The statistics do not need to be exotic. They need to be applied to a question that was framed carefully, a design that was protected, and a sample that was large enough to support the claim being made.
7 Frequently asked questions
Why do results sometimes reverse when a test is repeated? Reversals usually trace back to a difference in population, time window, metric definition, or sample size rather than to a flaw in the original statistics. A first test on a small or unusual slice can produce an estimate that does not hold in a broader or later sample. Repeating a test is informative only when the setup is comparable, and even then, some variation between runs is expected.
Can a test be too small to trust? Yes. A small sample produces wide uncertainty, which means the estimate can be far from the true effect in either direction. The remedy is not to interpret the point estimate more confidently but to recognize that the test cannot support a strong claim. If the decision is urgent, the team may need to act on judgment and prior evidence while labeling the test as inconclusive.
Does a statistically significant result mean the change should ship? Not by itself. Significance speaks to whether an effect is distinguishable from noise under the model, not to whether the effect is large enough to matter or whether guardrails held. Shipping decisions combine the primary metric, the uncertainty around it, guardrail behavior, cost, and the strategic context. The test informs the decision. It does not make it.