An A/B test compares two versions of a page by showing them to randomly chosen groups of visitors. If the only difference between the groups is the version they see, a difference in results can be attributed to the change. It is one of the most powerful tools for deciding with data, but only if the test is designed before it starts. Most failed tests do not fail because of statistics but because of decisions made too late.
Start from a hypothesis
A useful hypothesis has a precise structure: what I change, what effect I expect, and why.
If I move the shipping cost summary above the payment button, the rate of completed orders will rise, because many people currently abandon when they discover the costs at the last step.
The "why" is the most important part. If the test proves you right, you have learned something about your users. If it proves you wrong, you know which belief to revise. A test without a hypothesis, along the lines of "let's try a green button", produces at best a number, never a lesson.
Choose a primary metric
Before starting, choose a single metric that decides the outcome. It must be:
- tied to the goal: completed orders, submitted requests, confirmed sign-ups;
- sensitive to the change: a checkout change is unlikely to move visits to the home page;
- reliably measurable for both versions.
Alongside the primary metric, pick a few guardrail metrics that must not get worse: load time, errors, refunds, unsubscribes. All other metrics are exploratory: interesting, but they decide nothing.
Fixing the primary metric in advance protects you from a common temptation: once the test is over, searching through dozens of numbers for the one that improved and declaring victory.
How many visitors you need
The number of visitors required depends on three things:
- baseline rate: how well the page converts today;
- minimum detectable effect: the smallest difference worth discovering;
- desired level of protection against false positives and false negatives.
The practical rule is intuitive: the smaller the effect you are looking for, the more visitors you need, and the number grows quickly. Detecting an improvement of a few tenths of a point on a two per cent rate can require tens of thousands of visitors per variant. Sample size calculators do the arithmetic: use them before the test, not after.
If the result far exceeds the available traffic, the test is not feasible as designed. The alternatives are to test bigger, bolder changes, to pick a more frequent metric higher up the funnel, or to give up the test and decide by other means.
Duration
Even with plenty of traffic, a test should run for at least one or two full weeks. Behaviour varies between weekdays and weekends, morning and evening, the start and end of the month. A three-day test risks measuring the effect of the day rather than the change.
The duration should be set in advance and respected. Stopping a test as soon as the result looks favourable is one of the most frequent mistakes, because random differences are large at the start and shrink over time.
Assignment and consistency
Each visitor must always see the same version for the whole test. If they see A one day and B the next, the data gets mixed. Assignment must be random and independent of characteristics such as device, source or time of day.
You also need to decide who enters the test. If the change concerns the checkout, it makes sense to include only people who reach the checkout: including every visitor to the site dilutes the effect and lengthens the test.
A one-page document
In my work, before every test I write a short document with:
- the full hypothesis;
- a description of both versions, with images;
- primary and guardrail metrics;
- the audience included;
- sample size and expected duration;
- the decision rule: what we will do with each possible result.
The last item is decisive. Writing in advance "if the difference is not significant, we keep the current version because it is simpler" avoids endless debates once the test is over.
Before going live
A technical check prevents many disappointments: make sure both versions work in all major browsers, that the primary metric's events are recorded for both and that the traffic split matches the plan in the first few hours.
In short
- An A/B test starts from a hypothesis stating the change, the expected effect and the reason.
- Choose a single primary metric and a few guardrail metrics in advance.
- Calculate beforehand how many visitors you need; if it is too many, change the test.
- Run the test for at least one or two full weeks and do not stop it early.
- Write the decision rule before seeing the results.
