A/B testing pitfalls: sample mismatch, novelty and peeking

A well-designed A/B test can still give the wrong answer if something breaks while it runs or during analysis. The most insidious mistakes are not calculation errors, which a tool handles for you, but practical ones: traffic split badly, analysis done too early, an effect that fades after a few weeks. Here are the most frequent ones and how to notice them.

Sample ratio mismatch

If the test is meant to split traffic fifty-fifty, the two groups should have very similar counts. A small difference is normal; a large one is not. This problem is known as sample ratio mismatch, or SRM.

Example: after two weeks version A has 10,000 visitors and B has 9,200. At these volumes a gap like that is too large to be random. Something is treating the two versions differently. Typical causes:

• version B loads more slowly and some visitors leave before their entry into the test is recorded;
• version B has a JavaScript error that prevents the event from being sent in certain browsers;
• a redirect to the variant loses parameters or cookies for some users;
• bots are filtered differently in the two versions.

A test with SRM is not reliable, whatever the result. Many tools run an automatic check, but it is worth verifying by hand: a chi-square test of the observed proportions against the expected ones is enough. The rule is simple: before looking at the primary metric, check the split.

Peeking at results

Checking progress every day and stopping as soon as statistical significance appears is a classic mistake known as peeking. Every check is one more chance of a false positive: with enough checks, even two identical versions will sooner or later look different.

There are two solutions:

• decide the duration in advance and look at the result only at the end;
• use sequential methods, designed specifically for repeated checks and offered by some tools; in that case the thresholds are stricter and account for the checks.

Looking at the data during the test is useful for spotting technical problems such as errors or SRM. What must be avoided is deciding the outcome early.

Novelty and habituation

A striking change can attract clicks simply because it is new. Regular users notice something different and explore it. After a few weeks the interest wears off and the effect shrinks or disappears. That is the novelty effect.

The opposite phenomenon is change aversion: regular users find a new layout awkward and penalise it at first, even if it works better in the long run.

To recognise them:

• watch the effect day by day: if it declines steadily, suspect novelty;
• separate new and returning visitors: new visitors have no habits to break, so their result is closer to the lasting effect.

Segments discovered after the fact

The test ended with no significant difference. Then someone notices that among tablet users on Tuesdays, version B won clearly. Is that a discovery? Almost never. With dozens of possible segments, one of them will show a marked difference by chance.

Segments chosen after seeing the data are good for generating new hypotheses, not for drawing conclusions. If a segment looks promising, it should be checked with a new, dedicated test.

Interference between tests

If several tests run on the same page at the same time, their effects may add up or cancel out. A test on the headline and another on the button below it, run together without coordination, make it hard to tell what produced what. It is better to avoid this, or to assign visitors so that the combinations are balanced.

External contamination

External events can affect one group more than the other: an advertising campaign bringing different traffic, a payment system outage, a public holiday. Writing down everything that happens during the test helps to interpret strange results. In my work I keep a simple log of dates and events for every active experiment.

A final checklist

Before accepting a result, check that:

1. the traffic split matches the planned proportions;
2. the planned duration was respected;
3. the primary metric is the one fixed at the start;
4. the effect is stable over time and not only in the first few days;
5. no significant external event hit just one group;
6. the guardrail metrics have not got worse.

In short

• Always check the traffic split: a mismatch makes the test unreliable.
• Do not stop the test as soon as significance appears, unless you use sequential methods.
• Watch the effect over time and separate new and returning visitors to spot novelty.
• Segments discovered afterwards are for forming hypotheses, not for drawing conclusions.
• Record external events and overlapping tests that may contaminate the results.