An email campaign reports two hundred attributed purchases. It looks like a success, but an awkward question remains: how many of those two hundred customers would have bought anyway, even without receiving the email? The difference between what happened and what would have happened without the intervention is called incrementality, and it is the only measure that truly tells you what an action is worth.
Attribution and incrementality
Attribution models distribute credit for conversions among the observed touchpoints. They do not say whether those touchpoints caused them. A typical example is advertising on your own brand: someone who searches the company name and clicks the ad is attributed to the ad, but many of those people would have arrived anyway through the organic result just below.
| Question | Tool |
|---|---|
| Which channels touch people who convert? | attribution |
| How many extra conversions does a channel generate? | incrementality test |
The two measures answer different questions and both are needed.
The holdout principle
The most solid method is the same as an experiment: the audience is split at random into two groups, one receives the intervention and the other does not. Because assignment is random, the two groups are alike in everything else, and the difference in results can be attributed to the intervention.
For a newsletter:
- randomly draw a share of the list, for example 10 percent, which will not receive promotional emails for a set period;
- compare purchases per subscriber in the two groups over the same period;
- the difference, multiplied by the number of subscribers who receive the emails, estimates incremental sales.
If the group receiving emails buys 4 percent more than the holdout group, that is the real effect, not the two hundred attributed purchases.
Geographic tests
When you cannot control who sees an intervention, as with radio, billboards or many ad platforms, you can work by area. You choose zones similar in size and in their historical sales trend, switch the campaign on in only some of them and compare the trend with the others.
It requires care: areas are never identical, local events can disturb the comparison, and each zone needs enough volume. That is why an observation period before the test is often used, to check that the zones move in a similar way when none is exposed.
Planned pauses
The simplest and weakest method is to switch a channel off for a few weeks and watch what happens to total conversions. If nothing changes, the channel was probably producing little added value. The limit is that you are comparing with the past, so seasonality and other changes get mixed up with the effect of the pause. It is useful as a first indication, not as proof.
What you need
- Enough volume. If conversions are few, the difference between the groups will be drowned in noise. Estimate in advance how large the effect must be to be detectable.
- A metric decided beforehand. Purchases, revenue, qualified requests: choose it before seeing the results.
- A fixed period. Stopping the test when you like the result introduces bias.
- An accepted cost of exclusion. The holdout group forgoes the intervention; if it works, something is lost. That is the price of the information.
Common mistakes
- Choosing the holdout group non-randomly, for example from the least active subscribers.
- Letting the holdout group receive the intervention anyway through another channel.
- Reading the result over too short a period, ignoring purchases that arrive later.
- Generalising a result obtained in an unusual period, such as the holidays.
How to use it
You don't need to test everything, all the time. In my work I suggest starting with the most expensive channel, or the one where attribution looks most optimistic, and repeating the test a few months later, because incrementality changes as the offer, the audience and the competition change.
In short
- Incrementality measures conversions beyond what would have happened anyway.
- Attribution shows who touches the journey, not what causes the conversion.
- A random holdout group is the most reliable method; geographic tests are an alternative.
- Decide the metric, duration and required volume before starting.
- Start with the most expensive channel and repeat the test over time.
