People who receive email are already selected
Subscribers, recent customers and abandoned-cart audiences differ from people who never entered the list. High revenue after an email does not establish that the email caused all of it. The targeting rule can select customers who were likely to purchase anyway.
- Define eligible subscribers
- Randomise the send policy
- Measure the full outcome
- Match evidence to MMM
The incrementality guide explains the counterfactual. For email, define the eligible audience first and test a send policy within that audience where feasible.
The effect you can estimate depends on the intervention, comparison and outcome you define. The incrementality and experiments learning path connects this idea to the assumptions, practical checks and next steps.
To work through the connected decisions, read Incrementality tests: choose a commercially useful MDE and Paid search MMM: separate demand capture from demand creation.
Choose the intervention you actually want to change
A single campaign holdout, a reduced-frequency policy and a persistent programme holdout answer different questions. Specify which messages are withheld, how transactional messages are handled and whether customers can receive other marketing during the test.
Suppose a fictional eligible list has 20,000 customers split equally into treatment and control. Treatment produces £55,000 of revenue and control £50,000 over the chosen window. The simple difference is £5,000 for the treated half, subject to uncertainty and the design. It is not £55,000 of incremental revenue.
Measure substitution and longer-term effects
A promotional email can shift a purchase forward, change the product mix or replace an organic visit. Choose an outcome window that matches the decision and inspect relevant follow-up behaviour. Opens and clicks are delivery or engagement metrics, not a substitute for the commercial outcome.
Persistent holdouts can help assess a programme, but attrition, changing eligibility and other messages require careful handling. Preserve assignment and analyse according to the planned design rather than dropping customers whose behaviour became inconvenient.
Use the result to challenge aggregate estimates
An MMM may include email sends, delivered messages or another activity measure rather than meaningful media spend. A near-zero media cost does not imply infinite economic value: list acquisition, creative, discounts and operating costs may matter to the decision.
When calibrating, match the experimental policy, audience and outcome to the model's representation. The calibration guide explains why a triggered-email test should not automatically set the effect of all email activity.
Keep the outcome estimate, uncertainty, eligibility rules and delivery checks in the measurement record. If a model and holdout disagree, investigate scope and selection before deciding which number to favour. The measurement framework keeps these complementary evidence sources connected.
Sources and further practice
This article’s worked example is illustrative, not a measured client result. Use the linked methodology to check the assumptions before applying it.