Incrementality and experiments

Design holdouts and geo tests, understand statistical power and use experiments to challenge attribution and calibrate MMM.

The question is not how many customers saw an advert before buying. It is how many purchases would disappear without that advert. Randomised holdouts make the counterfactual explicit; geo experiments can help when individual randomisation is impractical.

Begin with Incrementality and Incrementality testing. Then connect the analysis to the MMM course exercises, where you practise making a decision from model outputs.

Before acting on a recommendation, connect this topic to budget decisions: the result needs both credible measurement and a commercially worthwhile action.

A practical route through the topic

1 Define the counterfactual
2 Plan power and duration
3 Run the holdout
4 Calibrate the model

Imagine equally sized randomised groups with 1,100 purchases in treatment and 1,000 in control. The estimated increment is 100 purchases, or 10% relative lift. It is not 1,100 incremental purchases. Whether that gap is persuasive depends on uncertainty, assignment, contamination and the pre-specified analysis.

Try it before reading further

Change the inputs below and explain what changed. These teaching tools illustrate assumptions; they do not estimate your business’s return from the information on this page.

Spend ($)

Platform-attributed conversions

Value per conversion ($)

People in the test (exposed) group

Test group conversion rate (%)

Control group conversion rate (%)

Platform ROAS

2.00

Incremental ROAS (iROAS)

0.96

Incrementality factor

0.48

Incremental conversions

600

Cost per incremental conversion

$83

What to check before using the result

Do not stop a fixed-horizon test as soon as it looks positive. Account for spillover, seasonality and geo-level assignment. A test in one channel and one period does not identify the return of every future campaign.

Primary outcome, eligible population and assignment unit specified
Minimum worthwhile effect and test duration agreed in advance
Treatment and control exposure checked for contamination
Effect estimate, uncertainty and spend reported together

Your first reading path

For an overview, visit marketing mix modelling training. If you use AI to assist with analysis, the prompt engineering learning page explains the complementary skill.

Step 1 Incrementality Incrementality is the share of sales that would not have happened without your marketing. Learn incremental lift, iROAS, and why platform ROAS overstates. Step 2 Incrementality testing Incrementality testing compares a group that sees your marketing with one that doesn't. Pick a test type, design the holdout and read lift and iROAS honestly. Step 3 Geo lift testing How geo lift tests work: randomized vs matched markets, pre-periods, synthetic control and DiD, Meta's GeoLift R package, Google's methods and test design. Step 4 Statistical power Power analysis and sample size calculator for A/B and geo tests: statistical power, minimum detectable effect, and why a null result needs a precision story. Step 5 MMM calibration MMM calibration uses lift test results to constrain a marketing mix model. How Robyn, Meridian and PyMC-Marketing do it, and how to match a test to the model.

Foundations from Mike’s learning briefs

Statistical Tests For Linear Regression The test must reflect the assignment unit, outcome and analysis plan, not just the software available. Ground Truth An experiment provides a bounded estimate for a particular population and period, rather than universal ground truth. Geo Models Geographic models can support experiments when the comparison markets provide a credible counterfactual. Statistical Significance Calculator Plan a worthwhile detectable effect before testing; significance after the event does not establish commercial value.

Turn understanding into practice

Your next step: make a better marketing decision.

Work through realistic business problems with guided lessons, practical templates and simulations.

Choose your learning plan Explore the MMM course See the curriculum before choosing a plan.

Explore every guide in this topic

Incrementality Incrementality is the share of sales that would not have happened without your marketing. Learn incremental lift, iROAS, and why platform ROAS overstates. Incrementality testing Incrementality testing compares a group that sees your marketing with one that doesn't. Pick a test type, design the holdout and read lift and iROAS honestly. Lift studies A brand lift study surveys exposed and control groups; a conversion lift study counts their sales. How Meta and Google run them, ITT vs TOT, and their limits. MMM calibration MMM calibration uses lift test results to constrain a marketing mix model. How Robyn, Meridian and PyMC-Marketing do it, and how to match a test to the model. Geo lift testing How geo lift tests work: randomized vs matched markets, pre-periods, synthetic control and DiD, Meta's GeoLift R package, Google's methods and test design. Difference in differences Difference in differences for marketers: the formula, parallel trends, event-study plots, staggered rollouts, and a calculator for geo tests and launches. Synthetic control The synthetic control method explained: donor weights, pre-period fit, placebo tests, augmented synthetic control in GeoLift, and how CausalImpact compares. Statistical power Power analysis and sample size calculator for A/B and geo tests: statistical power, minimum detectable effect, and why a null result needs a precision story.

Apply it to your marketing

Geo experiments: audit spillover before interpreting lift Map cross-border exposure, customer movement and shared campaigns so a geo test's treatment contrast matches the effect you intend to estimate. Geo-lift market selection: review more than pre-test fit Choose geo-test markets using comparability, power, operational feasibility and contamination checks, rather than selecting the prettiest pre-period chart. Incremental ROAS: report the interval, not just the ratio Translate experimental lift into incremental ROAS, handle a confidence interval that crosses zero and connect the result to contribution-based decisions. Incrementality tests: choose a commercially useful MDE Translate minimum detectable effect into business value before sizing an incrementality test, and distinguish relative lift from percentage-point changes. MMM calibration: keep an experiment evidence register Create a traceable record linking each experimental result to model assumptions, scope, uncertainty and the budget decisions it can support. Email and MMM: use holdouts to separate selection from lift Measure email's incremental contribution without mistaking engaged subscriber behaviour for advertising impact, and transfer holdout evidence carefully. Advertorials: explain the offer without disguising the advert Create a useful advertorial with clear disclosure, supported claims and a landing-page test that measures qualified customers rather than clicks. Advertising research: choose the method that answers the decision Connect customer interviews, creative tests, experiments and MMM to the specific advertising decisions each method can support. Local store marketing: connect online discovery to store sales Build a practical local marketing plan around accurate store information, useful offers and measured changes in visits and purchases. Sports marketing: turn sponsorship into a measurable programme Plan sports sponsorship and activation around audience fit, rights, customer outcomes and the full cost of delivery. Guerrilla marketing: design a memorable, measurable campaign Plan a guerrilla marketing campaign with a clear audience, practical permissions, a response mechanism and an honest measurement plan.

Frequently asked questions

Is attributed revenue incremental revenue?

Not necessarily. Attribution allocates credit among observed touchpoints. Incrementality compares outcomes with a credible estimate of what would have happened without the intervention.

Can an inconclusive test still be useful?

Yes. Report the interval and which effects remain compatible with the result. An inconclusive result may rule out a very large benefit without distinguishing a modest benefit from no effect.

Connect the ideas

Marketing mix modelling Choose a measurement question, audit your data and learn how an MMM turns channel activity into a budget decision. How marketing mix models work Explore adstock, saturation, baselines and confounding with visual explanations and interactive model mechanics. Marketing budget decisions Compare marginal returns, contribution margins and bounded scenarios before moving marketing budget. MMM tools, code and platforms Choose between R, Python, spreadsheets and managed MMM platforms, with tutorials, datasets and validation checkpoints.