Incrementality testing is running a controlled experiment to find out how much of an outcome your marketing caused: you split people, regions or time periods into a group that gets the marketing and a group that doesn’t, and the difference in conversions between them is the incremental lift. The simplest version is a holdout test, where a random 10 or 20 percent of the audience is kept away from a campaign so you can see what they do without it. Everything else, from geo tests to switchbacks to ghost ads, is a variation on that idea for situations where a simple random holdout isn’t possible.
In my experience of running a growth agency that spent over $10 million a year on Facebook ads, the hardest part of testing is rarely the statistics; it’s agreeing, before the test starts, what you’ll do with the answer. A test that ends in a shrug (“interesting, let’s keep an eye on it”) is usually a test that was designed around a channel rather than a decision. That’s why the planner on this page starts with the decision rather than the channel. If you’re not yet sure what incrementality means or why platform ROAS overstates it, start with the guide to incrementality.
The types of incrementality test
Every incrementality test needs a comparison group that’s a fair stand-in for what the treated group would have done without the marketing. The test types differ in what gets randomized (people, places or time) and how the control group’s experience is handled. The choice mostly comes down to what you can control and where the sales you care about happen.
User-level holdouts and conversion lift studies
In a user-level holdout, individual people are randomly assigned to test or control before the campaign runs, and the control group is prevented from seeing the ads. When you do this with your own channels, such as email, SMS or direct mail, it’s as simple as holding back a random slice of the list. When it’s done inside an ad platform it’s usually called a conversion lift study: Meta offers self-serve lift tests in Ads Manager for advertisers who meet its criteria, and Google Ads offers Conversion Lift based on users for some campaign types, with others available through an account representative. User-level tests give you the largest effective sample for the money, but they only see conversions the platform can track, and they’re designed and graded by the platform being measured. I cover these in detail in the guide to conversion lift studies.
Geo tests (matched markets)
In a geo test the unit of randomization is a region (a city, a DMA, a state or a postcode area). You change spend in the test regions, keep the control regions at business as usual, and compare total sales. Because every sale in a region counts, wherever it was made, geo tests capture retail, phone and cross-device conversions that a pixel can’t see, and they work for channels that can’t target individuals at all, like TV, radio and out-of-home. The trade-off is noise: you have tens or hundreds of regions rather than millions of people, so you need careful matching (or a synthetic control) and longer tests. Google notes that its geo-based Conversion Lift tends to need higher budgets than its user-based version for the same reason. The geo lift testing guide goes through the design step by step.
Time-based on/off tests
The bluntest test is to switch a channel off nationally for a few weeks and see what happens to sales. It’s tempting because it needs no targeting at all, but it’s a quasi-experiment rather than a true one: anything else that changes during those weeks (a competitor’s promotion, the weather, a news story) gets mixed into your estimate. On/off tests are most useful when you read them against a credible counterfactual forecast, such as your marketing mix model, and when the dark period is long enough for carryover to wash out. They also have a side benefit: a planned dark period gives your next MMM variation in spend it has never seen, which helps it separate that channel from everything else.
Switchback tests
A switchback test is an on/off test with the schedule randomized in short blocks: ads on for a randomly chosen set of days or half-weeks, off for the rest, with weekdays balanced across both. Randomizing the schedule protects you from most of the problems of a single dark period, because a one-off event is as likely to land in an “on” block as an “off” block. Switchbacks work best for channels whose effect arrives and fades quickly, like brand search or promotions, and poorly for channels with long carryover, where this week’s sales depend on last week’s ads and the blocks contaminate each other.
PSA tests and ghost ads
There’s a subtlety in user-level tests that matters more than it sounds. If you simply don’t show ads to the control group, you can’t tell which control users would have been reached by the campaign, so you have to compare everyone assigned to test with everyone assigned to control (an intent-to-treat comparison), which dilutes the effect with people who were never going to see an ad. The old fix was a PSA test, where the control group sees an unrelated public service announcement in the same slot, so you can compare people who actually saw something. However, you pay for the PSA impressions, and the delivery algorithm may show the PSA to different people than it would have shown your ad. Johnson, Lewis and Nubbemeyer’s ghost ads method, published in the Journal of Marketing Research in 2017, instead logs the control users who would have been shown your ad, without showing it, which the authors found reduces the cost of experimentation and improves precision relative to PSA and intent-to-treat designs. The American Marketing Association’s 2022 award citation for the paper notes that several ad platforms now offer ghost-ad measurement; the dilution problem itself is worked through with a calculator in the conversion lift studies guide.
Choosing an incrementality test for your decision
The test types above are tools; the starting point is the decision. A test is valuable when the evidence it can credibly produce could improve a consequential decision, which means the most uncertain channel isn’t automatically the most useful place to experiment. Before choosing a design, it’s worth writing down what action is currently being considered, which plausible results would lead to different actions, and whether a feasible experiment can distinguish those results with useful precision in time for the decision.
As a rough guide, if you can control who sees the ads and the sales happen where the platform can track them, a user-level holdout is the most precise option. If sales happen offline or you can only vary spend by region, use a geo test. If you can only switch national spend on and off, use a randomized switchback for fast-acting channels and a planned dark period read against your MMM for everything else. The type of question matters as well: “should we keep this channel?” is an on/off question, while “should we spend more?” is a marginal question, and an on/off test tells you about the average dollar rather than the next one (see marginal ROAS for why those differ).
The planner below encodes these rules of thumb. Pick the decision and your constraints, and it suggests a design, the steps to run it, and a minimum duration based on how many conversions you get and the smallest lift worth detecting.
The duration it shows is a floor, not a forecast. It assumes the only noise is the randomness in conversion counts, which is roughly true for a large user-level holdout and very optimistic for a geo or time-based test, where week-to-week and region-to-region variation dominate. For those, run a proper power analysis on your own history using the statistical power guide. Notice also the “conversions forgone” figure: a holdout has a real cost if the ads work, and a larger control group buys precision with sales, which is one of the trade-offs to agree with whoever owns the budget before launch.
How to design a holdout test, step by step
The details vary by test type, but the sequence is the same. In my experience most tests that fail do so at the first two steps rather than in the analysis.
- Start with the decision and the action for each result. Write down what you’ll do if the lift is clearly profitable, clearly unprofitable, positive but uncertain, or not detected. If every result leads to the same action, don’t run the test.
- Define the estimand in one sentence. Specify the treatment, the comparison, the outcome, the population and the period: “the effect of Meta prospecting compared with no Meta prospecting on orders from our own database among UK adults over four weeks”.
- Choose the outcome from your own data. Use orders, revenue or new customers from your own systems where you can, not platform-reported conversions, because the whole point is to stop relying on the platform’s attribution.
- Size the test before you start. Decide the smallest lift worth detecting (the one that would change the decision), then calculate the control share and duration needed to detect it. Include a cool-down for conversions that arrive after the ads stop.
- Randomize, then check balance. Assign users or regions to groups before the test, by randomization or matching, and check that the groups looked similar in the pre-period.
- Freeze everything else. Agree which changes (budgets, bids, creative, audiences, promotions) won’t be made mid-test, and who has to approve an exception.
- Read the result with its interval, then record it. Report lift and iROAS with confidence intervals, apply the action you wrote down in step 1, and log the result somewhere your next model can find it.
Common pitfalls in incrementality tests
Experiments are the most credible evidence most marketers will ever get, which makes it easy to forget that they can be wrong in quite ordinary ways. These are the problems I’d check for first, roughly in order of how often they bite.
Contamination between test and control
A test only works if the control group really doesn’t get the treatment. In user-level tests, other campaigns (including ones run by an agency or a reseller) can reach the holdout. In geo tests, people commute, travel and see national media, and TV signals don’t respect DMA boundaries. Contamination usually biases the lift towards zero, so a contaminated test tends to say a channel works less well than it does.
Tests that are too short or too small
Lewis and Rao showed that the effect of advertising on any individual’s purchases is tiny compared with how much purchases vary anyway, so ad experiments need far larger samples than most people expect. An underpowered test that finds nothing is often read as “the channel doesn’t work”, when the honest reading is “the test couldn’t have told us either way”. Short tests also miss carryover: if the ads keep working for weeks after they run, a two-week test measures only part of the effect. Stopping early on a good day makes both problems worse.
Novelty effects and seasonality
A new channel or creative can perform unusually well in its first weeks simply because it’s new, and a test run in that window overstates the long-run effect. Seasonality matters in a different way: the effect of an ad in the week before Christmas isn’t the effect of the same ad in February, so a result from a peak period may not transfer to the rest of the year. Neither invalidates the test; both limit what it can be used for.
Low power and the winner’s curse
When many underpowered tests are run and only the “significant” ones are reported, the reported effects are systematically too large, because the tests that happened to land on a lucky high draw are the ones that crossed the threshold. If you’re running a test program, keep a register of every result, including the boring ones, and treat surprisingly large lifts from small tests with suspicion.
Reading incrementality test results
When the test ends, the first number most people look at is the relative lift, but the number that should drive the decision is the incremental ROAS, read with its interval. A test can show a clearly positive lift while its iROAS interval still straddles breakeven, which means the ads work but you can’t yet tell whether they pay for themselves. The calculator below takes the group sizes and conversions from any two-group test, along with spend and value, and gives you lift, a 95% confidence interval, a p-value and iROAS with its own interval.
With the default numbers the lift is about 15 percent and clearly different from zero, but the iROAS interval runs from roughly 0.34 to 1.58, so on revenue alone the campaign might lose money or might return more than half again what it cost. That’s a common and useful result: it tells you the channel is doing something, and it tells you how much more evidence you’d need to know whether to scale it. If the interval had included zero, the right summary would be “lift not detected”, not “no effect”, and the width of the interval would tell you which effects the test was able to rule out. Note that this calculator assumes the two groups are independent samples of people; for geo tests, use difference-in-differences or a synthetic control instead.
Designing the next test: choose experiments for their decision value
One test rarely settles a question, and the best measurement programs treat each result as the input to the next decision about what to test. The lesson in the Vexpower course that covers this, “design the next test”, puts it as a pair of warnings: an experiment with excellent statistical power can still answer the wrong question, and a highly relevant experiment can still be too noisy to change the decision. The skill is in finding the test that’s both relevant and precise enough, for a decision that’s coming up soon enough to matter.
In practice that means a few habits. Plan how the result will be used before you see it, including whether it will become a prior or calibration input for your marketing mix model, and avoid treating the same experiment as both prior information and independent confirmation (the guide to calibrating MMM with experiments covers how to combine the two without double-counting). Keep a register of every test with its estimand, population, dates and interval, so that later tests build on earlier ones rather than repeating them. And accept that learning can include discovering that a proposed large reallocation remains too uncertain to justify, or that a practical constraint, rather than the measured return, dominates the choice.
If you want to practise a geo test end to end, the old Vexpower simulator Can we prove the ROI of ads? walks through one, and the guide to incrementality testing tools compares the vendors and free packages that will run these designs for you. Whichever route you take, the test that’s worth running is the one whose answer you’ve already agreed to act on.
