“A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook” is a 2019 Marketing Science paper by Brett Gordon, Florian Zettelmeyer, Neha Bhargava and Dan Chapsky that tested whether the observational methods advertisers commonly use could reproduce the results of randomized experiments. Using 15 large lift studies at Facebook, they found that the observational methods often failed, usually overstating the effect of the ads, and in half of the studies the estimated percentage increase in purchases was off by a factor of three across every method they tried.
This one matters to me personally, because the agency I co-founded spent over $10 million a year on Facebook ads, and most of the numbers advertisers optimize against day to day come from the platform’s own reporting rather than from experiments. A 2023 follow-up by Gordon, Robert Moakler and Zettelmeyer asked the obvious next question (if old-fashioned matching doesn’t work, does modern machine learning with thousands of features?) and the answer was still mostly no. I’ll cover what each paper did, why selection on exposure inflates lift, and what it means for how you measure, with a simulator you can use to watch the bias appear.
How the comparison of approaches to advertising measurement was set up
The authors (Gordon and Zettelmeyer at Northwestern’s Kellogg School of Management, Bhargava and Chapsky at Facebook) used 15 US advertising campaigns that Facebook ran as randomized controlled trials in 2015, comprising around 500 million user-experiment observations and 1.6 billion ad impressions. In each study, users in the advertiser’s target audience were randomly split into a test group, who could be shown the ads, and a control group, who could not. Because the split is random, the difference in conversions between the two groups is an unbiased estimate of what the ads caused. An earlier white-paper version of the work, from 2016, used 12 of these studies.
The clever part of the design is what they did next. They threw away the control group and analyzed only the test group, which is exactly the situation an advertiser is in when there’s no experiment: some people in the target audience saw the ad, some didn’t, and you have to infer the ad’s effect by comparing them. They then applied a range of standard program-evaluation methods to that test-group data: a naive comparison of exposed and unexposed users, exact matching on age and gender, stratification, propensity score matching, regression adjustment, inverse-probability-weighted regression adjustment, and a combination of stratification and regression. Each method was run with progressively richer sets of variables, from basic demographics up to census data, user activity and a composite of Facebook’s own user data. Then they compared every observational answer with the experimental one.
Study 4: a 73 percent lift that looked like 316 percent
The paper walks through one campaign in detail, which it calls study 4, and it’s worth following because it shows the mechanism clearly. An omnichannel retailer ran a two-week campaign in the first half of 2015, with 25.5 million users split 70/30 between test and control, and the outcome was an online purchase tracked by a pixel on the checkout confirmation page. The conversion rate was 0.045 percent in the test group and 0.033 percent in the control group. Only 37 percent of the test group was actually exposed to an ad, so after scaling the intent-to-treat difference up to the people who saw the ads, the experiment measured a lift of about 73 percent on the exposed (with a 95 percent confidence interval of 49 to 103 percent).
Now ignore the control group and compare, within the test group, the people who saw the ad with those who didn’t. Exposed users converted at 0.061 percent and unexposed users at 0.019 percent, which implies a lift of 316 percent, more than four times the true effect. The exposed and unexposed users weren’t comparable in the first place: exposed users were more likely to be female (96 percent against 88 percent), older, more likely to be married and more likely to use Facebook on mobile. Exact matching on age and gender brought the estimate down to 222 percent, which is still three times the truth. Only the richest variable sets got close to the experimental answer for this study, and regression adjustment with the richest set actually undershot it.
Study 4 is, if anything, one of the kinder cases. In study 9 the experiment measured a lift of 2.4 percent, and the closest observational estimate was 1,306 percent; the authors suspect the unusually small exposure rate (only 6.6 percent of the test group saw an ad) gave the targeting plenty of room to find the people most likely to buy anyway. And in study 15 the pattern flipped, with almost every method underestimating the lift, which is its own kind of warning: you can’t simply apply a discount factor to observational estimates, because you don’t know in advance which direction they’re wrong in.
What the 15 studies showed overall
Across all 15 studies, the observational methods mostly overestimated the lift measured by the experiments, although in some cases they significantly underestimated it. For the 14 studies with a checkout conversion outcome, the point estimates in seven were consistently off by more than a factor of three. The methods did better on registration and page-view outcomes than on purchases, which the authors explain by the nature of those outcomes: an unexposed user is unlikely to stumble onto a specific landing page on their own, so the unexposed group behaves much like a true control. Purchases are different, because people who never saw the ad still buy from the advertiser in the normal course of things, and those are exactly the people the targeting system was steering the ads towards.
Two other findings stuck with me. First, no method consistently beat the others: an approach that worked well in one study did badly in another, so there was no way to tell ahead of time which observational estimate to trust. Second, more data helped, but less than you’d hope, and in a sensitivity analysis the authors estimated that in more than half the cases, eliminating the bias would need additional data as strong as one or two of their better variable sets, which is unlikely to be easy to obtain even for people inside the platform.
Why selection on exposure inflates the estimates
The underlying problem is that ads are not delivered at random, and the forces that decide who sees an ad are correlated with who was going to buy. The white-paper version of the study describes three mechanisms. The first is the auction: an advertiser wins an impression by outbidding others, advertisers bid more for people who look more likely to convert, and the platform also favors showing ads people are likely to respond to. Selection can even come from someone else’s campaign (in study 4, exposed users may have skewed female partly because another advertiser was bidding aggressively for men). The second is delivery optimization: a campaign set to optimize for purchases learns which users convert and steers impressions towards them, so the exposed group becomes progressively more concentrated among likely buyers. The third is activity bias: to see an ad you have to be online during the campaign, and people who are online more are also more likely to complete online conversions, something Lewis, Rao and Reiley documented in earlier work.
Put in causal terms, purchase propensity is a confounder: it causes both exposure and purchase, so the raw comparison between exposed and unexposed users mixes the ad’s effect with the difference in who those people were. Matching and regression try to block that path by conditioning on observed variables, which works only to the extent that the observed variables capture the propensity. The randomized control group works differently: it doesn’t try to model the selection at all, because the control group contains the same mix of high- and low-propensity people the ads would have reached. If you want a refresher on the general idea, the guides to causal inference for marketers and confounding variables in MMM cover it with marketing examples.
The simulator below makes this concrete. Each simulated user has a latent purchase propensity, the ad lifts every exposed user’s conversion probability by the same percentage, and the targeting slider controls how strongly delivery favors high-propensity users. The last slider sets how much of that propensity is captured by a variable you can observe and match on.
With targeting at zero, delivery is effectively random and all three methods agree. As you increase the targeting, the exposed-versus-unexposed comparison climbs steeply even though the true lift never changes, and matching recovers only the share of the bias that the observable explains. Push the observable share to 100 percent and matching gets close; drop it to zero and matching is no better than the naive comparison. The RCT doesn’t move, because randomization doesn’t care how the ads were targeted. Try setting the true lift to zero as well, which is the most sobering setting: the observational methods happily report large positive lifts for an ad that does nothing.
Does machine learning close the gap? The 2023 follow-up
The obvious objection to the 2019 paper is that 15 studies is a small sample and that the methods were traditional ones. Gordon, Moakler (at Meta) and Zettelmeyer answered both in “Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement”, published in Marketing Science in 2023. They analyzed 663 experiments run on Facebook between November 2019 and March 2020, chosen to be representative of the large-scale experiments advertisers run on the platform in the US, which together measured 1,673 conversion outcomes across about 7.9 billion user-experiment observations and over 38 billion impressions. The median experiment ran for 30 days with about 7.3 million users.
This time the observational methods were state of the art: double/debiased machine learning (DML, a technique that uses flexible machine learning models for both the outcome and the exposure while correcting for the bias those models introduce), alongside the stratified propensity score matching (SPSM) that performed best in the 2019 paper, both implemented with deep learning. And they gave the models more than 5,000 user- and experiment-level features, including descriptive user data, interest features, the platform’s own estimated action rates (which feed into who wins ad auctions) and lagged conversion activity. That is richer data than most advertisers or their measurement partners could ever access.
DML did better than SPSM, but neither did well. The median experimental lifts were 29 percent, 18 percent and 5 percent for upper, middle and lower funnel outcomes respectively. The median DML estimates were 83, 58 and 24 percent, and the median SPSM estimates were 173, 176 and 64 percent. For the lower funnel outcomes most advertisers care about (purchases), the better method was still nearly five times the experimental median. The methods performed comparatively better for prospecting than for remarketing campaigns, when baseline conversion rates were lower, when experiments had more users and when a smaller share of the test group was exposed, which all point to the same thing: the observational methods work best when there’s a large pool of genuinely comparable unexposed people to learn from, and remarketing, by design, targets the people most likely to buy anyway.
The authors’ diagnosis is that this is more of a “data problem” than a “model problem.” To undo the selection you’d need to model whether the ad entered each auction, won it and was actually shown, which requires auction-level data that platforms generally don’t log or keep for measurement, and that would be enormously expensive to store at Facebook’s scale. Their conclusion is that randomized experiments will remain necessary to measure ad effects accurately. Two smaller details from the paper are worth knowing too: across all the experiments the average lift was 52 percent but the median was only 9 percent, and a quarter of the experiments didn’t have even 50 percent power to detect a 10 percent lift, which echoes the signal-to-noise problem Lewis and Rao described.
What these papers mean for how you measure advertising
The most direct lesson is about reported numbers. Any metric built on comparing people who saw an ad with people who didn’t, or on crediting conversions to the ads people saw beforehand, is exposed to exactly this bias, and the targeting systems that make modern platforms effective also make their observational measurement optimistic. That includes view-through attribution and most forms of multi-touch attribution, which only ever observe the journeys of people who were exposed. In my experience the gap is largest for retargeting and brand search, where the audience is already close to buying, which is consistent with the remarketing result in the 2023 paper.
The second lesson is to use the randomized tools when the decision matters. Platform conversion lift studies are the same kind of experiment these papers used as ground truth, and they cost far less than making a budget decision on a biased number. Where user-level randomization isn’t possible, randomizing at the level of regions with a geo lift test gives you a control group without relying on the platform. The guide to incrementality covers how to turn a lift result into incremental ROAS.
The third lesson is for marketing mix modeling, which is also an observational method, although its selection problem looks different. An MMM uses variation in spend over time (and sometimes across regions) rather than individual exposure, so the risk isn’t auction targeting so much as timing: budgets that rise in peak season, campaigns launched alongside promotions, spend that follows last month’s sales. If those drivers of demand aren’t in the model, the channel gets credit for them in the same way exposed users got credit for their own higher propensity. That’s why controls and seasonality terms matter so much, and why calibrating an MMM with experiments is the most reliable way I know to keep its channel estimates honest.
None of this means observational data is useless; it’s what you have most of the time, and a well-built model with good controls is far better than a dashboard. It means the experiment is the reference point, and everything else should be checked against one whenever the stakes justify it. For the wider body of research these two papers sit in, the MMM research papers page gives an annotated list, and the Vexpower marketing mix modeling course practises combining model and experiment evidence on its invented Northstar case.
