“The Unfavorable Economics of Measuring the Returns to Advertising” is a 2015 paper by Randall Lewis and Justin Rao, published in the Quarterly Journal of Economics, which shows that even very large randomized experiments struggle to measure whether advertising pays for itself. The reason is a signal-to-noise problem: what individual customers spend varies so much from week to week that the small sales lift a profitable campaign needs gets lost in the noise, so the confidence interval on ROI from a test with a million people can still be wider than 100 percentage points.
I think it’s the most important paper a marketer who cares about measurement can read, because it explains something most of us have felt without being able to put a number on. I’ve built marketing mix models for companies like Monzo Bank, Booking.com and The AA, and my agency spent over $10 million a year on Facebook ads, and the recurring frustration in both kinds of work was the same: the estimate of what a channel returns is always far less certain than the dashboard makes it look. Lewis and Rao put that frustration on a formal footing with real data. I’ll walk through what they did and what they found, then give you a calculator so you can run their arithmetic on your own numbers.
What Lewis and Rao actually did
Lewis and Rao were both at Yahoo! when they ran the work (by the time the paper came out they were at Google and Microsoft Research respectively), which gave them something academics rarely get: access to randomized advertising experiments at scale, paired with the advertisers’ own purchase data. The paper analyzes 25 display advertising experiments run between 2007 and 2011, 19 for well-known US retailers and six for financial services firms (online brokerages), collectively representing $2.8 million in ad spend. In each one a random set of targeted users was held out from seeing the advertiser’s ads, so the comparison between test and control is clean in a way that observational data never is. Earlier drafts circulated under the blunter title “On the Near Impossibility of Measuring the Returns to Advertising”, which is why you’ll see both in citations.
The experiments were large by any normal standard. Every one had more than 500,000 unique users and most had over a million. Campaign costs ranged from $0.02 to $0.35 per exposed user, with most close to $0.10, which the authors equate to 20–60 premium display ads or roughly seven to 10 prime-time TV commercials per person. Just as important, the retailers shared individual-level sales, online and in-store, along with their gross margins, so Lewis and Rao could express results as a real return on investment rather than clicks or some other intermediate metric.
The headline result is in the abstract: the median confidence interval on ROI was over 100 percentage points wide. For the retail experiments the median standard error on ROI was 26.1 percent, and for the brokerages it was 115 percent. In plain terms, a typical test at a typical big advertiser couldn’t tell you whether the campaign lost money, broke even or was wildly profitable.
Why sales noise swamps the ad effect
The source of the problem is visible in the retailers’ sales data. For an individual customer, sales over a campaign window combine whether they buy at all, how much they spend when they do, and how often, and all three vary enormously from person to person. Lewis and Rao found that the standard deviation of sales per person was typically around 10 times the mean (the coefficient of variation ranged from four to 23 for retailers but clustered between 10 and 15), and the median across the retail experiments was 9.9. Most people buy nothing in a given fortnight, and the ones who do buy spend wildly different amounts.
Now set that against what a campaign needs to achieve. The paper’s representative campaign costs $0.14 per exposed user, the retailer’s gross margin is 50 percent, and mean sales per person during the campaign are $7 with a standard deviation of $75. To earn a 25 percent ROI, the campaign has to cause an extra $0.35 of sales per person, because $0.35 of sales at 50 percent margin is $0.175 of gross profit, which is 1.25 times the $0.14 it cost. That $0.35 is a 5 percent lift on sales, which sounds like plenty, but the estimation challenge is to detect it against a $75 standard deviation. The ratio of the effect to the noise is 0.0047, which the authors point out is less than a fortieth of what statisticians conventionally call a “small” effect.
Another way to see it is through the R² (the share of the variation in sales that the ad explains). In a regression of sales on an indicator for being in the test group, the R² is a quarter of the squared effect-to-noise ratio, which for this campaign comes out at about 0.0000054. A very successful campaign explains roughly five millionths of the variation in what people spend. I find that number clarifying, because it tells you that no amount of clever modeling will rescue a test that is simply too small to see an effect this faint.
How big an ad experiment has to be
With an effect that small, the sample has to be enormous. Lewis and Rao work through the representative campaign: with 2 million users split evenly between test and control and a true ROI of 25 percent, the expected t-statistic against the hypothesis that the ads did nothing at all is 3.30, which gives about 95 percent power for a one-sided 5 percent test. Cut the experiment to 200,000 users and the expected t-statistic drops to 1.04, so even though the campaign is genuinely profitable, you’d fail to reject “no effect” about 74 percent of the time.
And “did the ads do anything?” is the easy question. A budget decision needs to know whether the return beats break-even or some hurdle rate, which means distinguishing one positive ROI from another, and that’s far harder because the gap between, say, a 10 percent and a 0 percent ROI is a much smaller difference in sales per person than the gap between 25 percent and nothing. The paper computes how much larger each experiment would need to be to reach an expected t-statistic of 3 (about 91 percent power). The median campaign would have to be nine times larger to reliably tell a wildly profitable +50 percent ROI from break-even, and 62 times larger to resolve a 10 point ROI difference, which is closer to the precision you’d want for an investment decision. The abstract sums it up: informative advertising experiments can easily need more than 10 million person-weeks.
The calculator below does the same arithmetic for your own numbers. Put in the mean and standard deviation of sales per customer over the window you’d evaluate (include the people who bought nothing, as zeros), what the campaign costs per person reached, and your gross margin, and it tells you how many users per arm you’d need to see the ROI at all, and how many to beat a threshold.
Try the defaults first, because they reproduce the paper’s representative campaign, then change the threshold from 0 percent to the ROI your finance team would actually want. With the defaults, telling a 25 percent ROI apart from break-even at 80 percent power takes over 14 million users per arm, against about 570,000 to tell it apart from zero. Then try raising the cost per user, which is the one lever an advertiser fully controls: a more concentrated test is easier to read, which is exactly the trade-off the paper discusses (more spend per person makes the effect bigger relative to the noise, but diminishing returns mean the ROI you measure at that intensity may be worse than the one you’d get at normal weight).
For a broader treatment of power, minimum detectable effects and conversion-rate tests, the guide to statistical power for marketing tests goes further, and the statistical significance calculator is handy for reading a finished test.
Why the same noise breaks observational methods
The part of the paper I think gets less attention than it deserves is what the tiny R² implies for observational methods, such as regression with controls, difference-in-differences or propensity score matching on non-experimental data. If the true ad effect explains five millionths of the variance in sales, then an omitted variable or a slightly wrong functional form that explains even 0.0001 of it (a tiny amount by any normal standard) is an order of magnitude larger than the thing you’re trying to measure. The authors compare this with the classic wage and schooling regressions in economics, where endogeneity is often found to be 10–30 percent of the treatment effect. In advertising, it can be many multiples of it.
Targeting makes this worse, because ads are not delivered at random by design. The paper’s footnoted example is a campaign spending 10 cents per person on a product with $30 of marginal profit, which breaks even if it converts one extra person in 300. If the targeted users have a baseline purchase probability just 10 percentage points higher than everyone else (a realistic degree of targeting, according to the authors), the selection effect is about 30 times larger than the causal effect of the ad. Comparing people who saw the ad with people who didn’t measures mostly who was targeted, not what the ad did. The paper cites an industry study reporting a 300 percent effect of online advertising from exactly this kind of exposed-versus-unexposed comparison, an estimate that would imply ad prices should be at least an order of magnitude higher than they are. The later Facebook experiments by Gordon, Zettelmeyer and colleagues tested this directly against randomized controls, and the observational methods mostly overstated lift, sometimes by a factor of three or more.
The caveats Lewis and Rao were careful to state
The authors are explicit that the results don’t apply to everyone. They’re unlikely to hold for firms or products with low baseline sales volatility, for businesses that get nearly all their customers through advertising (a company that lives on sponsored search, say), for products with little existing awareness, or for campaigns with much higher spend per person. They also argue that their firms are representative, in revenue base, margins and product types, of the large advertisers that account for most ad spending, so the conclusion applies to most advertising dollars even if it doesn’t apply to every advertiser.
A few other details from the paper are worth knowing because they cut against common intuitions. Lengthening the evaluation window usually doesn’t help: in related work the authors show that adding another week to the analysis only improves the test if that week’s expected effect is more than half the average effect of the weeks before it, so with any normal decay a long window adds more noise than signal. Controlling for pre-campaign sales helps, and they used it, but it doesn’t change the qualitative picture. And when a low-powered test does come back significant, the point estimate is likely to be exaggerated, because only the lucky draws clear the bar (Gelman and Carlin’s “exaggeration factor”), which is one reason the authors chose not to publish the individual point estimates at all.
They close on the implications for the market. If reliable feedback is scarce, there’s little competitive pressure on how much firms advertise, and they find otherwise similar firms in the same industry differing in ad spend by up to an order of magnitude. Noisy estimates also invite strategic reporting inside a company, since a media buyer can share the good results and quietly bury the bad ones. And the scale of experiment required gives the largest publishers an advantage, because only they can supply tens of millions of users per cell. The authors contrast advertising with other investments whose returns are hard to measure, such as management consulting, and note that what makes advertising different is “a veneer of quantitative certitude” that hides the uncertainty underneath.
What the unfavorable economics of measuring the returns to advertising mean for marketers
The first lesson is about experiments. Before you run a lift test, work out whether it can possibly answer your question, because a test that can’t distinguish a 10 percent ROI from a 0 percent one won’t settle a budget argument however long you stare at it. Decide the threshold that matters for the decision, use the calculator to see what sample that implies, and if the answer is out of reach, change the design rather than running it anyway: concentrate the spend, test a bigger change (turning a channel off, or doubling it), pool several campaigns, or accept that the test will only answer “did it do anything?”. The guide to incrementality testing walks through matching a design to a decision, and conversion lift studies covers the extra noise you add when many people in the test group never see an ad.
The second lesson is about geo experiments, which are how most offline and many online channels get tested. Lewis and Rao note that geo-randomized experiments are the state of the art for media that can’t be randomized per person, and that they’re more expensive because they can’t strip out the noise from everyone in a region the advertiser never reached. You’re also working with a few dozen regions rather than millions of users. That doesn’t make geo lift testing a bad idea, but it means the same power calculation needs doing at the region level, with regional sales volatility, before you commit a quarter’s budget to it.
The third lesson is about marketing mix modeling, and this part is my interpretation rather than something the paper tests. An MMM works with aggregated weekly or daily data instead of individual users, which averages away a lot of the person-level noise, but the underlying problem doesn’t disappear: a channel’s spend only moves within a limited range, its effect is small relative to everything else driving sales, and it’s often correlated with seasonality and other channels. That’s why a single model’s channel ROIs can have wide intervals even when the overall fit looks great, why the Bayesian approach leans so heavily on priors, and why calibrating your model with experiments is worth the effort. Lewis and Rao make a related point in a footnote about using an evolving prior across campaigns: because any single campaign’s signal is so weak, a Bayesian update on it mostly returns the prior, though it’s a promising way to learn the average over many campaigns. In practice, I’d rather report an MMM result as a range and say which decisions it can and can’t support than pretend to a precision the data can’t deliver.
None of this is a reason to stop measuring. Lewis and Rao themselves argue that randomized trials are real progress, because they inject unbiased information into a market that has very little of it, and they expose how biased the alternatives can be. The point is to measure with your eyes open: know how much a test can tell you before you run it, combine evidence from experiments and models rather than trusting any single number, and be honest about the uncertainty when you present the result. If you want to see how this paper fits alongside the other work that shaped the field, the annotated list of MMM research papers puts it in context, and the Vexpower marketing mix modeling course practises reading uncertain results on its invented Northstar brand.
