“Challenges and Opportunities in Media Mix Modeling” is a 2017 Google paper by David Chan and Michael Perry that sets out why media mix models often fail to give valid answers about advertising effectiveness, and what could make them better. Its challenges fall into three groups (data limitations, selection bias and model uncertainty), and its opportunities are better data, Bayesian and hierarchical models, and testing models on simulated data where the true answer is known.
It’s a short paper, 16 pages, and it reads less like research than like a senior statistician’s notes to a client who has been told too much. I’ve built MMMs for brands from Monzo to Time Out, and almost every problem I’ve run into is on Chan and Perry’s list, usually stated more plainly than I would have dared in front of the client. This page goes through each challenge, then the opportunities, and the self-assessment in the middle lets you score your own model against the paper’s list and see which of our guides deals with each problem.
Why Google wrote Challenges and Opportunities in Media Mix Modeling
The paper starts from the questions advertisers actually ask an MMM: what was my ROAS on TV last year, what would sales be if I spent more or less, and how should I split the budget? Those are causal questions, and the authors are blunt that a regression on a limited amount of aggregated observational data produces correlational results, which can only be read as causal under certain narrow conditions. That framing sets the tone for everything that follows.
They explain why advertisers end up with regression anyway. Randomized experiments are the gold standard, but answering everything an MMM claims to answer (effects at several spend levels, an optimal split across channels) would take many experiments over a long time, with real costs in lost sales from control groups and in sample size when effects are small, a point they credit to Lewis and Rao. The other route, the potential outcomes framework with a matching estimator, needs observations with the channel off for every combination of the other variables, which is nearly impossible with a hundred or so weekly data points. So regression tends to win by default rather than on its merits.
Before the challenges, the authors list when regression MMM results are generally more trustworthy, and it’s the most useful checklist in the paper:
- There are enough data to estimate all the parameters in the model.
- There is useful variability in the advertising levels and control variables.
- Model inputs vary independently.
- The model accounts for all the important drivers that might impact sales.
- The model captures the causal relationship between variables.
Every challenge that follows is one of these conditions failing. If you read nothing else, this list is a fair test to hold any model up against, including one a vendor has already sold you.
Data limitations: too few points, moving together, in a narrow range
The first challenge is simply quantity. Three years of national weekly data is 156 data points, and the modeler is often asked for 20 or more channels, each needing three or four parameters to capture lag and diminishing returns, so the parameters can outnumber the observations. The authors cite a rule of thumb of seven to 10 data points per parameter for a stable linear regression, before even asking whether the effects are causal, and note that typical MMMs fall short. The Bayesian answer to this is priors, which is what Jin et al. (2017) develops, and our data requirements guide turns the arithmetic into a checklist.
The second is correlated inputs. Advertisers tend to raise and lower channels together, often for sensible reasons, and that can tie a channel to a season or to another marketing lever. The paper’s illustration is two fitted planes for two correlated channels that fit the observed data equally well but have very different slopes. Such models can forecast well while the spend pattern stays the same and fail as soon as the advertiser moves one channel on its own, and small changes in the data or in seemingly unrelated variables can swing the estimates. That’s the multicollinearity problem, and regularization only hides it; breaking the correlation with deliberate tests is the real fix.
The third is a limited range. Advertisers settle into a comfortable spend level, so the model only sees a narrow slice of each response curve, and questions like “what if I double my spend?” become extrapolation. The authors make a subtle point here that I think is underappreciated: average ROAS also requires extrapolation, all the way back to zero spend, so a model can give a reasonable marginal ROAS (from the data around current spend) and a poor average ROAS at the same time. If your report leads with average ROAS by channel, that’s worth knowing, and our marginal ROAS page shows the two on one curve.
Selection bias, the hardest of the challenges in media mix modeling
The authors call selection bias “perhaps the largest hurdle” to valid MMM estimates. It occurs when a media variable is correlated with unobserved demand that also drives sales; with that demand left out of the regression, the model can’t separate sales caused by the channel from sales that would have happened anyway. Their first case is ad targeting, which they describe as particularly acute in digital: retargeting ads go to people who already visited your site, and paid search ads go to people who just searched for something related, so both channels follow demand by design. The confounding variables page has a simulator for exactly this.
The second case is seasonality. If you sell cold medicine and spend more in flu season, spend and sales rise together whatever the ads do. MMMs try to control for this with proxies such as category sales, but the authors warn there’s no guarantee the proxy reflects the true underlying demand, and an inaccurate proxy biases the coefficients. The same goes for spend timed around events or fashions the modeler can’t measure, or in reaction to competitors the modeler has no data on.
The third case is funnel effects, and it’s the one I see broken most often. If TV drives searches, which in turn drive paid search clicks, then putting TV and paid search side by side in one regression credits search with sales that TV caused. The paper’s advice is that downstream ads shouldn’t sit in the same equation as upstream ones, with graphical or structural equation models as alternatives, although those need the effect of TV on search to be estimated too, which is just as hard. The authors extend the point to controls: if you support campaigns with discounts, price is partly a result of your advertising decisions, and including it as an ordinary control won’t account for that. There’s more on the logic in our guide to causal inference for marketers.
Five good models, five different budgets
The paper’s most memorable section is a real example. A US advertiser with several years of weekly data and about a dozen channels was modeled five ways: a baseline, one with a different seasonality proxy, one using only the last two years, one that dropped branded search, and one with a linear time trend. Each model had an R² of 0.98 or 0.99, and four of them had an out-of-sample MAPE of 6-8 percent on the last 12 weeks (the trend model did worse, at 15 percent, because the trend didn’t hold).
Despite fitting equally well, the models disagreed on sales at different budget levels by up to 50 percent, and they disagreed on how to split the budget, including on the advertiser’s largest channel, which one model said should get the biggest share at every budget level while the others didn’t. The authors’ explanation is that MMM data has low signal-to-noise: a few controls such as seasonality, price and distribution already explain most of the variance in sales, so it’s easy to find models that predict well whatever they say about advertising. Their conclusion is that a focus on predictive accuracy won’t, in general, let a modeler rule out models that produce inaccurate ROAS.
This is the section to show anyone who equates a good fit with a right answer. The practical implication is that you should look at how much your conclusions move across plausible specifications, not just at the best one, which is what MMM validation and model diagnostics are for.
Check which of the paper’s challenges apply to your model
The ranking is a prompt, not a verdict. The data-quantity score uses the paper’s rule of thumb, but the rest reflect my judgment of how each situation maps to the paper’s challenges, and a single severe score (say, half your budget in paid search with no query volume control) can matter more than several moderate ones. If you’d rather practice on someone else’s mess first, the invented Northstar case from the course has messy inputs and a model review built around problems like these.
The opportunities: better data, better models, and simulation
The second half of the paper is more optimistic, and it doubles as a table of contents for Google’s MMM research over the following years. On data, the authors propose that the industry adopt a standardized data and reporting format for MMMs, with closer coordination between agencies and publishers, because ad exposure data often passes through several intermediaries and gets estimated differently by medium (print circulation as a proxy for readership being their example). They also note the “lowest common denominator” problem: the least granular channel sets the granularity of the whole model.
On models, they list the benefits of a Bayesian approach (informative priors, complicated models, honest parameter and model uncertainty, and uncertainty that carries through to the budget recommendation), pointing to Jin et al. for the details. They then describe two ways to get more independent variation into the data. Category models pool several brands in the same category, because different brands advertise independently, and can supply priors for a brand that wasn’t in the pool (Wang et al., 2017). Geo models use regional data, which increases the sample size, tends to reduce the variance of the estimates, and can mix geo-level and national-level channels in one hierarchical model (Sun et al., 2017).
For selection bias, they propose control variables that capture the demand a channel targets: search query volume for paid search (developed in Chen et al.’s bias correction paper), and site visits for retargeting. For funnel effects they suggest graphical models, which make the assumed causal structure explicit. Finally, they argue for evaluating models on simulated data from a tool such as Google’s Aggregate Marketing System Simulator, where the true ROAS is known, so you can see how a method performs as lag effects or channel correlation increase. They even suggest advertisers give vendors simulated data with a hidden answer as part of a request for proposals, which I think is still one of the best ideas in the whole MMM literature.
What the paper doesn’t cover, and what’s changed since 2017
The paper is about short-run measurement, and it says little about long-term or brand effects of advertising, which are often the questions marketers argue about most; for that you need a different literature, such as The Long and the Short of It. It also predates the privacy changes that pushed many digital advertisers back towards MMM, and it treats experiments mainly as the expensive gold standard rather than as an input to the model. The idea of calibrating an MMM with experiments, now standard in Robyn, Meridian and PyMC-Marketing, came later.
What’s striking is how much of the paper’s wish list Google went on to build. Google Meridian is a Bayesian model fitted on geo-level data with a hierarchy across regions, it lets you set priors (including on ROI, so experiment results can feed in), and it supports Google Query Volume as a control for organic demand. However, none of that removes the paper’s core point, which is that an MMM fit to observational data is only as causal as its assumptions, and that the modeler owes the end user an honest account of how uncertain the answer is and how much of it came from the data rather than the priors. If you want the rest of the research in order, the MMM research papers bibliography lists it, and the pillar guide to marketing mix modeling covers the basics this paper assumes.
