MMM validation is the work of checking a marketing mix model against evidence it wasn’t fitted to: its error on held-out weeks, how stable its channel estimates are when the data changes slightly, and whether its channel effects agree with independent experiments. A high R² or a low MAPE on the training data is where validation starts, not where it ends, because a model can fit history closely while giving the wrong answer about which channel drove the sales.
I’ve reviewed a lot of marketing mix models over the years, my own for clients like Booking.com and Time Out included, and the most common failure I see is a good fit being treated as proof that the model is right. Every library reports fit statistics and they’re easy to put on a slide, so they get mistaken for a certificate of accuracy. This page walks through the validation checks that actually separate a useful model from a well-fitted one, with an overfitting demo you can play with and a calculator for your own model’s errors.
Training fit, holdout fit and causal validity are different questions
The lesson in the Vexpower course on this is called “Validate beyond the fitted line”, and its core idea is that three questions get collapsed into one:
| Question | Useful evidence |
|---|---|
| Does the model describe the outcomes it trained on? | In-sample fit (R², MAPE, NRMSE) and residual patterns |
| Does it describe outcomes it hasn’t seen? | Holdout error, time-series cross-validation |
| Does its channel story support a causal decision? | Experiments, stability, plausibility, sensitivity to assumptions |
You can pass the first and fail the second (overfitting), and you can pass both and still fail the third. Two channels that always run together can swap credit between them with almost no change in predictive accuracy, so the model predicts sales well using the wrong allocation of credit. That’s why no single number validates a marketing mix model, and why Google’s Meridian documentation says outright that out-of-sample prediction error shouldn’t be the primary way model fit is assessed, since the goal of MMM is causal inference rather than forecasting.
In-sample fit: R², MAPE and NRMSE
The three fit statistics you’ll meet most often all measure how far predictions are from actuals, scaled differently. R² (r squared, the coefficient of determination) is the share of variance in sales the model explains, one minus the sum of squared errors divided by the total sum of squares around the mean. MAPE (mean absolute percentage error) is the average of the absolute errors as a percentage of actual sales, which makes it easy to explain (“we’re off by 4 percent in a typical week”) and is covered in more depth in the MAPE brief. NRMSE (normalized root mean squared error) is the RMSE divided by the range of the outcome, and it’s the error Robyn minimizes:
All three have the same weakness in-sample: you can improve them by adding variables. Every extra control, dummy or knot can only lower the training error, which is the kitchen-sink model problem. Adjusted R² penalizes the parameter count a little, but the honest fix is to measure error on data the model didn’t see. MAPE has an extra quirk worth knowing: it divides by actual sales, so weeks with very low sales get enormous percentage errors, and a week with zero sales breaks it entirely. Meridian reports wMAPE (weighted MAPE, total absolute error divided by total actuals) alongside R² and MAPE, which sidesteps that problem.
Holdout validation and out-of-sample error
Holdout validation means setting some observations aside, fitting the model on the rest, and measuring error only on the ones it never saw. The gap between training error and holdout error tells you how much of the model’s fit is signal and how much is memorized noise. The Vexpower course uses a deliberately simple demo for this: 32 invented observations along an S-shaped response curve, with every fourth one held out, and a polynomial whose flexibility you control.
Drag the degree from 3 up to 14 and the training RMSE falls steadily, from about 6.6 to about 3.3, because every extra degree of freedom lets the curve pass closer to the training points. The holdout RMSE does the opposite: it sits between about 14.5 and 15.7 from degree 3 to degree 9, then climbs past 20 from degree 10, and reaches about 37.6 at degree 14, where the curve swings wildly between the points it was fitted to. A team choosing the model with the best fit would choose the worst one. In a real marketing mix model the equivalent of “more degrees” is more channels split out, more flexible adstock and saturation, more seasonality terms and more dummies, and the same trade-off applies (the bias-variance trade-off brief covers the theory).
Two cautions come with this demo. First, eight holdout points is a small sample: degrees 3 to 6 differ by a fraction of an RMSE unit, which is noise, not a ranking. Second, the holdout set is only an honest test while you don’t use it to make choices. If you try twenty model specifications and keep the one with the best holdout error, the holdout has become part of the fitting process, and you need another untouched set to report. The demo’s data are observations along a curve, not a time series, so they also skip the question of which weeks to hold out, which matters a lot for MMM.
Time-series cross-validation with a rolling origin
Holding out random weeks from a weekly sales series leaks information, because adstock and trend mean each week’s neighbors tell you a lot about it. The more common approach for time series is to hold out the end of the series, train on everything before it, and test on the weeks after. Rolling-origin cross-validation repeats that several times: train on weeks 1-60 and test on 61-72, then train on 1-72 and test on 73-84, and so on, so you get several out-of-sample errors instead of one, and you can see whether the model’s accuracy is stable or depends on one lucky period.
The libraries take different positions here, which is worth knowing before you compare their numbers. Robyn’s time-series validation splits the series into consecutive train, validation and test slices (with train_size = 0.7, the other two get 15 percent each), optimizes the NRMSE on the validation slice and reports the test-slice NRMSE as the out-of-sample check. Meridian’s guidance goes the other way on large contiguous blocks: its trend and seasonality use knots that need nearby data, so a long held-out block at the end leaves those knots driven by the prior and produces poor forecasts that say more about the time effects than the media effects. It recommends a roughly balanced holdout across geos and time periods instead. Neither is wrong, they’re answering slightly different questions, but it does mean a “holdout MAPE” from one tool isn’t comparable with one from another.
DECOMP.RSSD and Robyn’s business error
Robyn adds a second objective alongside NRMSE that isn’t about prediction at all. DECOMP.RSSD (decomposition root sum of squared distance) measures how far the model’s split of credit is from the split of spend, across the paid channels:
If search gets 50 percent of spend but the model gives it 80 percent of the media effect, that channel adds 0.09 to the sum inside the square root. Robyn’s code also multiplies the result by one plus the share of channels with zero effect by default, to penalize models that switch channels off. Robyn then shows you a Pareto front of models that trade off NRMSE against DECOMP.RSSD, and you choose among them (the Robyn page has a Pareto-front explorer).
I wrote about this metric for Recast under the title “Decomp RSSD: Optimizing for politics”, and that’s the honest way to read it. It rules out decompositions so extreme that nobody in the business would accept them, which has real value, because a model that says one channel did everything will be ignored however well it fits. However, it’s a plausibility constraint, not validation. It pulls every model toward the status quo spend split, so if your current budget is badly wrong, DECOMP.RSSD is quietly pushing the model to agree with the mistake. Use it to filter out absurd models, not as evidence the remaining ones are right.
Parameter stability and refresh stability
A model whose channel ROIs change dramatically when you drop a few weeks, change a random seed or add a month of new data isn’t telling you much about those channels. Stability checks are cheap and revealing. Refit on the data minus the last eight weeks, or minus a different quarter each time, and compare each channel’s ROI and adstock rate across the fits. Rerun a Bayesian model with a different seed and check the chains still agree. For Robyn, look at how much the ROIs vary across the models on the Pareto front: if equally good models disagree by a factor of three on a channel, the data haven’t identified that channel.
Refresh stability is the same idea over time. When you refresh a model monthly or quarterly, the new data should move estimates by roughly the amount the new information justifies. If TV’s ROI goes from 0.8 to 2.4 between two refreshes with nothing different in the TV plan, the change is coming from the model, not the market, and every budget decision made on the earlier number was built on sand. I’d track each channel’s estimate across refreshes on a simple line chart and treat large unexplained jumps as a bug to investigate, often a multicollinearity problem or a weakly informed prior in a Bayesian MMM.
Experiment agreement is the strongest validation you can get
Everything above checks the model against itself or its own training data. The only check against the truth is an experiment. If a geo-lift test or a conversion lift study measured a channel’s incremental return, the model’s estimate for that channel and period should be in the same neighborhood, with overlapping intervals. When they disagree, the disagreement is information: either the model is misattributing (often to a correlated channel or an omitted confounder), or the experiment measured a different quantity (a different period, region, budget level or counterfactual).
There’s one rule to keep this honest: an experiment can be used to calibrate the model or to validate it, but not both. If a lift test became a prior in Meridian or a calibration input in Robyn, the model agreeing with it later isn’t independent confirmation, it’s the model repeating what you told it. Keep an evidence register of which tests went in, and save at least one for checking. If you haven’t run any yet, the incrementality testing guide is the place to start.
Calculate MAPE and NRMSE for your own model
If you have a column of actual weekly sales and a column of model predictions (ideally for holdout weeks), paste them in to get the main error metrics side by side. The bias figure is worth more attention than it usually gets: a model that’s off by 5 percent at random is very different from one that’s 5 percent too high every week.
An MMM validation checklist to run before you trust the model
The course’s lesson ends with a question to ask at work: what was held out, what influenced model selection, and what evidence supports the causal interpretation beyond fit? Here’s that question turned into a checklist. If you want to see the residual side of validation, the companion page on MMM model diagnostics covers reading residual plots, autocorrelation and structural breaks, and the marketing mix modeling overview puts validation in the context of the full modeling process.
None of these checks proves a model is right; together they make it much harder for a wrong one to survive. The course works through this with the invented Northstar brand in its marketing mix modeling path, where a beautifully fitted line turns out to be the start of the review rather than the end of it.
