MMM model diagnostics are the checks you run on a fitted marketing mix model’s residuals (each week’s actual sales minus the model’s prediction) to find what the model is missing: patterns over time, autocorrelation, changing error size, structural breaks and outliers. If the model has captured everything systematic, the residuals look like random noise; if they don’t, the pattern usually points to a specific business mechanism, such as an unmodelled promotion or a new distribution deal, that is distorting the channel estimates.
In my experience the residual plot is the most underused chart in marketing mix modeling. Teams look at the fitted-versus-actual line, see it hugging the sales curve, read off an R² of 0.93 and move on to the ROI table. However, the fitted line is drawn at a scale where a week that’s 8 percent off looks like a near miss. Plot only the misses and the same model often shows you a rhythm or a step that’s obvious once you look. I’ll show you what each pattern usually means for the business, using a small interactive model where the residuals give away a missed promotion and a structural break.
Why an average error can hide a business mechanism
The Vexpower course has a lesson called “read the residuals”, and its point is simple: one summary metric can’t show every structure in the errors. Suppose a model underpredicts by $20,000 one week and overpredicts by $20,000 the next. The mean error is zero, so on that measure the model is perfect. The mean squared error is 400 (in squared thousands of dollars) and the MAPE might be a respectable few percent, which also sounds fine. None of these numbers tells you that the misses alternate, or that they line up with something happening in the business.
That matters because the residual is where a missing variable goes. If a promotion every six weeks isn’t in the model, its effect doesn’t disappear; it ends up in the residuals of those weeks, or worse, it gets absorbed by whichever included variable happens to move with it. If paid social always gets a bigger budget in promotion weeks, the model will happily credit social for the promotion’s sales, and the error metrics will barely flinch. Summary metrics like MAPE and NRMSE, covered on the model validation page, are worth computing; residual plots are how you find out why they are what they are.
MMM model diagnostics start with the residuals over time
The first and most useful diagnostic is the residuals plotted in time order, with a zero line and your business calendar beside it. The widget below has 104 weeks of invented sales with two mechanisms the starting model leaves out: a promotion every six weeks, and a permanent jump in sales from week 65 (think of a new retailer listing). Media spend is set about eight units higher in promotion weeks, which is the realistic part, because promotions and media bursts are usually planned together.
With neither term in the model, the promotion weeks (in magenta) sit consistently above zero, the residuals drift from mostly negative before week 65 to mostly positive after it, and the media coefficient comes out at nearly double its true value of 0.6, because media is the only variable that moves with the promotions. Add the promotion dummy (a variable equal to 1 in promotion weeks and 0 otherwise; the dummy variables brief covers them) and the six-week rhythm disappears and media drops to about 0.67, close to its true value. Oddly, the Durbin-Watson statistic gets worse at that point (from about 1.6 to about 1.3), because the promotion spikes were breaking up the long runs caused by the missing break; one problem was hiding another. Add the break as well and the residuals finally look like noise. Notice that the R² stays between about 0.8 and 0.94 in every version, so the fit statistic would never have told you the media estimate was wrong; the residual pattern did.
Autocorrelation: the ACF plot and the Durbin-Watson statistic
Autocorrelation is the correlation between the residuals and themselves shifted by some number of weeks (the lag). The ACF (autocorrelation function) plot shows it for each lag, with a band at roughly plus or minus 1.96 divided by the square root of the number of weeks; bars outside the band are larger than you’d expect from noise. In the widget, the omitted promotion shows up as a spike at lag 6 with a smaller echo at 12, because a residual six weeks ago predicts the residual today. Seasonal patterns you haven’t captured show up as spikes at 52, and an omitted level shift or slow trend shows up as a long run of positive bars at small lags.
The Durbin-Watson statistic summarizes the lag-one autocorrelation in a single number:
It runs from 0 to 4. A value close to 2 means consecutive residuals are roughly unrelated; values well below 2 mean positive autocorrelation (an error this week tends to be followed by an error in the same direction next week), and values above 2 mean the errors flip sign more than chance would suggest. A common rule of thumb treats roughly 1.5 to 2.5 as acceptable, though the formal test depends on the number of observations and variables. In marketing mix modeling, a low Durbin-Watson almost always means something persistent is missing: a structural break, a trend the model is too stiff to follow, a carryover effect that’s longer than your adstock allows, or a competitor that entered the market.
Autocorrelation isn’t only a cosmetic problem. When errors persist, each week carries less independent information than the model assumes, so standard errors and credible intervals come out too narrow. The model looks more certain about each channel’s ROI than the data justify, which is exactly the wrong way round for a budget decision.
Structural breaks: when the business changed underneath the model
A structural break is a point where the relationship between your inputs and sales changes, either the baseline level (as in the widget) or a channel’s effectiveness. In the residuals it looks like a run of negative values that turns into a run of positive values, or the reverse, and stays there. The usual causes are business events: a new retailer or marketplace, a price change, a product launch, a website migration, a competitor’s entry or exit, a change in how conversions are tracked, or an accounting change in what counts as revenue.
The fix depends on what changed. If the level shifted, a step dummy from the break date is often enough; if a channel’s effect changed (say, a tracking change altered what “conversions” means), you may need to split the period or model the channel separately before and after. A Chow test can formally compare a model fitted with and without a break at a known date. Whatever you do, don’t smooth the break away. The course’s lesson on aligning time and geography makes the same point about data: when a source changes its definition halfway through the history, flag the break and investigate it rather than silently interpolating it, because a hidden break gets misattributed to whatever else changed around the same time, often a media channel or the baseline.
Outliers and one-off events
Outliers are individual weeks with residuals far larger than the rest. Some are real one-off events: Black Friday, a viral moment, a stock-out, a site outage, a pandemic lockdown, a major news story. Others are data errors: a duplicated export, spend recorded in thousands instead of units, a week that closed on the wrong day. The first job is to tell them apart, because the remedies are opposite. A data error should be fixed at source (the data cleaning brief and the MMM data requirements page cover this). A real event should be modeled with a dummy for that week, so it doesn’t drag the channel estimates around.
What you shouldn’t do is delete inconvenient weeks or add a dummy for every large residual. Each dummy uses up a row of information and makes the fit look better while explaining nothing, and a model with 15 one-week dummies has quietly removed 15 weeks of evidence about your media. My rule of thumb is that every dummy needs a named business event and a date someone outside the analytics team could confirm.
Residuals versus fitted, and heteroskedasticity
The second chart worth making is residuals against fitted values (switch the widget’s view to “Vs fitted”). In a healthy model the points form an even horizontal band around zero. Two shapes are common in MMM. A funnel, where the spread of residuals grows with the fitted value, is heteroskedasticity: errors are bigger in big weeks, which often means effects are multiplicative rather than additive (a promotion adds 20 percent, not a fixed $20,000). A curve, where residuals are positive at both ends and negative in the middle, means the model’s functional form is wrong somewhere, often a missing saturation curve or a relationship that should be in logs.
The homoskedasticity brief explains the assumption in more depth. The usual fixes are modeling log sales (the log-linear and log-log model briefs cover both), adding the missing non-linearity, or using a likelihood that allows the error size to vary. Like autocorrelation, heteroskedasticity mostly damages your uncertainty estimates: the model’s intervals are too wide in quiet weeks and too narrow in peak weeks, which is when the biggest budget decisions get made.
What each residual pattern means for the business
Here’s the translation table I’d keep next to any residual plot. The first four rows come straight from the course’s lesson; the point of every row is that a pattern is a prompt for a question, not an automatic diagnosis.
| Pattern in the residuals | A question worth investigating |
|---|---|
| Errors rise during promotions | Is the promotion measured, and is its assumed relationship adequate? |
| Long runs above or below zero | Is there an unmodelled change, trend or temporal structure? |
| Larger errors at high volume | Does the error model reflect changing variability (multiplicative effects)? |
| One geography behaves differently | Are coverage, population or local conditions aligned? |
| Spikes at a regular lag in the ACF | Is there a recurring event (pay day, monthly promotion, email cadence) missing? |
| A step change that persists | Did distribution, pricing, tracking or a competitor change on that date? |
| Positive residuals after big media bursts | Is the adstock too short, so the tail of the effect lands in the residuals? |
| One extreme week | Is it a real event that needs a dummy, or a data error that needs fixing at source? |
Diagnose before repairing, and don’t allocate the residual by wish
The temptation with a bad residual plot is to make it go away, either by adding flexibility (more knots, a time-varying baseline, more dummies) until it’s flat, or by deciding what the unexplained sales “really” were. Both are mistakes. An unexplained increase isn’t automatically organic growth, brand equity or the contribution of a favorite channel that isn’t in the model; a residual says the prediction and the outcome differ, not why. And a more flexible model can absorb a missed promotion into its baseline just as easily as into a media channel, which improves the fit while making the interpretation less stable.
The course’s advice is to check source definitions and business events before increasing flexibility. In practice that means taking the plot to someone who knows the business calendar, asking what happened in the weeks that stand out, and writing down a falsifiable explanation: “the positive residuals from week 65 are the new retailer listing; if so, they should be concentrated in the regions where that retailer operates.” Then go and check. If the model is a Bayesian MMM, the same discipline applies to posterior predictive checks, which are residual analysis with uncertainty attached.
Residual analysis won’t tell you a model is causally right, any more than a good fit will, and the marketing mix modeling overview puts it alongside the other checks in a full workflow. But it’s the cheapest way to catch the problems that most often turn a plausible model into a misleading one, which is why it gets its own lesson in the Vexpower MMM course.
