MMM calibration: how to calibrate a marketing mix model with experiments

MMM calibration is using the results of incrementality experiments, such as conversion lift studies and geo tests, to constrain what a marketing mix model says about the channels that were tested. In practice that means either telling the model which of its many plausible answers agrees with the experiment (Robyn’s approach), turning the experiment into a prior on the channel’s return (Meridian’s), or adding the experiment as an extra observation the model has to explain (PyMC-Marketing’s). The aim in every case is the same: anchor a model built on correlations to evidence built on randomization.

I’ve built marketing mix models for companies including Monzo, Booking.com and The AA, and advised Google, Appsflyer and Recast on MMM, and the most common mistake I see with calibration is a judgment call rather than a technical slip: treating the experiment’s number as the truth and forcing the model to match it, without checking whether the experiment measured the same thing the model estimates. A precise answer to one question can still be uncertain evidence about another, and I’ll show you how to decide how much weight an experiment deserves with your own numbers.

Why calibrate a marketing mix model at all

A marketing mix model estimates each channel’s contribution from how sales moved with spend over time. The trouble is that channels tend to move together (budgets go up in the same quarters, campaigns launch at the same time), so the data often can’t tell the model whether a sales bump came from search, social or the TV burst that ran the same week. Many different combinations of channel effects fit the history almost equally well, and the model’s choice among them can depend on modeling decisions as much as on the data. The guides to multicollinearity in MMM and confounding variables go into why.

An experiment cuts through this for the channel it tested, because randomization breaks the link between that channel’s spend and everything else. If a geo test says social returned about $1.40 per dollar in the test regions, that’s evidence the model’s history alone couldn’t provide. Robyn’s documentation recommends calibrating with experiments whenever possible, and cites a whitepaper by Analytic Edge reporting that uncalibrated models differed from ground truth by 25 percent on average. Calibration also tends to help channels you didn’t test, indirectly: once the model can’t give social’s credit to search, search’s estimate has to come from search’s own variation.

Precise evidence can still answer a different question

The lesson on this in the Vexpower course starts with a scenario most analysts will recognize. Northstar, the course’s invented brand, has a starting belief about a channel’s return, and a new experiment produces a different estimate with a small standard error. It’s tempting to replace the model’s result with the experiment’s number. The first question to ask, though, is whether the two pieces of evidence concern the same quantity, because they very often don’t:

Each row is a way that an experiment can be perfectly valid and still not be a direct measurement of the model’s parameter. Meridian’s documentation makes the same point about the counterfactual: the MMM’s counterfactual is zero spend, while an experiment that cuts the budget rather than switching it off measures a marginal lift, which is a different estimand. On a channel with diminishing returns, the marginal return is lower than the average, so plugging a budget-cut test straight into the model’s ROI will understate the channel. A short test that misses carryover will understate it too, while a test run on the best-performing campaign will overstate the whole channel.

How Robyn calibrates: MAPE.LIFT and calibration_input

Robyn, Meta’s open-source MMM package in R, treats calibration as an extra objective in its model search. Normally Robyn’s evolutionary optimizer (Nevergrad) searches over adstock and saturation hyperparameters to minimize two errors at once: NRMSE (how well the model fits sales) and DECOMP.RSSD (how far the decomposition is from the spend share, a “business sense” penalty). When you provide experiment results, Robyn adds a third, MAPE.LIFT, the mean absolute percentage error between the effect each candidate model predicts for the test period and the effect the experiment measured. Models that disagree with your experiments are penalized in the same Pareto trade-off as models that fit badly.

You supply the experiments as a calibration_input data frame passed to robyn_inputs(). The columns, from Robyn’s demo script, are the channel (which must match a paid or organic media variable), liftStartDate and liftEndDate (inside the modeling window), liftAbs (the absolute incremental effect in the same metric as the dependent variable), spend (which should match the channel’s spend in your data for those dates to within about 10 percent), confidence (for a frequentist test, one minus the p-value), metric, and calibration_scope:

calibration_input <- data.frame(
  channel = c("facebook_S", "tv_S"),
  liftStartDate = as.Date(c("2025-05-01", "2025-01-01")),
  liftEndDate = as.Date(c("2025-06-10", "2025-03-01")),
  liftAbs = c(40000, 120000),        # incremental revenue, same metric as dep_var
  spend = c(12000, 86000),           # spend in the test, must match dt_input
  confidence = c(0.85, 0.80),        # e.g. 1 - p-value
  metric = c("revenue", "revenue"),
  calibration_scope = c("immediate", "immediate")
)
InputCollect <- robyn_inputs(InputCollect = InputCollect, calibration_input = calibration_input)

A few details in the demo’s guidance matter more than they look. Robyn currently takes a point estimate, not a distribution, so the experiment’s uncertainty only enters through how you choose and weight tests. The calibration_scope should be "immediate" for experiments, because a test can’t hold out the carryover from ads that ran before it started; the demo reserves "total" for calibrating against another MMM. The lift must match the spend in the variable: if a channel normally spends $100k a week and the test held out 70 percent, the lift you enter is for the $30k that kept running, not the $70k. And at the output stage, robyn_outputs() has a calibration_constraint argument, 0.1 by default, which means only the 10 percent of models with the lowest MAPE.LIFT go forward to Pareto selection. The demo also recommends Weibull PDF adstock when calibrating, because its extra flexibility gives the optimizer more room to match the test.

How Meridian calibrates: experiments as ROI priors

Google Meridian is a Bayesian model, so calibration happens through priors rather than through an extra objective. Meridian parameterizes each media channel by its ROI and lets you set a prior on that ROI directly, which is about as natural a place as there could be to put an experiment’s result. Meridian’s documentation describes calibration as using experiment results and other domain knowledge to set channel-specific priors, and provides two helpers in prior_distribution for building a LogNormal prior, one from a mean and standard deviation and one from a range:

from meridian.model import prior_distribution

# One entry per media channel, in the order of your media columns.
roi_prior = prior_distribution.lognormal_dist_from_mean_std(
    [1.4, 2.0],   # experiment iROAS for channel 1; existing belief for channel 2
    [0.6, 1.0],   # experiment SE widened for transfer; a vaguer SD for channel 2
)
prior = prior_distribution.PriorDistribution(roi_m=roi_prior)

The documentation is refreshingly clear that there’s no single formula for turning an experiment into a prior. It suggests using the standard deviation to express how much you trust the result to transfer (a tight prior says you’re confident, a wide one says you’re sceptical), plotting the prior to check its percentiles look sensible, and asking a list of relevance questions about recency, duration, media granularity, population, complexity and estimand before using a result. For automation, Meridian provides a CalibrationBuilder that translates experiments into ROI priors with adjustments for recency, duration (using adstock to scale for carryover the test missed) and spend scale, and integrates with its own GeoX experiment design. If the prior only applies to part of the modeling window, the roi_calibration argument in ModelSpec can restrict it, though the documentation recommends applying it to the full window where possible. Because ROI must be positive under a LogNormal, a test that found no effect or a negative one needs more thought than plugging it in.

How PyMC-Marketing calibrates: lift tests as observations of the saturation curve

PyMC-Marketing takes a third route. Instead of turning the experiment into a prior, it adds each test as an extra data point the model has to explain. You call add_lift_test_measurements on the model with a data frame where each row has the channel, x (the spend level the test started from), delta_x (the change in spend), delta_y (the measured change in sales) and sigma (its standard error), all in original units. The model then compares the lift its own saturation curve implies for moving from x to x + delta_x with the lift you measured. PyMC-Marketing’s notebook describes lift tests as additional observations of the saturation curve that constrain the parameters during inference.

import pandas as pd

df_lift_test = pd.DataFrame({
    "channel": ["social_spend"],
    "x": [1717.0],        # usual weekly spend before the test
    "delta_x": [-700.0],  # weekly spend removed in the test
    "delta_y": [-975.0],  # weekly change in revenue the test measured
    "sigma": [312.5],     # standard error of delta_y
})
mmm.add_lift_test_measurements(df_lift_test)

By default the likelihood is a Normal on the signed lift, so a test with a negative or zero estimate can be used honestly rather than forced positive (the notebook notes that an earlier Gamma default was changed, and offers a Student-t for heavier tails). The approach has a nice property for the collinearity problem described above: the notebook’s own example shows two channels whose spends are perfectly correlated, so the model can’t tell their saturation curves apart, and lift tests at different spend levels are what let it recover them. It also handles the counterfactual issue directly, because a budget-cut test is entered as a move along the curve rather than as an average ROI.

Matching the experiment to what the model estimates

Whichever library you use, the decision that matters most happens before any code runs: how much should this experiment be allowed to move this model? The calculator below is a simplified version of the “evidence compass” from the course. It combines a model estimate and an experiment estimate by weighting each by its precision (one over its standard error squared), which is what a Bayesian update does when both are normal. The checklist lets you say where the experiment and model measure different things, and each “no” inflates the experiment’s standard error by a transfer factor, so it gets less weight.

With the defaults, a model estimate of 2.4 with a wide standard error and an experiment of 1.4 with a tight one combine to about 1.5, and the channel’s interval narrows by roughly two-thirds. That’s the case for calibration in one picture. Now switch off “same counterfactual” and “long enough to capture carryover”: the experiment’s weight falls, the combined estimate moves back towards the model, and the warnings tell you why. Switch off “same KPI” and the calculator refuses to combine at all, because no amount of widening fixes an experiment that measured app installs being used to calibrate a revenue model. Inflating a standard error is an honest way to express doubt about transfer, but it doesn’t remove bias; if you know the experiment understates a channel with long carryover, the better fix is to adjust the estimate (Meridian’s CalibrationBuilder does this with adstock) or to run a longer test.

Don’t count the same evidence twice

The last trap is quieter. If an experiment already informed a prior, presenting the same result later as independent confirmation that “the model agrees with the test” exaggerates your confidence, because the model was told the answer. The same applies to Robyn: a calibrated model matching the experiment it was calibrated on is not validation. The fix is an evidence register, a simple table of every test with its estimand, dates, population, interval and where it entered the model (prior, calibration input, or held out for validation), so that each piece of evidence is used once and the held-out tests can still tell you whether the model is any good. The guide to MMM model validation covers what to do with those held-out tests.

• The experiment's KPI matches the model's dependent variable (or has been converted)
• The counterfactual is the same, or the test is entered as a marginal change along the curve
• The test window overlaps the model period, or extra uncertainty is added for recency
• The tested scope (campaign, regions, users) is noted, and uncertainty widened if it isn't the whole channel
• Carryover the test missed is either adjusted for or acknowledged
• Robyn: calibration_scope is immediate and spend matches the data within 10 percent
• The experiment is used once: as a prior, a calibration input, or validation, not two of these

If you’re still deciding which experiment to run, the guide to incrementality testing covers how to choose a test for its decision value, and the guide to conversion lift studies explains the intent-to-treat dilution you’ll need to account for before a platform’s lift result goes anywhere near your model. For the Bayesian side of priors in more depth, see Bayesian marketing mix modeling, and for the bigger picture of how models and experiments fit together in a measurement program, the guide to marketing mix modeling is the place to start.