A Pareto candidate is not a causal certificate
Robyn searches across configurations and returns candidates with trade-offs between objectives. A candidate can be attractive on those objectives while still relying on a poor control set, implausible baseline or unsupported channel split. Model selection therefore needs an explicit review beyond choosing the first identifier in an output file.
- Preserve candidate set
- Check prediction and decomposition
- Compare external evidence
- Stress-test the decision
Use the Robyn guide to understand the workflow and the validation guide to distinguish prediction metrics from causal evidence. Save the candidate set so selection can be explained later.
A reproducible workflow makes data checks, modelling assumptions and decision sensitivity visible. The mmm tools, code and platforms learning path connects this idea to the assumptions, practical checks and next steps.
To work through the connected decisions, read Robyn hyperparameters: investigate estimates at the bounds and MMM data leakage: audit the path from raw data to holdout.
Compare candidates on the decision they imply
Imagine candidate A and candidate B have similar prediction error. A attributes most campaign-period sales to television; B attributes them to paid search. If the channels always launched together, that disagreement may reflect weak separation rather than one obviously incorrect optimiser result.
Put the competing decompositions beside the actual campaign calendar. Ask which independent variation supports the split. Do not select A merely because a stakeholder expected television to perform well, or B because its allocation promises a larger lift.
Build a review table before seeing the preferred answer
Record a prediction check, residual patterns, contribution plausibility, baseline behaviour, parameter-bound hits, experimental agreement and the feasible allocation. Define rejection reasons in advance where possible. A channel effect that contradicts an experiment needs investigation of scope and assumptions, not automatic deletion of the experiment.
If the same holdout repeatedly guides model choice, it becomes validation data rather than an untouched final test. Keep a later evaluation period or explicitly acknowledge that reuse. More searches can overfit the selection process even when each individual fit is regularised.
Report the recommendation's stability
Suppose all plausible candidates support a modest increase in video, but disagree about whether search should be reduced by 5% or 30%. The robust conclusion is narrower than either candidate's optimum. A bounded increase and a targeted search experiment may be more defensible than committing to the largest predicted gain.
Run the same business constraints across the shortlisted models. Compare incremental contribution under the same budget and horizon. If the sign of the recommendation changes, show the disagreement and identify what new evidence could resolve it.
The selection memo should include model IDs, data and package versions, why alternatives were rejected, calibration scope and the action approved. The calibration guide explains how experiments can constrain interpretation without making every model assumption disappear.
Sources and further practice
This article’s worked example is illustrative, not a measured client result. Use the linked methodology to check the assumptions before applying it.