Calibration needs an audit trail
A calibration result often travels from an experiment report to a spreadsheet, then into a model configuration and finally a budget presentation. At each step, its scope can be lost. An evidence register keeps the original result separate from the assumptions used to apply it.
- Record the tested contrast
- Preserve result and uncertainty
- Document translation
- Review transfer over time
The calibration guide explains the statistical role. The register is an operational safeguard: it lets a reviewer trace a channel assumption back to what was actually tested.
The effect you can estimate depends on the intervention, comparison and outcome you define. The incrementality and experiments learning path connects this idea to the assumptions, practical checks and next steps.
To work through the connected decisions, read Meridian ROI priors: translate experiments carefully and MMM refreshes: separate new data from a changed business.
Record fields that change the interpretation
Include the experiment ID, owner, dates, geography, eligible population, assignment unit, channel, campaign, treatment contrast, outcome, spend definition, estimate, interval and analysis method. Add delivery deviations, contamination and outcome-maturity notes.
Keep the original report and data version. A test of a temporary 20% budget increase in one region should not become a generic national channel ROI merely because the spreadsheet column is labelled return. Preserve the contrast in plain language.
Separate evidence from translation
Imagine an experiment estimates incremental orders, while the MMM uses net revenue. The register should hold the original order estimate and its uncertainty. A separate translation record should state the assumed value per order, returns treatment, target model window and any added uncertainty for transfer.
If the same test appears in two reports, retain one underlying evidence ID. Otherwise a synthesis can accidentally count the same observation twice. Record dependencies between experiments that share controls, periods or customers.
Review applicability as the business changes
A creative change, pricing shift or new audience can weaken the relevance of an old test. Age alone is not a universal expiry rule, but changed mechanisms deserve review. Mark evidence as directly applicable, partially transferable or unsuitable for the current decision, with a reason.
Compare the model's recommendation with and without influential evidence where appropriate. If one uncertain translation drives the whole allocation, that is a useful priority for a new experiment. The measurement framework helps turn the register into a learning agenda.
For each model release, save the evidence IDs used, the translation configuration and the sensitivity results. This makes a refresh reproducible and prevents a new analyst from mistaking an old judgement for a directly observed fact. The register supports scrutiny; it does not eliminate the need to assess experimental validity.
Sources and further practice
This article’s worked example is illustrative, not a measured client result. Use the linked methodology to check the assumptions before applying it.