Leakage can happen before the model is fitted
A time holdout is useful only if future information has not already entered the features or preprocessing. Whole-history scaling, interpolation across the split, revised labels and repeated model selection can all make evaluation look stronger than a genuine forward prediction.
- Freeze the evaluation date
- Fit preprocessing on training
- Preserve causal time order
- Keep a final test untouched
Start with the validation guide, then trace every transformation from the raw export to the final design matrix. The split date belongs near the start of the workflow, not just in the estimator call.
The response you infer depends on transformation choices and what the data can separate. The how marketing mix models work learning path connects this idea to the assumptions, practical checks and next steps.
To work through the connected decisions, read MMM in Python: validate the weekly data contract and MMM in R: read residual diagnostics before trusting fit.
Work through a scaling example
Suppose a channel's average spend is £10,000 in the training period and £30,000 in the future period. Scaling all rows by the whole-history average lets future spend influence the representation used during training. The effect can be especially consequential when saturation parameters are expressed relative to that scale.
Fit scaling parameters on the training rows and apply them unchanged to validation rows. Keep the fitted preprocessing object or its explicit values. If the future period is far outside the training range, flag the extrapolation rather than silently refitting the scale.
Distinguish valid history from future information
Carrying past adstock into a holdout can be valid: at the forecast origin, past media exposure is known. Resetting that state to zero can create an artificial discontinuity. Using future media or future outcomes to construct earlier transformed rows is a different matter.
Likewise, interpolation that uses observations on both sides of a missing training value may leak if one side lies after the prediction date. A real-time pipeline needs a rule based on what was available then. For revised revenue or refunds, retain data-as-of snapshots to reproduce that information set.
Audit selection as well as preprocessing
If you try many specifications and choose the one that performs best on the same holdout, that period has become validation data. It is no longer an untouched final assessment. Use expanding windows for tuning and reserve a later period where feasible.
Write a leakage checklist covering split dates, scaling, imputation, feature selection, outcome revisions, adstock state and the number of candidate models evaluated. The R tutorial and Python tutorial provide concrete workflows to inspect.
Finally, describe the prediction task honestly. A conditional forecast using known future media and controls differs from a forecast made before those inputs were known. Both can be useful, but their errors should not be presented as the same operational test.
Sources and further practice
This article’s worked example is illustrative, not a measured client result. Use the linked methodology to check the assumptions before applying it.