Start with the effect that would change the decision
The minimum detectable effect, or MDE, is an effect size used in planning a test's statistical sensitivity. It should be connected to a business decision. A design that can detect only a very large lift may not tell you whether a modest but profitable effect exists.
- Define the business threshold
- Translate the outcome effect
- Plan sample and duration
- Pre-specify the analysis
Use the power and sample-size guide for the calculation assumptions. First write down the spend decision, primary outcome, assignment unit and smallest effect worth acting on.
The effect you can estimate depends on the intervention, comparison and outcome you define. The incrementality and experiments learning path connects this idea to the assumptions, practical checks and next steps.
To work through the connected decisions, read Incremental ROAS: report the interval, not just the ratio and Geo-lift market selection: review more than pre-test fit.
Keep relative and absolute effects separate
Suppose the control conversion rate is 2%. A 10% relative increase means a treatment rate of 2.2%, which is a 0.2 percentage-point absolute increase. It does not mean a 12% treatment conversion rate. Confusing those quantities can make a sample-size plan wildly optimistic.
For 100,000 eligible people under the simplified example, a 0.2 percentage-point increase corresponds to 200 additional conversions. If each conversion contributes £30 before advertising, that is £6,000 of incremental contribution before the test's media cost. The accounting helps decide whether detecting that effect would be useful.
Match the calculation to the assignment
A person-randomised binary conversion test and a geo-randomised revenue test need different planning assumptions. Customers within a market share shocks, so treating every purchase as an independent randomised unit overstates the information in a geo design.
For geo tests, use an appropriate simulation or design-specific power method. For individual tests, document baseline rate, allocation, significance level, power and any multiple-testing or sequential-analysis plan. There is no single sample-size number that works across all incrementality designs.
Resolve feasibility before launching
If the required duration is too long, revisit the question rather than quietly accepting an underpowered design. Options include a larger eligible population, a different intervention, a better-measured outcome or accepting that only larger effects can be distinguished. Each changes the interpretation.
Pre-specify the primary outcome, duration, exclusions and decision rule. Use the break-even framework to connect the effect to contribution, while retaining uncertainty in value per conversion.
After the test, report the estimate and interval even when the significance threshold is not crossed. An inconclusive result may still rule out very large effects, but it does not prove there is no commercially meaningful lift. Compare the interval with the business threshold defined before launch.
Sources and further practice
This article’s worked example is illustrative, not a measured client result. Use the linked methodology to check the assumptions before applying it.