A power analysis tells you, before a test runs, how likely it is to detect an effect of a given size, and from that how big the test needs to be or how small an effect it can realistically find. Statistical power is the probability that a test flags a real effect as significant, and by convention you design for 80 percent power to detect the smallest effect that would change your decision, which is called the minimum detectable effect (MDE).
I think power analysis is the most neglected step in marketing experimentation. In my experience, teams will happily argue about whether a p-value is 0.04 or 0.06 after a test, but very few ask beforehand whether the test had any real chance of finding the effect they care about. The result is a steady stream of “not significant” readouts that get interpreted as “the channel doesn’t work”, when the honest conclusion is “we didn’t collect enough data to say”. I’ll show you how to run the numbers before you commit budget, with a calculator for both conversion-rate A/B tests and geo tests.
The four numbers in every power analysis
Every power analysis connects four quantities, and fixing any three determines the fourth:
- Significance level (alpha): the false-positive rate you’ll tolerate when there’s truly no effect, conventionally 0.05.
- Power (one minus beta): the probability of detecting the effect if it’s really there at the size you specified, conventionally 80 percent.
- Minimum detectable effect: the smallest true effect you want that 80 percent chance of catching, as an absolute or relative lift.
- Sample size: visitors, users, regions or region-weeks, plus how they’re split between treatment and control.
There’s a hidden fifth ingredient, which is the noise in the outcome (its variance). For a conversion rate the variance comes from the baseline rate itself, . For revenue per user or sales per region you have to estimate it from historical data, and that estimate is usually the most important and least scrutinised input in the whole calculation. For two equal-sized groups and a continuous outcome with standard deviation , the sample per group needed to detect a difference is:
At alpha 0.05 and 80 percent power the z-values sum to about 2.8, and is close to 16, which gives the rule of thumb Robert Lehr published in 1992 and Gerald van Belle’s Statistical Rules of Thumb made famous: you need roughly per group. The key thing to notice is the square on the effect size. Halving the lift you want to detect quadruples the sample, which is why “can we detect a 1 percent lift?” is so often a much more expensive question than people expect.
Sample size calculator for conversion and geo tests
The calculator has two modes. Conversion mode is the classic two-proportion sample size calculation: give it a baseline conversion rate and the relative lift you care about, and it returns the sample per arm and in total, how many days that takes at your traffic, and the inverse, the smallest lift your available traffic can detect. Geo mode treats each region as one unit and asks you for the noise between regions and within regions week to week.
The power curve is the part I’d show stakeholders. It makes the point that power only means something for a particular effect size. A test with 80 percent power for a 10 percent lift might have 30 percent power for a 5 percent lift, so if the true effect is 5 percent, you’ll most likely come back with nothing. Once you’ve run the test, the statistical significance calculator is the other half of the job, for reading the result.
Why a null result needs a precision story
The course’s lesson on experiment power starts with exactly this situation. The invented Northstar brand runs a small regional experiment, the estimate is positive but the uncertainty is large, and the report labels it “not statistically significant” and concludes the channel doesn’t work. That conclusion doesn’t follow from the label. A noisy design can have very little chance of detecting an effect the business would still care about, so “we found no significant effect” is only informative if the test was precise enough that a meaningful effect would have shown up.
In practice that means a null result needs a precision story: report the confidence interval and set it against the decision. If the interval runs from minus 5 percent to plus 20 percent and your channel needs a 10 percent lift to break even, the test hasn’t told you whether to keep spending; it’s told you that you need a bigger test. The same logic applies in the other direction, because a point estimate above your hurdle isn’t enough if the interval still includes values below it. Power is also a property of the design under assumed effects, not a probability that the effect in one finished test is real, so don’t quote “80 percent power” as if it were confidence in your result.
Power for conversion-rate tests vs geo tests
For a website A/B test, the unit is the visitor and you usually have thousands or millions of them, so the main constraint is the baseline rate and the lift. A 3 percent baseline and a 10 percent relative lift needs a bit over 100,000 visitors in total at the usual settings, and low baselines get expensive fast because the signal is tiny relative to the binomial noise.
Geo tests are a different animal, because the unit is the region and you might only have 10, 50 or 210 of them. The noise has two parts: regions differ from each other (a city of 8 million isn’t a town of 80,000), and each region bounces around from week to week. Running the test for more weeks only shrinks the second part. If regions differ a lot and you compare them naively, more weeks barely help, which you can see in the calculator’s geo mode with the pre-period slider at zero: doubling the weeks hardly moves the curve, while doubling the regions does. That’s why every serious geo design uses the pre-period, whether through matched pairs, difference in differences or a synthetic control, to explain away most of the between-region variation. Push the pre-period slider up to 80 or 90 percent and watch the minimum detectable effect fall, and notice that test length starts to matter once the remaining noise is mostly week to week.
The geo calculator is deliberately simple, so treat it as a first pass. Real regions aren’t independent (national media, seasonality and spillover hit them together), weekly noise is autocorrelated, and treated regions are rarely an even split of similar-sized units. The more realistic approach is the one Google’s Jon Vaver and Jim Koehler described in 2011 and that Meta’s GeoLift automates in its market selection: replay your design on historical data, many times, as if a test had run, and see how wide the resulting intervals are. I cover that on the geo lift testing page.
Why most marketing tests are underpowered
The deeper reason marketing tests come back inconclusive is that advertising effects are small relative to how noisy sales are. Randall Lewis and Justin Rao made this point most forcefully in their 2015 Quarterly Journal of Economics paper, analyzing 25 large field experiments run with major US retailers and brokerages. Individual-level sales are so volatile, compared with the per-person effect of an ad campaign, that the median confidence interval on ROI was over 100 percentage points wide, and some questions would need more than 10 million person-weeks of data to answer. I go through their argument and the arithmetic behind it on the Lewis and Rao page.
The practical consequences are worth spelling out. A campaign can be profitable and still be nearly impossible to distinguish from zero in a single test, so the absence of significance is weak evidence against it. Tests that do come back significant from small samples tend to overstate the effect (only the lucky draws clear the bar), so a surprisingly big lift from a small test deserves a replication before a budget shift. And because each test is noisy, the real value comes from accumulating them: a log of past tests, each with its interval, feeds a marketing mix model as priors or calibration far better than any single result can.
How to get more power from the tests you can afford
Once the calculator has told you the bad news, there are a handful of levers. Test a bigger change: a heavier spend increase (or a full holdout) produces a bigger effect, and the sample needed falls with its square, though you then learn about a different point on the saturation curve. Choose a less noisy outcome closer to the ad, such as sign-ups rather than revenue, as long as it still relates to the decision. Use the pre-period to explain away noise, through pre-period covariate adjustment for user tests (often called CUPED) or matched designs and synthetic controls for geo tests. Use more units rather than more time where you can, and an even split between treatment and control, which minimizes variance for a fixed total. Finally, don’t peek and stop when the result looks good, because repeated looks inflate the false-positive rate well above your alpha unless the analysis is designed for it.
Power analysis is the step that makes the rest of an incrementality testing program worth the money, and it’s also the cheapest step: it takes minutes and happens before any budget is committed. The Vexpower MMM course has an experiment planning lab that uses the same two-sided normal test with independent, equal-sized groups, so you can see the distribution of estimates narrow as the number of units grows.
