Difference in differences (DiD, or diff in diff) estimates the effect of a change by comparing how much a treated group moved before and after the change with how much an untreated comparison group moved over the same period. The comparison group’s change stands in for what would have happened to the treated group anyway, so subtracting it strips out the trend, seasonality and anything else that hit both groups, and what’s left is your estimate of the effect.
It’s the estimator behind most marketing experiments that aren’t user-level A/B tests: the geo lift test where some regions get extra spend, the matched market test for a TV campaign, the launch in one country before another, the price change in some stores but not others. It’s also one of the oldest tools in causal inference, and its most famous application is economics rather than marketing: David Card and Alan Krueger’s 1994 study compared fast-food employment in New Jersey, which raised its minimum wage in 1992, with neighboring eastern Pennsylvania, which didn’t. I’ll show you the formula first, then the assumption it rests on and the ways it breaks in marketing data.
The difference in differences formula
The simple two-group, two-period version needs four numbers: the treated group’s average before and after, and the comparison group’s average before and after. The estimate of the treatment effect is the treated group’s change minus the comparison group’s change:
In the course’s invented Northstar case, the campaign region rises from £100,000 to £135,000 and a similar comparison region rises from £100,000 to £120,000 without the campaign. The campaign region’s before/after change is £35,000, but £20,000 of that also happened where there was no campaign, so the DiD estimate is £15,000. Equivalently, you can think of it as building a counterfactual for the treated region (its pre-period level plus the comparison group’s change, £120,000 here) and measuring the gap between that and what actually happened.
With more regions and more weeks, the same estimate comes out of a regression with a fixed effect for every unit (which absorbs permanent differences in size), a fixed effect for every period (which absorbs shocks common to everyone), and a dummy that’s one for treated units in the post-period:
This is called two-way fixed effects (TWFE), and when every treated unit starts at the same time it gives exactly the same answer as the four-means calculation. On the course’s 12-region geo dataset, both report 9. The regression form is handy because it gives you standard errors (cluster them by region, since weeks within a region aren’t independent), lets you add controls, and extends naturally to the event study below.
Parallel trends: the assumption everything rests on
DiD doesn’t assume the two groups are the same. A big region and a small one are fine, because the unit fixed effects (or the before/after subtraction) remove level differences. What it does assume is that, without the treatment, the treated group would have changed by the same amount as the comparison group. That’s the parallel trends assumption, and it’s untestable in the strict sense because you never see the treated group’s untreated path after launch.
In marketing, parallel trends breaks in some predictable ways. The most common is selection: you put the campaign in the regions that were already growing (or the ones the regional manager lobbied for), so the treated group was on a steeper path before you touched it. The opposite also happens, where you pick the regions that had a bad quarter, and they bounce back on their own (regression to the mean). Different seasonality is another, such as coastal regions that peak in summer compared with inland ones, as are local shocks like a store refit, a competitor closing or a regional promotion. A local event in one treated region isn’t removed by subtracting a shared trend, which is exactly the extra one unit in the geo dataset’s DiD of 9 against a true effect of 8.
The calculator below lets you break the assumption on purpose. The violation slider says “without the campaign, the treated group would have changed by this much more than the comparison group”, and the bias in the DiD estimate is exactly that amount, one for one. Switch to weekly mode and the calculator also estimates the drift from your own data by fitting a line to the pre-period gap between the groups.
The point of the slider is to make you put a number on your worry. If a plausible violation is two units and your estimate is 15, the conclusion survives; if a plausible violation is ten, the test hasn’t told you much yet. In my experience, this is a much more productive conversation to have with a finance team than arguing over whether a p-value is below 0.05, because it’s about the business story of why the regions might have diverged.
Event-study plots: checking the pre-period
An event-study plot shows the gap between treated and comparison groups period by period, relative to the launch date, usually as regression coefficients with confidence intervals and the last pre-launch period as the reference point. Before launch the coefficients should hover around zero with no slope; after launch they show how the effect builds, holds or fades. The “Diverging pre-trends” example in the weekly mode of the calculator shows what a problem looks like: the gap is already climbing before launch, and a naive DiD credits that climb to the campaign.
Flat pre-trends are reassuring, but they aren’t proof. A test can have perfect pre-period alignment and still be ruined by something that happens only in the post-period. Jonathan Roth’s 2022 paper “Pretest with Caution” also showed that standard pre-trend tests often have low power to detect violations, and that only reporting estimates which pass such a test can bias them, so treat the plot as evidence to weigh rather than a gate to pass. (His 2023 survey with Pedro Sant’Anna, Alyssa Bilinski and John Poe is the best single overview of the recent literature if you want to go deeper.) The post-launch shape matters for marketers too: if the effect is still rising at the end of your test window, you’ve probably undercounted the carryover, and if it decays to zero within a week, a longer test won’t help.
Staggered rollouts and the two-way fixed effects trap
Marketing launches are often staggered: a new channel goes live in three markets in March, five more in May and the rest in September. The tempting analysis is the TWFE regression above with a treatment dummy that switches on at each market’s own launch date. However, research over the last few years has shown that this regression can give misleading answers when effects change over time or differ between launch groups, which in marketing they almost always do.
Andrew Goodman-Bacon (2021) showed that the TWFE estimate is a weighted average of every possible two-group, two-period comparison in the data, and that some of those comparisons use already-treated markets as the control group for newly treated ones. If the early markets’ effect is still growing, their change gets subtracted from the later markets’ change, which can drag the average down and, in extreme cases, flip its sign. Brantly Callaway and Pedro Sant’Anna (2021) proposed estimating a separate effect for each launch cohort in each period, using only not-yet-treated or never-treated markets as comparisons, and then averaging those with weights you choose; their R package did implements it. Liyang Sun and Sarah Abraham (2021) showed the same contamination problem affects event-study regressions and proposed an interaction-weighted fix. The practical upshot is simple: if your treatment starts at different times in different units, use one of these estimators (or keep a clean never-treated group and analyze each cohort separately) rather than the textbook regression.
# Callaway & Sant'Anna: one effect per launch cohort and period, then aggregated
library(did)
out <- att_gt(yname = "sales", tname = "week", idname = "market_id",
gname = "launch_week", # 0 for markets never treated
data = panel, control_group = "notyettreated")
es <- aggte(out, type = "dynamic") # event-study style aggregation
ggdid(es)How marketers use difference in differences
Geo tests are the obvious use, and on the geo lift testing page I work through the course dataset where a treatment-only before/after says 17, DiD says 9 and the known effect is 8. DiD is also how most people read a launch: a new store, a new market, a feature released on one platform first, or a rebrand in one country. It works for “off” tests too, where you pause a channel in some regions to see what it was really contributing, which is often the cleanest way to check an incrementality claim from a platform.
A few practical habits make these readouts much more trustworthy. Use as long a pre-period as you can and plot it before you compute anything. Pick the comparison group before you see post-period data, and write down why. Report the result as an incremental number with an interval and an implied iROAS, not just a percentage lift, so it can be compared with your marginal ROAS from other sources. With only a handful of regions, the usual clustered standard errors are unreliable, so a placebo check (pretend each comparison region was treated and see how often you get an effect as big as yours) is a good sanity test; the same idea is central to synthetic control, which is the natural next step when no comparison group trends like your treated unit. Before you run anything, use the statistical power calculator to check that the test can detect an effect worth acting on.
DiD results are also some of the best evidence for anchoring a marketing mix model, because they come from a deliberate change rather than whatever variation happened to be in the historical data. The calibration page shows how to combine a test result with a model estimate, and the MMM course has a lesson where a competitor closure in the campaign region shows you, interactively, what a comparison group can and can’t remove.
