A holdout test deliberately removes advertising from one slice of your audience or geography while running normally everywhere else. The slice that didn't see ads is the control; the slice that did is the treatment. The difference in conversions or revenue between the two is the closest thing to a clean measurement of advertising's incremental impact.
Two main flavours
- Geo-holdout: pause ads in a comparable region (or city) for a defined period, run normally in others. Easy to set up; needs comparable geo pairs.
- Audience-holdout: Meta and Google can serve a randomized 'no ads' control inside your normal audience. Higher rigor; requires sufficient volume for statistical power.
Why operators run them
Platform-reported ROAS is biased upward by attribution credit for would-have-happened conversions. A geo-holdout is the only way to know what your ads are actually buying. Most accounts that run their first holdout discover their incremental ROAS is 30-60% lower than their reported ROAS — a sobering number that changes how they budget.
Pragmatic cadence
A full holdout costs revenue (the control group converts less). Most operators run them quarterly, on the smallest geo or audience slice that still gives statistical power, for the minimum duration that produces a clean signal — usually 2-4 weeks. The cost of running them is the price of knowing if your ad spend is real or theatre.