Incrementality testing for agencies: measuring what your ads actually caused

TL;DR

Attribution divides credit among the touchpoints a conversion touched; incrementality asks the harder, more honest question: would this conversion have happened anyway, without the ad? The two often disagree, and the gap is where budget gets wasted — channels that capture existing demand (brand search, retargeting) look like heroes in attribution while adding little true lift. You measure incrementality with a controlled experiment: hold a group back from seeing the ads (a geo holdout, a platform conversion-lift study, an audience holdout) and compare. It's more work than reading a dashboard and isn't worth it for tiny budgets — but for clients spending real money, it's the difference between optimising a lens and optimising reality. Below: why attribution misleads, the test methods from simplest to hardest, and when it's worth running.

---

Every agency has a client whose branded search campaign shows a gorgeous ROAS — and every experienced media buyer suspects that most of those people would have searched the brand and bought anyway. Attribution can't tell you whether that's true; incrementality can. For accounts where the spend is large enough to matter, it's the most important measurement question you can ask, and almost nobody asks it.

Attribution vs incrementality

• Attribution (even good data-driven attribution) starts from a conversion and distributes credit across the touchpoints that led to it. It answers "who gets the credit?"
• Incrementality starts from a causal question: of the conversions credited to this channel, how many would have happened without it? It answers "what did the spend actually cause?"

The trap is that the channels which harvest demand you already created — brand search, retargeting, sometimes shopping — score brilliantly on attribution because they're close to the conversion, while contributing little incremental lift. Cut them and revenue barely moves; attribution would have told you they were your best performers. That's not a rounding error — it's how agencies pour budget into channels that are taking credit for conversions they didn't cause.

How you actually measure it (simplest to hardest)

1. Geo holdout test. Turn the campaign off (or down) in a representative set of regions, leave it on in comparable ones, and compare conversion rates. Differences in the regions are the incremental lift. Google's geo-experiment tooling and open-source approaches make this the most accessible rigorous method for many clients.
2. Platform conversion-lift studies. Meta and Google can run lift tests that hold out a randomised control group from seeing the ads and report the incremental conversions. Less control than your own geo test, but turnkey — and a good first step.
3. Audience holdout / ghost ads. Withhold the ads from a slice of the target audience (or serve a placebo) and compare. Cleaner causally, more setup.
4. Marketing mix modeling (MMM). A top-down statistical model attributing outcomes to spend across channels — useful at larger scale and for offline/brand effects, but heavier and less granular. Increasingly accessible via open-source models, still a project.

You don't need to start with the hardest one. A single geo holdout on the client's biggest-spend channel often reveals more than a year of attribution-report staring.

When it's worth it (and when it isn't)

Be honest with clients about the cost-benefit:

• Worth it: large, sustained spend; a channel you suspect is over-credited (brand search, retargeting); a big budget-reallocation decision; a client who'll act on the result.
• Not worth it: small budgets where the test cost and the lost exposure during the holdout outweigh the insight; channels too small to reach statistical significance; a client who won't change anything regardless.

And one prerequisite that's easy to forget: incrementality testing assumes your tracking is sound. If conversions are mis-recorded or half your traffic is Unassigned, the experiment measures noise. Fix the data capture first — the test is only as trustworthy as the conversions it compares.

The agency angle

Incrementality is a genuine differentiator. Most agencies report attribution and call it measurement; an agency that can say "we tested it, and your brand-search budget is 80% non-incremental, so we moved it here and total revenue rose" is operating on a different level — and it's a defensible story at renewal and a reason to expand the engagement. It reframes the agency from "we optimise the numbers in the dashboard" to "we optimise the client's actual business."

Where this fits

Incrementality is the layer above attribution — and like everything above it, it rests on the conversions underneath being captured cleanly across the client's tags, platforms, and pixels. Phloz keeps that substrate legible: each client's tracking modeled and health-checked as part of the tracking-infrastructure map, so when you run a lift test you're comparing data you've verified, not hoped for. The CRM for PPC agencies and pricing pages cover the workflow — but the mindset shift is the point: attribution tells you who got credit; incrementality tells you what you actually caused, and only one of those is worth optimising.