How we improved California Wine Club's ncCPA by 33% using incremental attribution
53 weeks after our dismissal, meet our new favourite tool
Never underestimate the company that quietly listens to feedback and iterates on its product.
Last year, Meta released incremental attribution. When we reviewed it last year, we were skeptical.
On paper, it was a great product. Meta has been way ahead of other platforms by offering incrementality testing products via Conversion Lift for many years and incremental Attribution was built on that.
But the first release was a damp squib. In one of our tests we saw our off-platform incremental CPA increase 5x when using it. It just wasn’t effective at finding the people on platform.
Fast-forward a year, and boy has it had it a glow-up.
How incremental attribution works
All attribution these days is modelled in some way. Ever since iOS14.5, platforms have had less “true” data than they had before. Thanks to cookies and ad blockers and privacy-modes, it’s harder to track a user journey deterministically.
And so even the “7 day click, 1 day view” attribution model we’ve all defaulted to over the last few years is a predictive model.
For businesses with no brand or organic or other presence, we find on average that model under-attributes the true performance. And for those with high volumes of external stuff happening it does the opposite.
So the old attribution model was always broken.
The new one is based on good stuff: all of the conversion lifts that Meta have run.
For those unaware, these are holdout tests that expose your ads to 80-90% of an audience and not the last 10-20%. You count the number of purchases in your unexposed audience and treat it as a baseline, then count the volume in the exposed audience. The difference is your incremental CPA.
Using that as a model made great sense. But last year, it just didn’t work out.
California Wine Club: on-platform and off-platform improvements
We’ve been working with California Wine Club for over a year now.
Summer in DTC is never a peak period, and so we wanted to run a backlog of media buying tests. One of which was incremental attribution.
Now, given our history with IA, we ranked it with low confidence – however, we always like to revisit experiments every 6-12 months. Why? Meta’s algorithm is constantly changing. As well as the major updates like andromeda and gem and lattice, there’s also the smaller day to day updates they push all the time.
Not only that, but people change. Browsing habits, culture, usage of the platforms. With every change comes an opportunity to re-test something.
The results
We decided that the test needed to be a total account switchover and we’d measure before/after. In a dream world we’d run a conversion lift, but time and cost got in the way.
And so we ran a full switchover of our hero campaign setup.
Now, ignoring the fact that new campaigns usually take time to get up to speed, this straight away got off to a great strat.
The key findings were this:
Platform CPA itself barely changed (3% up in one like for like period)
Platform Incremental CPA went down a big chunk (29%)
And off-platform new customer CPA went down even more (33%)
This was an account where spend was – owing to off-peak – static recently. But as a result of this we’re scaling spend and those ncCPA figures remain stronger than before.
Interestingly, before IA, most of the acquisitions attributed were viewthrough, whereas now they’re predomiantly click.
Testing on other clients
As you can imagine, a test like this created quite the excitement in our #growthstrategists chanel.
Since this July test, we’ve now begun rolling out incremental attribution tests with a series of other clients.
Other early results are positive:
One client saw platform CPA come down 20% overnight
Another we saw a 7% decrease in blend, while platform remained flat
We’ve begun testing it with earlier stage clients where data is more volatile to good effect to.
But it’s not all singing and dancing.
We have also seen one instance of almost no impact, and at the same time an increase in ncCPA.
Outgoing thoughts
From automatically turning on AI optimisations to the buggy Ads Manager experience, it’s easy to get frustrated at Meta. But this is one of those great examples of algorithm improvements taking place in the background helping things along.
In the space of six weeks, IA has gone from a ‘steer clear of this’ to a position of almost being the new default.
It’s not 100% success rate (nor is anything). And so testing is vital. And we would highly recommend running our approach to testing which is:
Wholesale account switchover: before and after time window tests
Without this, our hypothesis is that Meta – given the choice of two ads to serve in the auction, will opt to serve the easier to attribute one (non-IA).
We have seen stronger impacts where accounts were viewthrough-driven, but it is impactful in most accounts even when that’s not the case.
How are you finding Incremental Attribution?
Josh Lachkovic is the founder of Ballpoint. We’re the growth agency you hire when you want to dominate your competition. We find profitable growth through psychology-first creative. We’re AI-native, meaning you get the technology power of a tech startup powering a boutique agency. Get in touch if you want to scale spend from £100k to £500k.