Creative testing
How we're approaching in August 2024
When I was head of growth at Thriva, I always had conversations amongst my network of ‘what’s working on Facebook?’ Or ‘how are you approaching horizontal scaling?’ ‘When was the last time you retrained your pixel?’1
When I was in-house I had a belief that there was a single truth that solved all the problems.
It was not long after starting Ballpoint that I realised there is no single answer. There’s no single setup that works and if you find it or ask the right people you win.
That’s not to say we don’t have our own playbook. We’ve got a playbook where 80% of it works 80% of the time for 80% of clients. But with every rule there is usually an outlier. (NB: Except with the AI setting that adds the catalogue incorrectly every time, that’s got a 0% hit rate).
Today then we’re exploring creative testing.
There is no right or wrong way to run creative testing. Though I’ve certainly got a few strong opinions on a few of them. But today, I want to share a few of the ways we run creative testing.
Multiple concepts in one ad set
This is the basic creative testing approach inside a manual2 campaign, ABO.
If you’ve got a very low expected number of conversions due to budget/CPA, then this is often a reasonable place to start. At low conversion volume, concept testing is the most important spend you can do. Don’t mess around with variants at this stage, once you’ve got your first JTBD messaging working.
Generally speaking as you scale, we advise moving away from this, but as a very first step it’s a reasonable way to go.
As with all non-ASC campaigns, you can be specific about audience testing and exclusions too. True heads exclude everyone who’s engaged with the brand in any way. In reality, we usually exclude customers but allow for engaged.
Upsides: very fast to set up, and you can control spend levels very tightly, setup mimics how you will eventually scale.
Downsides: no chance for messaging testing, not fully isolated, spend will be based on what Meta deems appropriate
Variants within an ad set
This is our preferred approach.
Here the ad set is for the creative concept, and the ads within it allow us to explore variants. This could be different hooks of a UGC, or it could be different messaging tests.
Depending on what we’re trying to test we sometimes invert this setup: i.e. if we’ve got three messaging tests and three concepts, we might run a messaging group at each ad set level, and the 3 different concepts are in each of those ad sets.
Upsides: you get better learning at a concept level, which is where most heavy lifting is done. Plus it continues to emulate how social accounts scale.
Downsides: more expensive than all concepts in one ad set. And you still don’t get precise data
All in ASC testing
ASCs can take a lot more ads than manual campaigns can. That means you can, budget allowing, stick 8-20 different ads in a single ASC. When we do this it will likely include both the concept and variants so that we get a broader view of what works best.
Again this mimics the scaling power of an ASC very well. It means you can test everything in an even easier setup.
Upsides: the easiest way to test, and you get the efficiency benefit of an ASC
Downsides: ASCs overindex on engaged/aware users and so that means your testing reflects an audience who already knows you. Plus ASCs can take longer to get up to speed so potentially a need for more than 7 days of testing time.
One ad per ad set, strict AB test
The OG setup.
Here we have one ad per ad set and then isolate each ad set using Meta’s A/B tool to run a properly isolated test. This approach is great for those who want to have as much control as possible over testing and identify a “true CPA” of a creative.
This is mostly downsides for us with the exception of where there are strategic elements we want to understand a specific answer to.
Upsides: the most precise learnings.
Downsides: expensive (imagine running 3 variants, 5 ads, getting 50 conversion events per week! £££££££!), doesn’t reflect how Meta generally works. Just because in low controlled spend something gets good CPA doesn’t mean it can scale. Psychologically gives you belief you can control things more than you can.
Cost capped testing
As with all above methods but you can apply cost caps rather than run on maximum volume.
Upsides: test a huge number of ads, never need to turn an ad off
Downsides: if cost controls don’t work on your account, you won’t get any impressions here either. Also cost controls can overspend.
Match your business stage
If you’re super early stage, don’t get too cute with your creative testing. Remember that once you’ve landed on a JTBD-focused approach to selling, the concept is the most important lever you have. Do a high frequency of testing, keep it simple.
Your testing volume should grow in line with spend and conversion volume, and as it does, you’ll want to explore another testing setup.
There is no one size fits all though, and conditions change all the time. And so don’t stay wedded to one idea, and know that it won’t make or break your strategy if you do.
One person I met back in those heady days of 2017 was a firm believer that pixels got lazy and you had to retrain it by deleting it and starting again every few months. One month when CPAs were up we gave it ago, it wasn’t the silver bullet we were hoping for.
When we use the phrase Manual to describe campaign setup, we mean “not ASC”