You don't experiment to 'find the answer' – experimentation is the answer

There is no 'best creative' or 'best media strategy'.

I’m a music geek and for a long time was a member of a forum called I Love Music. About a decade ago, I remember a fierce debate taking place about the ‘best’ album. The debate went back and forth, until someone said “what’s the aim of this? There isn’t any ‘best’ album, are we on some pursuit of a singular record that stops us listening to all other records?”

There is a sense among a lot of people I speak to that experimentation within business has a similar desired aim. We are looking for the ‘best’ answer, the ‘best’ result, the ‘best CPA’, the ‘best conversion rate’. All with the implication that once we’ve found it, we can move on from experimentation and onto more important things.

But just like with music, there is no best result. The joy (and results) lie in the discovery,

Experimentation is the answer, not the way to find the answer

The ‘Growth Process’ is a way to prioritise work, organise a team, drop ineffective work, learn as fast as possible, and grow through the path of least resistance.

The growth process is visually displayed in a kanban board starting with a backlog, becoming experiments and then measured by results.

Over the last year, I’ve run 18 growth processes. The businesses which grew fastest and most profitably, were those with the highest volume of experiments. It’s not a huge sample, but we discovered a positive correlation between rate of experimentation and improvement in outcomes.

It’s not just us. Sean Ellis believes your experimentation rate should be your team’s north star metric.

Predicting the future is impossible – and yet so much of human behaviour seems to think it can

Startups are full of hyper-rational technologists. (Spoken as a proud technologist, and recovering hyper-rationalist).

For a long time, I used to believe that to predict the future, we just needed more data. Get enough data, and we can predict it all. That’s the predictive machine learning dream, right?

As a result, startups are often full of people searching for ‘the answer’. And they want that answer to inform the future.

Not only this but we’re also highly susceptible to influence from our experiences and others.

Confirmation bias means that we seek out evidence that reinforces our world view. I’ve sat in too many growth meetings where people try to find silver linings in failed experiments because they were the owner.

Founders and senior leaders also suffer from overconfidence bias. Everything leading up until this point has taught us that confident people get ahead further and faster. As a result, leadership very often falls fowl to overconfidence.

And our hindsight bias means that when we do review old events, we feel we can explain them.

Humans inherently believe they can control a lot of their environment. And those who work in startups seem to over index on this. But reality isn’t controllable. It’s messy, complex, and unpredictable.

This reveals itself in an infinite number of examples. From big splashy examples like New Coke through to the new website some team just pushed live because ‘it’s obviously better than the old one.’ Hint: it probably isn’t.

Failure rates in experimentation show this

For those that do experiment, we quickly see all of this in the data. Consider these benchmarks from Ron Kohavi’s excellent article on AB Testing¹

Last week, Microsoft overtook Apple as the most valuable company in the world. So clearly they’re doing something right. But if that’s the case, why are two-thirds of their core experiments, and 85% of Bing’s experiments failing?

A very high failure rate is nature of the experimentation process if you’re doing it well. Like with OKRs if you find yourself succeeding most of the time, you’re probably experimenting wrong.

Building a business of thousands of wrong turns

Venture Capitalist Chris Dixon penned the phrase the Idea Maze² over 10 years ago. He challenged the canonical view that startup success was built around eureka moments.

Great success isn’t just one big idea that works and riches follow.

Instead, the building of a startup is a maze that takes years to get through with almost infinite possible combinations of paths. Your job is to find the right path before you run out of money.

If we accept that principle as true (which in my experience it is), then we are faced with a decision of how we faces those potential turns in the maze.

Would you rather make big calls, be dogmatic in your approach, and charge ahead? The chances of getting this path correct round down to zero.

Or would you rather embed a system that recognised that wrong turns are inevitable, and therefore it is better to learn as much as you can throughout that process.

---

The marketing platforms are constantly changing

At a macro-level then, the case for experimentation is clear, but this boils down to the micro ways we execute our businesses as well.

If we’re advertising via Meta, TikTok, or Google. We are at the whim of platforms which we won’t ever know the full extent of their workings. The way each ad platform decides to display an ad is based on an auction algorithm. It takes into account various factors which we can explain with things like ‘the bid’, ‘relevancy’, and ‘expected conversion’. But behind these clear concepts is a myriad of data: infinite points which only machines can understand.

Not only that, but these platforms are constantly evolving.

Some of these changes are big: like when Meta went on its Reels push, you could see the impact it had on placement impressions. But the vast majority will be tiny changes, which isolated you may not notice an impact, but as time progresses all have outcomes on performance.

This means what “works” on these platforms changes.

Just consider the last two years Meta: audience structure has shifted away from being lookalike and interests-based, to broad. Automatic placements beat manual placements in almost every test. The number of ideal ads per ad set has shifted. The use of exclusions has shifted. Reels exploded, and Meta introduced ASC.

We have ‘best practice setups’ which we see work in 80% of cases, but there are always exceptions to the rule. And because the platforms evolve, what works for six months won’t be guaranteed to work for the next six months.

This is why the only ‘media buying strategy’ you can run is experimentation.

Culture is more fragmented and evolves faster than ever

Go back 20 years and pop culture was far more homogenous. There were fewer sources that people would learn and discover from. The breadth of tastemakers was more narrow, and their influence was far greater.

Think about it with the aforementioned example of music. You got signed by a big label that did the big MTV push, the big radio push, the big music press push. If you get picked up, you fly. As a mid-millennial, my early years were full of defined musical scenes and sounds. You could get a far more solid sense of a year by listening to the canon.

Today, everything is more fragmented. A new artist today can build a micro audience and spread across the world with no interaction from the old gatekeepers. Not only do they do this independently, but more quickly as well.

TikTok accelerated this change. Algorithms have replaced tastemakers.

Someone’s niche community can change incredibly rapidly. That means a change in tastes, behaviours, likes, dislikes, and everything in between.

All of that is reflected in the content of that community.

As advertisers, it is our job to reflect that culture back to the community. Every aspect of your ads plays into that. And so, with a community’s culture constantly evolving, a creative strategy cannot be one and done.

It’s not just ads. This is in everything your brand says and does. Tastes in design mean your experiments in CRO must be constant. Evolution of copy needs to be ongoing. To say nothing of eventually you’ll need to move beyond initial segments into new ones.

You’re not searching for a “best ad”

One of the great things about the internet is that it can connect you directly to those microcosms.

Whether you’re browsing Reels or your feed and getting served ads, these ads are for you. A few years ago everyone was talking about how each person’s For You page would look different due to who you are; but that’s been true of your ads experience for even longer.

Paid social strategy needs reframing around the ‘no best ad’ idea

Because the internet is really a global connection of billions of microcosms, your ad strategy can be framed around specific customers ideas. I talked about this previously in Creative is your Targeting.

Some ads will talk to your entire market very effectively, but others may talk to a very small niche within your market only. This can manifest with certain ads only being able to spend certain amounts. When you increase spend, CPA rises because you’ve hit the ceiling of that niche.

Experimentation is the only way

We experiment not as a means to an end to finding the right way to do things, but because the process itself is the right way to do things.

At a high-level, building companies is a maze of 10,000 turns.

At a psychological level, our own biases often cloud our judgements into believing we know the answers and can predict the future.

At a societal level, culture is more fragmented and faster changing than ever before. As a result, what was true yesterday isn’t necessarily true today.

And at a technology level, the ad platforms are constantly evolving as well.

Experimentation allows us to take as much advantage of these uncertainties as possible, and maximise our chance of being able to capitalise on good luck when it eventually comes our way.

1. A/B Testing Intuition Busters, Common Misunderstandings in Online Controlled Experiments. Ron Kohavi, Alex Deng, Lukas Vermeer

2. The Idea Maze. Chris Dixon.