80% of experiments fail
Which is why you need to optimise for experimentation rate.
Almost all experiments fail. And that is a good thing.
Failed experiments are a gift. Why would you want to keep on doing something that doesn’t work? Think of the energy you personally put into various things. If they have no – or negative – impact, then learning that allows you to focus attention elsewhere.
And yet in experimentation processes, this fights against a lot of our human instincts to be right.
Experimentation is how you get from 0 to 1 and 1 to n.
When growth was invented at Facebook in the late 2000s, it wasn’t a new moniker for marketing. It was a cultural shift of approach, and at the centre of that culture was experimentation.
I saw Sean Ellis talk a few years ago and he joked that the true north star metric of a team should be number of experiments being run. While I disagree that it’s a north star – because it doesn’t reflect user value – it should be a nuance metric where you never drop below a minimum.
Are you happy with the level of experimentation you’re doing?
Most people will say no.
Many, many teams have good intentions. But most are not following through. Why?
Coming to terms with failure
One of the biggest contributing factors is a misunderstanding of what the reality of experimenting.
If you get in to experimenting because you need something to change, then after a handful of losses, you’ll lose your way.
From the start, you must come to terms with failure. Failure is how we learn and is the byproduct of experimentation.
Fall in love with the process, not your desired result
Experimenting well is a discipline. It has to be organisationally embedded. Early stage, that means an entire culture built around it. Late stage that means teams that run with that culture autonomously within them.
It has to be a discipline, because of the failure rate of experiments.
We all have a bias towards our ideas succeeding. Whenever someone starts something – whether a new feature, a copy change, a new line of photography, a product launch, a or redesign – they believe it will have a positive impact.
At best this confidence is post-rationalised with ‘we’ve spoken to a load of customers who have told us X’, often it’s far worse. But whatever that change is, you’ve got ~80% chance that it’s going to fail.
Ronny Kohavi1, who led experimentation at Microsoft, Bing, and then joined Airbnb in a similar role VP role, reveals the failure rates at each.
At Microsoft, two-thirds of experiments fail
At Bing, 85% of experiments fail
At Airbnb, 92% of experiments fail
If you have this failure rate at some of the strongest experimentation cultures, and most successful companies, in the world, then they’re going to be similar for you too.
Coming to terms with this is important.
One of the reasons I see experimentation cultures break down is because with good intentions, but aren’t ready for the grind of fails. You can easily run a dozen experiments in a row and have them all fail.
For an unprepared team, that is incredibly demoralising and demotivating. And a team that doesn’t have the full buy-in throughout it – and therefore the accepted culture of failure – is one which tries to find positives in the failures.
Lessons from over 500 growth sprints.
I’ve managed or coached over 600 growth sprints during my career from pre-PMF, no-revenue startups, to public companies. Whatever your size there’s a series of things I often see that go right and go wrong.
Here’s a list of suggestions on how to get the process right.
10 tips on running effective experimentation processes
Become comfortable with failure. At school, we’re taught to get ahead by never failing. Building a company (and life) is the opposite. You have to lean in to it.
Test counter-intuitive ideas. Sean Ellis said that the best wins often came from the Friday night tests: those designed to meet the quota. Rory Sutherland would agree. Logic will only get you so far in growth.
Experiment in the right place: you need enough data points to experiment. Until you’ve got 10,000s of visitors, you can’t properly AB test on site. But your ads will have a high enough impression-to-action rate that makes sense.
Celebrate all conclusive experiments: as well celebrating the fails (after all we just learnt what we never need to do again), take time to celebrate wins. Surprisingly often, in a West Wing way, we ask, ‘what’s next?’ and move on.
Document everything. When a new team member joins and has a million ideas, it’s important there’s a place they can go and search through what’s happened before to help sharpen their ideas. I’ve seen experiments repeated far too many times because of lack of documentation.
Set a minimum experiment threshold. Tie this directly to size, but stick to it. The ideas that happen due to the minimum will surprise you.
Make growth organisational, not just the job of one team. If you’re a founder, make your team a growth organisation from top-down. Make sure it’s not just one group talking about fails all the time, but companywide.
Optimise for long term customer value, not short term success. We can all boost revenue in the short term, but we need to focus on long term value. Make sure incentives are correctly aligned.
Ensure you action you wins. I’ve seen scrappy landing pages outperform well-designed product pages, but the LP isn’t rolled out because it’s not the ‘full version’ and the experiment was just the ‘MVP’. If it’s a winner, roll it out
Master uncertainty. We cannot predict the future and we are hugely biased as humans. Even past data rarely helps us predict the future. Embrace this uncertainty, and test everything.
Bites of the week
We’re launching on ProductHunt this week – would really appreciate an upvote or review, please head over here.
The Acquired episode on Nike was brilliant – and even for someone who has read Shoe Dog
Want further reading on experiments? Read 👉 using psychology to come up with experiments, How Pact acquired its first 10k customers, and Social Norms.