Should we be on Amazon? The DTC case study with Mother Root
Should we be on Amazon? The DTC case study with Mother Root
What is the incrementality of being on Amazon?
I speak to a lot of early stage founders of consumer brands.
For most, DTC is a question mark – but Amazon is a de facto yes.
Amazon is both a shopfront and a marketing channel. A place that many customers begin their journeys, and even more, end them. The explosion of so many consumer brands over the last decade has in large part been down to Amazon.
One of our clients is Mother Root.
They're a non-alc aperitif that outside of the beer must, by many calculations, be the fastest growing brand in the category.
Earlier this year, we ran an experiment to pull Mother Root from Amazon.
Why? We had a hypothesis that Amazon was not incremental. We had a hypothesis that it was diverting customers away from the data-rich platforms of DTC. We had a hypothesis that it was bad for business.
Bethan is the founder of Mother Root, and when we went live with this experiment, the post attracted quite a lot of attention.
5 months later, we're back on Amazon but our strategy has shifted.
This is a post that explores the journey behind that experiment. The insights that led us to the decision, the way we chose to execute it, and the way it's impacted decision-making afterwards.
Amazon for many is a no-brainer. This post will hopefully argue that it shouldn't be.
Dichotomy: so many brand founders love the performance-y-allure of Amazon
Most founders of consumer brands love Brand with a capital B. Many come from big CPG conglomerates, or have experience in bigger corporate consumer.
The types of brands they worked on before their startup were likely big budget, retail-distribution and TV ad plays. I.e. a very specific type of marketing that requires huge amounts of money.
And most of those founders who fall into this category have a very strong idea of what their own brand should be.
Visual identity, ethos, values, colour, packaging. These things matter.
And while they know that big flashy brand marketing campaigns are likely out of the window for a long time, it's still back of mind part of the aspiration.
Even I – a performance marketer by background – when launching my CPG-cum-education wine play back in 2019 started caring about these details more than I’d ever done before. It wasn’t even conscious, I just suddenly started caring.
And so it's always felt like an odd dichotomy the Amazon play.
Amazon and paid search have huge amounts in common
The biggest commonality is that there are two ways to make sales on either channel1.
1. You satisfy the people who are searching for that product
I.e. people are searching for "post workout protein snacks" and you create a product that satisfies that search query. The search volume of that query shows you what the opportunity is.
Reality check here: most generic search queries have way, way, way, way, way less volume than you'd like.
2. You capture existing brand demand
I.e. people throw your brand name into Google/Amazon, you serve them an ad and hey presto you got a sale.
Brand demand is created through other types of advertising and marketing. And while it will – if you run it – look like the best return on marketing spend imaginable, it is basically not incremental2.
Now brand has the dual impact of not only driving the customers in category 2, but it also increases the likelihood of conversion in category 1. Better brand awareness and affinity = better category sales when at the point of decision.
In short, Amazon is an old school performance channel in both marketing and sales terms. But importantly one that mimics the supermarket that Amazon's digital marketplace emulates so well.
Psychologically, everyone talks about Amazon. There is case study after case study talking about the huge returns you can make on it.
And there are many businesses, now doing 8 or 9 figures in annual revenue, who got big in the early days on Amazon.
Launching your brand on Amazon is the obvious thing to do.
"The correlation between Meta spend and Amazon revenue is pretty strong"
This was me in one of our growth meetings in Q4 of 2023.
Ballpoint had built the first version of Mother Root's growth dashboard, and in it we were tracking all company metrics.
It was one of those brilliant moments validating what is a logical assumption.
Every paid social advertiser knows that Meta works across the full funnel. Now we were seeing it work across multiple sales channels rather than simply where we directed the traffic.
Our hypothesis as to why this happened was twofold:
1. The brand impact of our Meta advertising (all performance ads at this point) had a positive conversion increase impact on the category 1 Amazon shoppers
2. Amazon was a preferred sales destination for many shoppers and so while the Meta ad convinced them to buy, they would have rather completed their purchase on Amazon.
The outcome of that meeting was that we refactored our headline CM3 calculation to include Amazon data too.
We previously were calculating:
(New customer CM2 from Shopify) minus (all Meta + Google spend)
The new calculation added in:
(New customer CM2 from Shopify + Amazon) minus (Meta + Google + Amazon spend)
If Meta's impact was felt across sales channels, it made sense to benchmark it in that way as well.
This was all good. If people wanted to shop on Amazon and they wouldn't have done on Shopify, it makes sense to give them the choice. Likewise, if people are starting their journeys on Amazon, it's important to amplify that too.
The first signs Amazon wasn't all good
In January, Mother Root went out of stock.
Both Shopify and Amazon went offline, and we paused ads.
Very quickly, we decided to launch a 'pre-order for the restock' on Shopify. Bethan, the founder, put out a couple of organic reels, we spun up some ads and went live again.
Within days, we were seeing the best CPA we'd ever seen.
Shopify CM3 was better than usual combined CM3.
Shopify blend CPA was better than usual combined blend CPA.
Part of this was no doubt FOMO from the re-stock. Part of it was likely an offer.
Plus we were still in January, the peak period for alcohol abstinence. Surely we couldn't read too much into this.
We sustained our highest level of customer acquisition to date for the 11 day period of being off Amazon. Almost as soon as Amazon was switched back on, we saw daily customer acquisition volume return to baseline.
How incremental was Amazon?
For those unfamiliar with incrementality, the way I'd visualise in this instance is as so:
Imagine instead of this all being online, we were advertising in a city square.
Amazon has a superstore on one side of the square, and Mother Root has a retail unit on the other. We're advertising on a giant billboard that's got directions pointing to the Mother Root retail unit.
Now most people see the sign, look over at the retail unit and head in to buy. But some people think 'well, Amazon already have my card details, and my shipping details, I can go and order the same thing with one-touch.'
The question we have to ask is: how many of the sales on Amazon would have happened anyway regardless of the billboard?
And a follow on, of how many sales would have taken place if Amazon didn’t stock it – or the retail unit didn’t exist?
Until this period, we had assumed that those who were going to Amazon wouldn't have purchased if the only option was the Shopify. And that, if it wasn't for the simplicity and availability of Amazon, we never would have got that customer.
But this 11 day period signalled two things:
1. That if Amazon didn't stock it, they'd seemingly buy instead from the Shopify
2. By not being on Amazon, conversion volume went up by about 12%*
Amazon no longer looked like an okay bet. While we chalked the improved conversion volume down to the offer and FOMO, the former prompted discussion.
More sales in Shopify is strategically better than the equivalent in Amazon
Now if sales are happening somewhere, what does it matter, right?
The issue is that when a sale happens on Shopify, Meta gets the data for that in its feedback loop. When it takes place on Amazon, it doesn't.
Conversion volume on Meta is vital for efficiency, but also for learning. The more conversion volume you have, the more concurrent tests you can run, the faster you can learn, the more you can scale.
As an agency, ‘learning efficiency’ is a core metric we look at.
Our ‘Going off Amazon’ experiment
As we entered Q2 we wanted a way to test some of these hypotheses.
There's not loads of research in this area, and so we didn't have lots to fall back on.
And so we set about a test.
The core hypothesis:
By coming off Amazon, we will increase customer acquisition and CM3
The secondary hypothesis:
As a result, we will see more data in Facebook, which will increase the rate of learning. Due to the increase in Test Budget Conversions Per Week (TBCPW).
The plan was to come off of Amazon for all of May.
The results
Coming off Amazon we saw the following happen:
CM3 increased 82%
Meta experimentation rate grew by 51%
Daily acquisition volume increased by 9%
The primary metrics were all good and clear: by not being on Amazon, we had increased CM3, as well as the ability to scale.
The second order effect was that while CM3 had substantial uplift, our CPAs actually increased significantly. This was in parallel to an uptick in AOV as a result of the optimised conversion funnel for Mother Root.
Post-analysis
The eagle-eyed will notice that we've been back on Amazon since June. Ultimately we have strategic reasons for doing so, but merchandising on Amazon in relation to Shopify is now something we experiment with regularly.
Ultimately, you want to capture everyone who could potentially become a customer. If someone starts their journey on Amazon, you want to be there and prominent to capture that sale.
While this was a brilliant experiment to run, it also had its own experiment design flaws to it. The biggest of all is that using time-based comparison periods is never ideal, and a geo hold out test would have been far preferable.
A huge props to the Mother Root team on this, as well as their fractional CFO Stuart, and especially to Shaun, who runs their Amazon. We collectively learnt a huge amount and hope this invites others to test these ideas out to see what works for them. And if you want to explore a test like this and discuss it in more detail, drop me a line as I'd love to hear about it.
If you're a product company and by that I mean a brand focused on a small selection of individual products. As opposed to a marketplace company, or an inventory-driven company where you may stock thousands of items to capture sales that way.
I say basically not because the research is often conflicting, but from big studies through to smaller ones, through to data reported by reputable sources, our agency recommend either turning it off entirely, or minimising it to the fractions of pence on the pound.