Smoke testing is the most customer centric thing you can do
On MasterClass' recent smoke test
About a month ago I got a series of ads for a language learning AI tool.
I’m actively learning French at the moment and so the ads were relevant.
After the ad was a salesy landing page. I entered my card details and bam, I got the above screen. The “product is not currently available.”
This “product” was a smoke test by MasterClass: a business whose annual recurring revenue is now forecast at nearly $150m. A smoke test is a landing page for a product or feature that doesn’t exist.
Many people dislike the smoke test. They believe it is bad for trust, bad for brand, and creates a bad experience of value for the customer.
In fact, my view is these are one of the most customer-centric things you can do.
Most startups default die
We first need to remember what most startups and challenger brands are.
They are businesses that aren’t yet profitable.
They are businesses that have likely taken on investment – and therefore there is an expected outsized level of growth to deliver returns to those investors. That means to survive is to grow.
They are businesses often doing new things where there is less predictability.
Even for those that have found significant success – maybe they are doing tens or even hundreds of millions of pounds of annual revenue – if the business just carried on exactly as it is, would eventually decline.
Startups and challenger brands permanently need forward momentum to exist. Stasis = death.
This means your time in business is limited. Whether your runway is three months or three years, you have limited time and resource. You have a set number of attempts to make it work. There are only so many arrows in the quiver.
Customer value = business value
The second thing to remember is why you’ve founded your company.
Whether you’re creating a protein crisp company or legal tech saas, you grow by creating value for your customers. Whether that’s providing a moment of happiness or saving the customer thousands of pounds. Creating customer value creates business value. (And chance of success).
Let’s say you’ve got 12 months runway left.
Maybe in that time you could go and build 2 or 3 big new innovations in that time.
If you’re a consumer tech business, maybe that’s 3 big new features that product + engineering work on. If you’re a consumer goods company, maybe that’s three new flavours, or parallel product launches.
Smart builders listen to customer problems and create solutions for them. But success is rarely predictable. And even in the most customer-centric of businesses, I’ve seen features created – features that were heavily requested – only to be barely used and ultimately a waste of time.
What people say they want does not always correlate to what they actually use or pay for.
Now if you don’t get those 3 decisions right, you run out of runway, and your business either dies or changes significantly. I know this first hand (read my experience with this here).
If your business no longer exists, how much trust are you building then? How much affinity with the brand? How much customer happiness can you create?
As a first fundamental rule, your business needs to exist to be able to create any joy for customers.
Why smoke testing is the best thing to do for customers
Smoke testing – and more broadly, experimentation – therefore is the best way to create value for customers. Because it allows you to focus on things that really matter for your customers.
One of two things would have happened for the MasterClass test.
The smoke test delivered affordable customer acquisition.
That indicates that the market really wants this product or feature.
It gives them an indicative cost of acquiring customers, so you can understand how the unit economics would need to stack up to make it work.The smoke test delivered expensive customer acquisition.
That indicates the market doesn’t care for the product
It gives them indication of a much higher level of LTV you would need to make the unit economics work.
If it’s good acquisition, then the upside is you’ve found a product people want. You can now focus resources on building it as quickly as possible, armed with the data to tell you how important it is for the world. You create customer value that creates business value.
If it’s bad acquisition, the upside is you can scratch this product off your roadmap. Or deprioritise it behind other things that more people want. You create customer value that creates business value. You’ve just saved 3 months of time and cost building something no-one wants and increased your chance of survival.
The downside for both is you’ve pissed off a selection of customers. When I run a smoke test with clients, it’s aiming to get 50 customers onto it (the amount to get good CPA on Meta). I’m sure with MasterClass’ budget they did higher to get more statistical data.
Let’s say it was a big number like 1,000. They’ve pissed off 1,000 people.
That is a good thing. If pissing off 1,000 people means they’ve now got more chance of servicing the hundreds of millions of potential customers – fulfilling their mission to help anyone learn from the world’s best – then they’ve done right by the customers.
Maybe it’s a version of the trolley problem, but would you piss off 1,000 people if it meant you could make 100m people happy?
Go build a smoke test today
Any marketer will tell you how difficult it is to build and maintain attention. That’s a curse for marketers, but it’s also a blessing for building companies. Today’s news is tomorrow’s fish and chip paper.
If you’ve got a new product or feature on your roadmap that you’re about to start building, or maybe you’ve already begun, then stop and go build a smoke test.
Spend resource in making the best landing page you can. Put money into a series of ads. Then put enough spend to be able to acquire 50-100 customers into the product. Take their payment details so that you really know they want it (checkouts/carts don’t correlate to purchase). And then use that data to inform your journey.
Pissing off 50 people might just be the most customer-centric thing you can do.