The startup guide to marketing measurement
Our measurement pyramid for startups of all stages
If you’ve ever got an hour to kill, ask a couple of marketers how their work should be measured. Measurement is one of those wedge issues in marketing like how performance and brand should play against one another, or whether it’s more important to talk to customers or look at analytics reports.
Doing this stuff is important.
I had a conversation last year with a CEO who saw marketing only as a cost. “If you don’t see it as a revenue generator, then you must never have seen proper marketing,” I replied. But that is the reality of how many view it.
But when you’re a startup, you’ve got two stages of being. (1) Find product market fit, then (2) grow as fast and big as your unit economics allow. Is there time to get bogged down with measurement?
Well, this guide is designed to help you understand what measurement practices to adopt at different stages of your company life.
The majority of startups I’ve worked with sit within the £0-10m/year of revenue stage, but I’ve also worked with a select number of brands working from £10-500m. This guide takes into account the practical experiences of measurement at all of these different stages.
Before we get into it, a few words on what this is not. This is not a guide to metrics or KPIs. I won’t be going into why you should be measuring CPA or CM3 or how to think about margins. This is not an accounting guide.
This guide also assumes that all marketing you do is designed to sell something. (Hint: every piece of marketing you do should have the intent to sell. You’re either selling today or selling tomorrow. If you’ve got no interest in selling, then you should probably get out of marketing).
And if you ever want to kill an hour, I’m one of those people who will happily chat measurement with you all day long.
This post is split into two core sections: a brief history of marketing measurement, and then the modern marketing measurement pyramid.
A brief history of marketing measurement
“Half my advertising spend is wasted; the trouble is, I don’t know which half.”
Marketing measurement has come a long way. And every wave of technological development has brought about new eras of change.
Prior to the 1960s, those who did measure used a mixture of focus groups and market research to understand what people said about ads.
But the changing global climate in the 60s (recession, war, radical social change – sound familiar?) pushed pressure into a new focus on ROI.
The computer brought with it new abilities to segment customers and analyse data. Previously this would have taken too many human hours. Now it could be done digitally.
Alongside this, as marketers became more sophisticated, there was the rise of what a 1966 McKinsey article called “test marketing”1. This, as McKinsey’s John D. Louth wrote, could help companies analyse new products, new packaging, and sales incentives. Testing sales incentives in one market and not another, is recognisable as a form A/B testing. This falls under the banner of what we’d today call ‘incrementality tests.’
Incrementality is one of those pieces of marketing jargon that annoyed me for years. All it means is ‘did this activity do the thing we expected?’ You might ask why this needs to be a word at all, but there we go.
The 1950s was the decade that the so-called Four Ps were introduced: Product, Price, Place, and Promotion. This was an academic introduction into marketing, that gave marketers remit to each area. Those who have been making the distinction of working in ‘growth’ and not just ‘marketing’ may see some overlap here. Growth practitioners are often united in the access to ‘Product’ in their workflows. Turns out we’re just borrowing a 70-year-old formula.
The other big parallel movement of measurement was the advance of econometrics in marketing. A branch of economics, it allowed for basic regression analyses to measure the impact of marketing activity on sales.
By the 1970s, statisticians at the University of Chicago took this further by developing the first approaches of marketing mix modelling (MMM). These models continued to develop and improve as with the advent of technology.
In part, this was a data challenge. By the 1980s, point-of-sale scanner data meant there was data to track sales countrywide. This could begin to be tied back to marketing campaigns in MMM.
By the 90s, computers that were usable by everyone sat on every office desk around the world. With that came stats, data, and spreadsheet software that gave this analytical ability to anyone.
It all changed in the 2000s
The internet changed everything for three reasons.
First, the rise of e-commerce, meant that people could buy things at home easily. There was no need to go to a store or fill in a form on a mail-order catalogue.
Second, the levels of data that Google and Facebook had meant that they could predictively find people who were in-market right now.
Third, the way the internet was until the late 2010s meant you could track those journeys very easily.
These three things combined were important.
In the 1990s, if you were selling cat food, you had to do some TV advertising. You’d need to be ‘always on’ because not everyone was in-market to buy that cat food at the exact same time.
As a customer, at any given point in the month, I may just have bought some cat food and so had no need to think about any. And my TV or newspaper had no idea what I was thinking about at the time. When it was time to go buy some, I had to go to shop and then remember which brand I wanted.
“This turned a large part of marketing into a job of maths and technical aptitude over one of psychology and creativity.”
Post-Facebook, post-e-com, post-data. Facebook would get those ads in front of me at the time I wanted to buy. Relevancy would be high. Facebook could begin to determine using millions of signals who wanted cat food. And then I could just click through and buy, removing the need to have to remember your brand or advertising.
Byron Sharp’s go-to text How Brands Grow was released right at the beginning of this digital advertising era.
Two of the key concepts in that book were the ideas of mental availability (the act of staying top of mind with your customer) and physical availability (being present in-shop when you are ready to buy).
As long as you sold online, you didn’t need to worry about physical availability, because the shop was a click away. And mental availability was less important because you could rely on technology increasing relevance for your ads.
In the early 2010s, there were few mainstream privacy concerns online. Tracking was easy.
This was the real era of attribution taking off. We had attribution before of course. Coupon redemption was a form of attribution. But it exploded with the internet.
If you spent £10,000 on Google Ads, you could see how many people clicked the ads and went on to purchase. This was and remains revolutionary.
You can also see why it suits technologists. The precise logic of putting an advert live, and then immediately tracking back a sale, with raw data behind it is appealing. This turned a large part of marketing into a job of maths and technical aptitude over one of psychology and creativity.
It also propagated the idea of a ‘single source of truth’. The notion that if you can just get the data stack right, there’s one trusted way to measure everything.
If the 1950s started with barely any measurement, the 60s through to the 00s were decades of advance in trying to prove out that the marketing work we did was having an impact. The 10s became a decade of extremely narrow focus: attribution was the only game in town.
Then came the privacy wave (iOS14.5, cookies, etc)
Everything changed over the last few years. People started to question the data these platforms had. Revolts and backlashes happened. And now our browsers and computers and operating systems all have built-in privacy measures.
This creates a negative feedback loop:
The data being fed back to the platforms is reduced
And so, the platforms are less likely to understand if an ad is effective
And so, their targeting becomes much worse
The platforms have done a lot to improve things, but the reality is we aren’t ever going back to the ‘old way.’
The technology change isn’t the only thing though.
The consumer is now technologically more aware.
There are more digital-first brands springing up every day.
And the bigger, more established brands, who previously ignored e-com, are everywhere too.
Even the luxury sector – a sector designed to add friction to the buying journey, and one that has private entrances in shops for the 0.001% – advertises on Instagram. Just yesterday I saw an ad for Loro Piana. That would have been unthinkable eight years ago.
And while during the 2010s, we saw the advent of ‘DTC’ as a business strategy, the last few years have revealed that DTC is just a channel. And so buying journeys have again broadened out.
There are of course people who are in-market right now, ready to buy, preferably on your online store, and they just need to see your ad. But it’s harder to reach them. And the majority of consumers aren’t in this state of mind right now.
As a result, our approach to marketing online had to change. And the 2010s era of attribution-only measurement has (thankfully/sadly, depending on your outlook) come to an end.
The modern pyramid of measurement
Let’s be clear, there is no one solution to marketing measurement.
There is no single source of truth. Depending on where you are in your company journey, you’ll likely need to employ a mixture of all three core measurement methods.
The half-century that ran from 1960 to 2010 was defined by its advancements in technology and the broad exploration of measurement methods. The 2010s were defined by the narrowing of those practices – but we’ve now learnt that narrowing was temporary.
Today, that pendulum has swung back and our approach has had to broaden again.
We are in fact entering a golden age of measurement.
We no longer dismiss the measurement industry’s history. But we also bring into it the technological practices of the last ten years. We use marketing mix modelling, but now it’s powered by machine learning.
The way we think about measurement at Ballpoint is in a pyramid. Three layers where you’ll start at the bottom and work up. Every company will likely do some form of all three, but their complexity will grow over time.
At the bottom of the pyramid is attribution. This is the first layer you’re likely to need to get firing. Attribution today is a brilliant way to measure activity within a channel.
Then you have incrementality testing. Put simply: does this action cause the desired reaction?
And finally, you have modelling, which looks at the impact multiple channels have together.
Below are the different ways to think about each, with notes on how to think about them during different stages of the startup journey.
Attribution
Attribution is almost always the very first place to begin with measurement. Attribution aims to put value to a piece of marketing activity. And the complexity comes when thinking about all of the touchpoints in that customer journey.
Last-click, which is many people’s first introduction to measurement thanks to Google Analytics, assumes the final touch point is the most important. If you click a Google Ad for the product, go and purchase, then you take your Google spend, divide it by those purchases, and arrive at your last-click CPA.
First-click, by contrast, would assign all credit to the very first part of that journey. If your billboard was the first place the customer saw your product, then the billboard gets all credit.
If you’ve got multi-million-pound budgets, you’ll know that logically both the billboard and the Google Ad will have had some contribution to a purchase.
But most brands today start digital-only. And most brands start single marketing channel and single sales channel only.
If the only marketing you’re doing is Pinterest Ads to your Shopify store then you can likely assume that for most customers those touch-points are the same. These early days are simple and logical and rational. You can see that spending £5k, then £25k and then £100k increases your number of sales.
The difficulty arises as you start to broaden your channel mix, and suddenly the journey isn’t as simple.
Meta and Google have both done lots to develop their attribution models in recent years. By default in Meta, you’ll be measuring ‘1 day view, 7 day click’ – this means if someone purchases within either 7 days of clicking your ad, or 1 day of viewing your ad, Meta will attribute that sale back to the ad.
And then you have multi-touch attribution (MTA) models, which try to weigh all different digital points together. One approach is to split a conversion’s responsibility between all channels (linear modelling), another is to weight it with increasing importance towards the last-click. Implementing MTA is usually tough, requires lots of data, and still has many limitations.
Advantages and disadvantages of attribution
Attribution’s core benefit is speed. You can attribute live in most marketing platforms. Open up FB Ads every hour and you’ll see how your performance is evolving throughout the day.
Attribution’s biggest disadvantage is understanding incrementality of those channels.
Just because Google says it’s a £32 CPA, doesn’t mean it is. In fact almost all channels will either over or under attribute.
Even multi-touch attribution models are deeply flawed and so do not give a full understanding of channel performance.
As a general rule of thumb, Meta and TikTok under-attribute, and Google and Microsoft will over-attribute.
I’m an early-stage brand, what should our attribution approach be?
At Ballpoint, we always recommend that attribution works brilliantly for comparing in-channel activity to other in-channel activity.
I.e. if you want to measure the difference of impact between two campaigns or ads in Facebook, then checking the comparative CPAs of those is a great way to do so.
However, that CPA isn’t the whole story. It doesn’t take into account the user journeys where the ad had the impact, even though the data wasn’t there.
In short:
Use Facebook CPAs to measure Facebook activity
Use Google CPAs to measure Google activity, and so on…
Do not take those CPAs as the whole story – there will be other activity that comes from those ads that isn’t attributed
Some channels likely over-attribute (Google Ads)
and some channels likely under-attribute (see TikTok Ads)
Incrementality testing
Incrementality testing has existed for decades. When a brand would run TV advertising in the state of Ohio but not in Michigan, that was an incrementality test.
How do you measure it? Run the activity and the holdout activity, and then measure the impact on sales in two regions. If you spend $1m on TV in Ohio and $0 in Michigan, you’ll want to see $1m worth of purchase behaviour in the former.
It wasn’t just TV, but print, DM, out of home, radio, these can all be tested with incrementality testing.
The internet brought lots more incrementality testing. The explosion of A/B testing – whether in your email marketing or with website conversion rate – is baked into the idea of incrementality testing. What is the true impact of that piece of activity?
When it comes to the ad platforms, they have a brilliant ability to exclude audiences at the source. Meta and Google both have conversion lift capabilities. These split an audience, exposing an experiment cohort to ads, and a control cohort to no ads. The difference between the two is the true incremental value of your activity.
That said, these tools are only available to certain advertisers, at a certain scale, with account reps. They are also expensive to run: you need to run activity for set periods, with holdout groups or control groups. That means likely less efficiency during that test period. This is the price of accuracy.
Advantages and disadvantages of incrementality testing
The core benefit of incrementality testing is knowing that the activity you did was responsible for the action it says it did. It is the most accurate way of understanding this.
The biggest downside is that to properly run this on acquisition channels, is usually fairly expensive to do accurately. Getting access to the relevant tools on Meta or Google relies on having account reps. Or building complex statistical models.
I’m an early-stage brand, what should our incrementality approach be?
First, it’s easy to do incrementality testing when you own more of the data. With your email marketing, you can easily run these tests. Split an email list in half, give one half one journey and the other another, then measure the incremental difference in revenue between the two.
Second, you can run lightweight incrementality tests by splitting locations up. Want to test a new marketing channel? Why not only run the activity in Scotland for the first month. Then look at your Scottish data after that month, if you’ve got a target CPA of £30, and you spent £20k, can you identify 666 purchases in Scotland that you couldn’t in England?
As you get bigger, there are other tools available to you. Once you are multi-channel, and spending high five or low six figures a month or more on marketing, you can start to run proper experiments. If you have an account rep, then you can run conversion lift tests. And if you don’t, you can run these using an R package like Meta’s GeoLift.
Modelling
Over the last few years, you’ll have no doubt heard the phrase ‘marketing mix modelling’ (MMM) come to the foreground. MMM is a statistical analysis technique used to measure the impact of a whole marketing mix including activities like discounts and pricing (hello Four Ps). You can set any number of inputs to it to understand how each one played its part.
MMM models historic data to give you an understanding of how these different elements play together.
MMM is great for providing the broadest picture possible. But it’s also not relevant until you’re big. Put simply, if you’re running just one or two channels, then do not waste your time with MMM. You should be able to identify the impact of those channels with incrementality testing.
If you are a later-stage brand, then embedding MMM is the final part of the pyramid. You can either build an MMM using open source (like Meta Robyn), or purchase it from one of the various vendors who have sprung up in recent years.
MMM isn’t the only form of modelling. There’s two ways early-stage brands can think about modelling their marketing effectiveness before they get to MMM stage.
Blended CPA
If you spend £30k on marketing in a month, regardless of channel, you will want to see that the activity is doing something.
The simplest way here is to understand a blended CPA. Take all that marketing spend and divide it by the number of new customers in a month.
Increase the spend by 50% the following month and re-measure. Did your new customers increase by 50% as well? If so happy days, try increasing spend again.
Blended CPA is rudimentary but a vital part of early stage measurement. It’s particularly important because it recognises the nuance of channel attribution, before you get to a stage you can afford incrementality tests.
Basic modelling using post-survey
The ‘how did you first hear about us’ question is a great basic way to understand cross-channel effectiveness. Ask this question at the end of a survey and start to understand the blend of where users are first hearing about you.
You can then use this to run further experiments afterwards. Does PR account for 15% of your this response, but only 2% of marketing budget? What happens if you try tripling PR budget while keeping other channels steady, does this increase the % here?
Advantages and disadvantages of modelling
True modelling (as with MMM) has the biggest advantage of demonstrating cross-channel effectiveness of your efforts. Its biggest downside is the amount of data, time, and money needed to do so.
I’m an early-stage brand, what should our modelling approach be?
The lighter touch modelling methods, however, are worth introducing early. They’ll give you a good steer to then experiment in other ways.
I’d be looking at blend CPA from day one, and introduce the post-survey question as early as possible. As your marketing activity shifts over time start to model how that response changes.
Closing thoughts
Marketing measurement is not perfect.
There is no single source of truth.
It is important to build each element of our pyramid into your measurement approach. Start at the bottom: get attribution in place first. Run simple A/B experiments where you can. And blend your channels simply to understand overall CPA.
As you grow bigger, ensure you’re not increasing measurement complexity for the sake of it. Sooner or later, this becomes someone’s entire job, and then soon an entire team’s job. This doesn’t free you up to do more marketing, it usually does the opposite.
If you’re a consumer brand in the UK with strong PMF, you’ll likely be able to get £5-10m per year with one or two channels. You don’t need MMM when you’re only running Meta and Google Ads.
It’s also something I’d factor in when thinking about complexity in your overall strategy.
Being single sales channel (i.e. Shopify) is hugely advantageous because it reduces complexity. As Charles Instone said in the interview2 I ran with him, selling in multiple vendors only complexity. It’s harder to measure, journeys are muddier.
If you’ve got a measurement question and want to chat it through, then please do drop me a line. This is one of my favourite topics.
https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-changing-face-of-marketing
