12 Examples of Incrementality Tests to Understand Marketing Measurement Approaches

I’ve been here at Skai for a little over a year now talking to marketers about the ongoing industry data deprecation issues, the challenges of multi-touch attribution, and of course, about how incrementality is the cookieless measurement solution marketers need.

At its core, incrementality is a pretty straightforward concept, but it’s a big word with some heavy underlying data science and can sound a little intimidating. In short, the best way to describe it is that it’s a testing methodology where a marketer makes a specific change to a Test Group in order to determine the incremental value of a marketing strategy or tactic. 

A very simple example of an incrementality test would be:

Testing the value of Facebook ads on Search ad performance

• Two equal sets of markets are chosen: A) Control Group: A set of markets with only Paid Search running (the campaign as it is today) and B) Test Group: A set of markets with Paid Search and Facebook ads running (the campaign with Facebook added in the mix)
• A test period (maybe 3-4 weeks)
• Once the test is over, it is determined that the Test Group performed X% better when running with Facebook ads

That’s a very basic example. Incrementality can be used to test a variety of channels, tactics, and elements. In my very first post here at Skai back in January, Marketing Measurement Cannot Be Reliant on Individual Customer Journeys: Incrementality is the Fix, I went into a bit more detail:

Incrementality is the experimental process, and resulting measurement, that allows a business to identify the composition of outcomes that were the direct cause of an advertising tactic, and those that would have occurred even without it as a result of other tactics or external factors.

I also offered this accompanying visual:

Incrementality requires testing. How do you choose what to test? 

While most marketers understand in theory the principles of the Test/Control methodology of Incrementality measurement, what most marketers still have a bit of trouble wrapping their heads around is what to even test in the first place.

With measurement solutions like multi-touch attribution (MTA), most of the work is upfront to connect platforms and then let the MTA tool tell you the value of each of your channels. But, with Incrementality, the marketer has to figure out an area within their program to better understand, translate that question into an incrementality test, and then run the test & analyze the results. 

12 Incrementality Test Examples

I took a stab at explaining how to think about testing design in my post, Putting Incrementality Measurement Into Action: How to Figure Out What to Test, which is a good read for those of you that want to get into the nitty-gritty.

But, today, I wanted to take a more practical approach to explain incrementality testing. Below, you will find 12 easy-to-follow types of incrementality tests that should better illustrate how marketers can use them to answer difficult measurement questions.

Testing the Impact of One Media Decision

1. New Tactic Test: Quantifying the value of adding a new channel, tactic, or partner into your media mix.

How can we better manage the testing of new display DSP partners?

How can we understand the impact of adding upper funnel video into the media mix?

What is the true impact of investing in Google’s latest product offering

2. Existing Tactic Test (Holdout Test): Quantifying the value of your existing media investments.

What impact do our current Facebook investments have on category sales?

Does retargeting help grow our business? Would we have gotten those customers anyway?

How can we better tie our upper funnel investments to revenue metrics?

3. Growth Test. What would be the impact if we increased our activity/investment?

How would increasing our Online Video budget by 50% impact registration volume?

How many more leads would we get if we doubled our Instagram spend?

Would traffic to our site be impacted if we spent more on paid search?

4. Reduction Test. What would be the impact if we decreased our activity/investment?

What would happen to repeat purchase volume if we cut back on our Display Retargeting investments?

If we slashed our upper funnel video budgets, would lead volume be affected?

How would our RoAS change if we invested 30% less in our ABM programs?

Testing the Impact of Multiple Media Decisions

5. Investment-Level Test. What is the right level of budget for maximum impact & efficiency?

How much should we be investing in Non-Brand Search to maximize our ROI?

What is the right level of investment in Snapchat to reach 100k new customers this year?

How can I figure out what my CTV budget should be next year to achieve our target of net-new customers?

6. Cross-Platform Tests. How does one channel’s activity/investment impact another channel?

How can we better understand the impact of Facebook & Pinterest on total revenue?

Does my new out-of-home marketing program increase email’s ability to drive more traffic to our website?

How much should I spend on YouTube to best increase organic search conversions?

7. Cross-Channel Tests. Measuring the impact of multiple channels working together to achieve an objective.

How can we better understand the value of our online and offline shopper marketing?

What is the interplay of organic and paid search to drive revenue on my site?

Does spending more on both email and direct mail increase conversions from my digital advertising?

8. Partner Tests. Investigate which partners work best for your programs.

How can we quantify the efficiency of our various DSP partners?

Which social advertising provider works more efficiently to generate leads? Pinterest or Snapchat?

Should I switch from one programmatic data vendor to another to get better results?

9. Audience Tests. Analyze your audience strategy effectiveness.

Do we need to be investing in both prospecting & retargeting? Which is adding greater value for us?

Is this segment targeted best with email or with social advertising?

Is the additional cost of data segmentation offset by increased results?

10. Creative Tests. Discover which creative and copy directions work best.

Which of Pinterest’s creative formats drives the greatest sales from new customers?

How well does my new promotional campaign work to generate sales?

What call-to-action is most effective at driving customers beyond the initial inquiry?

Testing Omnichannel Impact

11. Online-to-Offline Tests. 

How can we better understand the impact of Google & Microsoft shopping on both online and in-store sales?

Do my app marketing ads drive calls to my call center?

What is the impact of my Facebook ads on in-store visits?

12. Offline-to-Online Tests. 

What is the impact of our direct mail campaigns on my website’s new registrations?

Which radio spots lift sales in my online retail partners?

What is the value of my out-of-home on my app orders?

Examples of the kinds of marketing program elements that can be measured with incrementality testing 

Incrementality testing can span channels, tactics, and business outcomes. The following are just some examples of what you can test with an incrementality program. If you don’t see a marketing element that you wish to measure, just ask us—chances are it can be part of an incrementality test.

Channels, Platforms, & Tactics 

Tactics & Audiences

• Brand, Non-Brand, Competitive, Shopping
• Prospecting, Retargeting, Existing Customer, Lookalike
• Investment Level
• Pricing & Promotions
• Bidding/Optimization Algorithms, Frequency Capping
• Creative Formats & Types
• High/low performing stores or markets
• High/low value/profit products

Business Outcomes (KPIs) to Measure 

• Revenue / Purchases: 
• Online, in-store, omnichannel
• Key Products & Categories
• Purchase Funnel: Adds to cart, new customer, repeat customer, upsell
• Brand Awareness (Search Query Volume)
• Signups, Registrations, Openings
• Leads, Applications, & Calls

Get started with incrementality testing now to build your best-in-class measurement practice before cookie limitations render your MTA 

Skai’s Impact Navigator measures the real-world effectiveness, or incrementality, of a marketing tactic in the only place that matters: the real world, with real people, as part of a real marketing test measuring business responses that matter like revenue impact, client acquisition, and brand engagement.

Leave the guesses and hunches to your competition, and speed ahead with your growth strategy powered by real consumer insights.

“With Skai’s Impact Navigator, we can measure the real impact in the marketplace before starting significant investment, and focus on where there’s really incrementality.” 

          Marina Casas, Advertising Specialist – Privalia [Read the case study]

For more information or to see a brief demo of Impact Navigator, reach out today.

Request a Demo