• DE&I Commitment
  • Careers
  • Locations
  • Book a Meeting
    Book a Meeting
  • Company
    Learn a little more about us, our values, and our team
    Our Values
    We don't just talk the talk; we live by our core values
    About Us
    Empowering brands to realize their potential with data, insights, and technology
    Leadership
    Meet our leadership team
    Secure Data Architecture
    Our promise of data security and privacy. We keep your data safe from publishers, competitors and bad actors
    Corporate Social Responsibility
    We are committed to making a positive impact on our communities and our planet
  • Platform
    Plan, manage, optimize, and measure your campaigns with our omnichannel platform
    Our Platform
    A platform that connects all walled garden media
    Connected Media
    Create and manage campaigns across search, social, retail media and apps, in one platform
    Connected Data
    Make data-driven decisions as you plan and strategize
    See the industries we serve
    Learn how our customizable solutions can help with your unique needs
    Explore our partner integrations
    See the media, retailer, and data partners we work with
  • Clients
  • News and Events
    Check out recent announcements and see what we’re up to
    News
    Check out our recent media coverage
    Events
    Join us for our next conference or webinar
    Quarterly Trends Webinar
    Get the latest insights on digital marketing campaign performance for Q4 2022
    Retail Media for Grocery Webinar
    Join us to dig into trends and challenges shaping the industry in 2023
  • Resources
    From new releases, to industry trends and best practices, Skai has you covered
    Blog
    Read the latest insights and thought leadership from our industry experts
    Capabilities
    Take your campaigns to the next level by enhancing your platform capabilities
    Research
    Explore our reports and whitepapers so you can keep up on the latest industry trends
    Subscribe
    Sign up to get the latest updates straight to your inbox
    Quarterly Trends Report
    Learn digital advertising campaign performance trends from Q3 2022
Back to Blog

Selecting the Right Split for an Incrementality Test

Yossi Synett, Vice President, Research @ Skai™

August  03, 2018

This is the third post in our series on incrementality testing. In our first post, Omri Cohen introduced some of the reasons why advertisers are using incrementality testing and gave a brief description of what an incrementality test looks like. In our second post, Tomer Shadi dove deeper into one key motivation for the incrementality testing approach—the failure of attribution to capture incremental value driven by paid media channels. In this post, we’re going to explore incrementality testing methodology—how it differs from more common A/B testing, different approaches for creating test and control groups, and why getting the right test set up is really important.

Let’s start with a refresh on what an incrementality test is from our first post:

“An incrementality test compares the revenue or relevant KPI generated between a test group and a control group. By exposing the test group to an advertising tactic versus the unexposed control group, marketers can easily isolate the affected variables, clearly, assess immediate business impact, and formulate next steps with confidence supported by data.”

From this description we can lay out the three stages of an incrementality test:

  1. Preparation – Split some part of my addressable market into A & B groups
  2. Intervention – Expose one of the groups to a new variable that may impact performance, allowing enough time for any difference to become apparent
  3. Measurement – Examine the performance of groups A and B pre- and post-intervention to understand the impact

In this post, we’ll focus on the first stage and get you ready to select the best type of split for an incrementality test.

A/B Test vs Incrementality Test: The Crucial Difference

So far, you might be thinking that this all sounds similar to traditional A/B testing, where you might test things like subject lines, images, or landing pages to see which variant performs best.  But the truth is, an incrementality test is very different. In incrementality testing we are trying to measure the impact of the test on business-level metrics such as revenue, new customers, or site visitors. In traditional A/B tests, it’s often about media optimization and we look for the impact on more specific campaign performance metrics such as CTR, attributed conversion rate, etc. As outlined in Tomer’s post, we cannot rely on attribution to understand business-level impact and as such the performance data we use to measure impact in an incrementality test must not rely on attribution.

Measuring impact on business metrics also means we have to be very thoughtful as to how we set up our split test. In an incrementality test, the split into groups A and B should be done in such a way that the intervention performed on one group will have little to no impact on the other group. Without this guarantee, the results of the test may totally miss or greatly exaggerate the impact of the intervention. In other words, you need a clean split with minimal crossover. Let’s examine three of the most common split types used in traditional A/B testing and see which, if any, are appropriate for an incrementality test:

  • Auction split
  • Audience split
  • Geo split

For each type of split we’ll evaluate it against the criteria mentioned above as well as more general criteria that are important for all kinds of A/B test splits:

  • Ability to measure impact without relying on attribution
  • Ability to intervene in one group without impacting the other
  • Good correlation between performance metrics in both groups
  • The randomness of the split

Auction Split

How it works: An auction-based split randomly assigns a user to group A or B in real time, i.e. when they are about to be exposed to an ad. This approach is used by Google in their Drafts & Experiments product (a recently released beta version provides an option to select a cookie-based audience split as well).

Pros: Theoretically this allows for a totally random split, which is ideal from a statistical point of view, and should lead to a good correlation between the groups.

Cons: An auction-based split has one potential flaw in that the random assignment occurs every auction, so the same user can be exposed to advertising from both groups A & B.

Right for incrementality testing?: NO! This flaw rules out such an approach for any kind of incrementality testing since the probability is high that intervention in one group will have an impact on the other. Furthermore, since there’s no clean separation of users between groups A and B, there’s really no value in looking at performance data without attribution, as there’s no way to associate unattributed conversions or revenue with either groups A or B.

Audience Split

How it works: An audience split assigns users to groups A and B randomly but reproducibly such that the same user will always be assigned to the same group. This is generally done using hashed cookies or other forms of the user identifier.

Pros: Like an auction-based split, this also creates a very random split of two well-correlated groups.

Cons: There are many limitations when it comes to incrementality testing. First, the split is only as good as your testing technology’s ability to identify unique users, which is trickier in today’s multi-screen, app-filled world. Cookie-based audience splits are likely to assign multiple devices/browsers from the same user to different groups, making true audience-based splits largely possible only for publishers who have a high percentage of cross-device logins (the approach used by Facebook in their lift testing). In order to measure impact without relying on attribution, you need to be able to assign transactions to either group A or B based on the user identifier, without the necessity of a preceding click or impression. Facebook is able to make this assignment for transactions recorded by their pixel but it is not transparent—they don’t expose the user level audience assignments to allow third-party technologies to evaluate performance based on data external to Facebook. A further weakness of audience-based splits is that they cannot be used for measuring offline impacts, such as in-store or call center, or offline ads such as TV and radio. This is because it’s very difficult to reliably connect online user identifiers to offline transactions.

Right for incrementality testing?: NO! Given the significant amount of drawbacks to this type of testing, we’d have to conclude no on this one, too.

Geo Split

How it works: A geo-based split assigns users to groups using the ability to geo-target both traditional and digital marketing campaigns. Geo splits generally work at the city or DMA level—cities or DMAs are randomly assigned to groups A and B.

Pros:  Geo splits significantly simplify measurement since one can easily look at both online and offline transactions by geo without having to perform attribution. Furthermore, since they don’t rely on user identifiers, they actually have the potential to reduce the chance of intervention on one group influencing the other. A further advantage is that a geo-based split is highly transparent, meaning you can easily evaluate the results of the test against multiple data sources — even those that weren’t considered in the planning of the test. As such, a geo split is the only approach that allows you to measure halo effects such as the impact of investment in one channel on the revenue attributed to another.

Cons: Geo splits are less random than audience or auction-based splits, but using a split methodology that actively tries to create balanced and well-correlated groups overcomes this problem.

Right for incrementality testing?: Yes…if done correctly! At Skai, we’ve been using the geo-based approach for A/B testing and incrementality testing for more than four years (check out these case studies with Belk and Experian to see some of our work). We’ve applied machine learning approaches to create our own algorithm that creates geo splits with balanced and well-correlated groups. We have found the advantages of the geo-based approach to be significant in our ability to run successful tests that yield meaningful results and stand up to analytical scrutiny.

While this is certainly not an exhaustive list of splits, you won’t find another type that lends itself as well to an incrementality test as a geo split does.

Looking Ahead

Keep an eye out for our next post in this series, where we’ll focus on different methodologies for the measurement part of an incrementality test, as well as some other things to think about such as: handling conversion latency; approaches to testing multiple marketing tactics, strategies or publishers; and how to use the insights from incrementality testing to affect your business.

Interested in learning more about how Skai can help you better test, execute, and orchestrate your digital marketing efforts? Contact us today to set up a discussion.

Book a Meeting of Skai

  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share via Email
  • Copy Link
    Copied!
Tags: incrementality, Measure, measurement, Optimize, xTLx
  • Previous Post
  • Next Post

Subscribe to Updates

Media that matters.
Marketing that works.
© 2023 Kenshoo, Ltd. All Rights Reserved.
Privacy Policy. Cookie Policy. Recruitment Privacy Policy.
  • Connected Data
    • Market Intelligence
    • Our Approach
    • By Need
    • By Solution
  • Connected Strategy
    • Dynamic Marketing Mix
    • Budget Forecasting
    • Strategic Consulting
  • Connected Media
    • Overview
    • Retail Media
    • Paid Search
    • Paid Social
    • App Marketing
    • Auditing
    • Expert Services
  • Measurement
    • Incrementality
    • Experiments
    • Cross-Channel Attribution
  • Resources
    • Blog
    • Glossary
    • Case Studies
    • Training & Enablement
    • Developer Hub
Privacy Preference

We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.

Privacy Preference

Save All

Save

Accept Only Essential Cookies

Manage Cookie Preferences

Cookie Details Privacy Policy Imprint

Privacy Preference

Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

Save All Save Accept Only Essential Cookies

Back

Privacy Preference

Essential cookies enable basic functions and are necessary for the proper function of the website.

Show Cookie Information Hide Cookie Information

Name
Provider Owner of this website, Imprint
Purpose Saves the visitors preferences selected in the Cookie Box of Borlabs Cookie.
Host(s) .skai.io, skai.io
Cookie Name borlabs-cookie
Cookie Expiry 1 Year
Name
Provider Owner of this website
Purpose This cookie stores selections made by the user in the Accessibe tool in order to maintain those settings on future visits. These cookies help us make our website compliant with our obligations under US law.
Privacy Policy https://accessibe.com/privacy-policy
Cookie Name acsbState, acsbReset
Cookie Expiry n/a
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name wordpress_sec_,wordpress_test_cookie,wp-postpass_*, wordpresspass_*, wordpressuser_*
Cookie Expiry Session / 1 Year

We use these cookies to enhance functionality and allow for personalisation, such as live chats, videos and the use of social media.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Owner of this website
Host(s) .chilipiper.com, skai.chilipiper.com
Cookie Name fs_uid, CHILI_PIPER_CLUSTER, guest-session, _sp_ses*, _sp_id*
Cookie Expiry Session / 2 Years
Accept
Name
Provider Owner of this website
Host(s) .comeet.co, www.comeet.co
Cookie Name visid_incap_, nlbi_#######, incap_ses_, referrer22_00a, incap_ses_1364_2167377
Cookie Expiry Session / 1 Year
Accept
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name moduleFormPardotDownload
Cookie Expiry 30 days

Statistics cookies collect information anonymously. This information helps us to understand how our visitors use our website.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for website analytics. Generates statistical data on how the visitor uses the website.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name _ga,_ga_*,_gat,_gat_*,_gid
Cookie Expiry 2 Months
Accept
Name
Provider Hotjar Ltd., Dragonara Business Centre, 5th Floor, Dragonara Road, Paceville St Julian's STJ 3141 Malta
Purpose Hotjar is an user behavior analytic tool by Hotjar Ltd.. We use Hotjar to understand how users interact with our website.
Privacy Policy https://www.hotjar.com/legal/policies/privacy/
Host(s) *.hotjar.com
Cookie Name _hjClosedSurveyInvites, _hjDonePolls, _hjMinimizedPolls, _hjDoneTestersWidgets, _hjIncludedInSample, _hjShownFeedbackMessage, _hjid, _hjRecordingLastActivity, hjTLDTest, _hjUserAttributesHash, _hjCachedUserAttributes, _hjLocalStorageTest, _hjptid, _hjSessionUser_2229986, _hjIncludedInPageviewSample, _hjIncludedInSessionSample, _hjAbsoluteSessionInProgress, _hjFirstSeen
Cookie Expiry Session / 1 Year

Marketing cookies are used by third-party advertisers or publishers to display personalized ads. They do this by tracking visitors across websites.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Linkedin
Cookie Name lidc, li_gc, lang, AnalyticsSyncHistory, UserMatchHistory, li_sugr, bcookie, TDCPM, TDID, bscookie, ln_or
Cookie Expiry Session / 1 Year
Accept
Name
Provider Skai
Accept
Name
Provider 6sense
Cookie Name _gd_session, _an_uid, _gd_visitor, _gd_svisitor, 6suuid
Cookie Expiry Session / 400 Days
Accept
Name
Provider Pardot
Purpose Cookie name associated with services from marketing automation and lead generation platform Pardot. The visitor value is the visitor_id in your Pardot account. This cookie is set for visitors by the Pardot tracking code.
Host(s) .pardot.com, pi.pardot.com, skai.io
Cookie Name pardot, visitor_id*, lpv*
Cookie Expiry Session / 10 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for conversion tracking of Google Ads.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name IDE, 1P_JAR, NID, SOCS, CONSENT, AEC, _gcl_au, OTZ, test_cookie
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Cookie by Facebook used for website analytics, ad targeting, and ad measurement.
Privacy Policy https://www.facebook.com/policies/cookies
Cookie Name _fbp,act,c_user,datr,fr,tr,m_pixel_ration,pl,presence,sb,spin,wd,xs
Cookie Expiry Session / 1 Year

Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Wistia
Host(s) .wistia.com
Cookie Name cb_anonymous_id, _sp_ses.2b40, _li_dcdm_c, __hssrc, _gcl_au, _clsk, hubspotutk, _sp_id.2b40, __hssc, __hstc, _uetsid, _uetvid, _gid, _ga, _ga_GQR109DZ3Y, _lc2, fpi, _ex-pricing-cta, _fbp, cb_group_id, cb_user_id, _clck
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Used to unblock Instagram content.
Privacy Policy https://www.instagram.com/legal/privacy/
Host(s) .instagram.com
Cookie Name pigeon_state
Cookie Expiry Session
Accept
Name
Provider Openstreetmap Foundation, St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS, United Kingdom
Purpose Used to unblock OpenStreetMap content.
Privacy Policy https://wiki.osmfoundation.org/wiki/Privacy_Policy
Host(s) .openstreetmap.org
Cookie Name _osm_location, _osm_session, _osm_totp_token, _osm_welcome, _pk_id., _pk_ref., _pk_ses., qos_token
Cookie Expiry 1-10 Years
Accept
Name
Provider Twitter International Company, One Cumberland Place, Fenian Street, Dublin 2, D02 AX07, Ireland
Purpose Used to unblock Twitter content.
Privacy Policy https://twitter.com/privacy
Host(s) .twimg.com, .twitter.com
Cookie Name __widgetsettings, local_storage_support_test
Cookie Expiry Unlimited
Accept
Name
Provider Vimeo Inc., 555 West 18th Street, New York, New York 10011, USA
Purpose Used to unblock Vimeo content.
Privacy Policy https://vimeo.com/privacy
Host(s) player.vimeo.com
Cookie Name vuid
Cookie Expiry 2 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Used to unblock YouTube content.
Privacy Policy https://policies.google.com/privacy?hl=en&gl=en
Host(s) google.com
Cookie Name CONSENT
Cookie Expiry 6 Month

Borlabs Cookie powered by Borlabs Cookie

Privacy Policy Imprint