• DE&I Commitment
  • Careers
  • Locations
  • Book a Meeting
    Book a Meeting
  • Company
    Learn a little more about us, our values, and our team
    Our Values
    We don't just talk the talk; we live by our core values
    About Us
    Empowering brands to realize their potential with data, insights, and technology
    Leadership
    Meet our leadership team
    Secure Data Architecture
    Our promise of data security and privacy. We keep your data safe from publishers, competitors and bad actors
    Corporate Social Responsibility
    We are committed to making a positive impact on our communities and our planet
  • Platform
    Plan, manage, optimize, and measure your campaigns with our omnichannel platform
    Our Platform
    A platform that connects all walled garden media
    Connected Media
    Create and manage campaigns across search, social, retail media and apps, in one platform
    Connected Data
    Make data-driven decisions as you plan and strategize
    See the industries we serve
    Learn how our customizable solutions can help with your unique needs
    Explore our partner integrations
    See the media, retailer, and data partners we work with
  • Clients
  • News and Events
    Check out recent announcements and see what we’re up to
    News
    Check out our recent media coverage
    Events
    Join us for our next conference or webinar
    Quarterly Trends Webinar
    Get the latest insights on digital marketing campaign performance for Q4 2022
    Retail Media for Grocery Webinar
    Join us to dig into trends and challenges shaping the industry in 2023
  • Resources
    From new releases, to industry trends and best practices, Skai has you covered
    Blog
    Read the latest insights and thought leadership from our industry experts
    Capabilities
    Take your campaigns to the next level by enhancing your platform capabilities
    Research
    Explore our reports and whitepapers so you can keep up on the latest industry trends
    Subscribe
    Sign up to get the latest updates straight to your inbox
    Quarterly Trends Report
    Learn digital advertising campaign performance trends from Q3 2022
Back to Blog

Paid Search Bid Strategy A/B Testing Done The Right Way

David Delaplane, Channel Strategist, Local @ Skai™

September  24, 2019

A/B testing is an important component in marketing and marketing experiments, in general, are key to getting great performance. In this article, we’ll examine some of the key testing rules you should follow to guarantee a fair and successful test.  

A/B Testing is a concept that gets kicked around quite a bit in digital advertising. The inconvenient truth is that most agencies and direct advertisers launch tests without considering the most important factors of an experiment. Why are you running an A/B Test, what should you consider, and how will you decide on a single success metric?

A/B Testing step-by-step

The first step you should take before launching any A/B test is to decide on one success metric and one constraint (if necessary). Common success metrics are revenue,  conversions, profit, or clicks. A constraint would be that the metrics directly correspond to a KPI for example; ROAS, CPA, CPC, or spend. If you do not decide on the success metric and constraint ahead of time you run the risk of your subjective feelings coming into play.

A/B Testing requires that you are only testing one variable at a time. This means that everything—and I mean everything—is constant. For example, if you’re testing Manual Bidding vs. Google’s Smart Bidding Target CPA. The only variable, in this case, would be Smart Bidding, so all of the other settings must be the same in both the test and control group—location targeting, budget, ads, keywords, etc. Next, you should take into account how much volume you are testing and is it enough data—too little and the results might not be accurate and too much isn’t necessary so there’s no need to let it run too long which can waste the budget and your time. A good rule to follow is to have at least 100 daily conversions in the entire test before the A/B split.

The next rule is to know your “True North” especially if you are testing a bid strategy. If you are testing a bid strategy, confirm that both groups are bidding toward the same conversion source. Google’s Smart Bidding traditionally uses its own pixel to track conversions, however, it’s possible to import your own conversion data into Google Ads for bidding. Remember to consider this prior to launching any bid strategy tests, “Are both bidding solutions optimizing to our True North and the same metric?”

The last rule that’s usually overlooked is having a baseline period during which no other changes are made to the control and test group. This gives you something to compare when the test is complete as you measure the lift of the test versus the control groups. It also gives you a chance to gut check the actuals post-split to guarantee traffic is, in fact, being divided between each group correctly.

During the baseline, it’s important to make sure that both groups are managed/optimized in the same way. Including new ad copy, new keywords, budgets, etc. In my experience, a two to three-week baseline will be sufficient to establish a comparison before you launch the test.  

a/b testing timeline

A/B split approaches

Now that we have covered the rules to consider before launching a test, let’s review the different split methodologies that are available to you as a marketer.

The three most common split methods are:

  1. Geographic Split (“Geo-Split”)
  2. Google Drafts & Experiments (D&E)
  3. Campaign Split

Geo splits are the most sophisticated because it can be used to measure the incrementality effect of the test, it’s transparent, and this method makes it possible for cross-publisher testing.

Google’s Drafts & Experiments is another common split method. Within this tool, there are two D&E settings: Cookie Based and Search-Based Split. Cookie Based Splits will only show one version of your campaign even if the user searches the same keyword multiple times. Search-Based Splits randomly assigns users to both groups every time they search for those keywords. This can help you see results faster, but you open yourself up to the risk of an invalid test. “Why might you ask?” Because the split randomly assigns a user to a group, it’s possible the same user could see both the test and control campaign if they search more than once. 

Ultimately, a Cookie Based Split is the preferred D&E method as it helps ensure a user isn’t exposed to both the test and control groups, which could influence the outcome of the results.

How long to run an A/B test?

Now that we have covered the testing guidelines, split methods, and settings, we can move on to a testing timeline. The truth is that A/B testing takes time. Depending on your test volume and your bandwidth, a test can take anywhere from 10 to 15 weeks. Below is an example testing timeline for reference.   a/b testing process

In this example, the test is broken into three different parts—define the hypothesis, the experiment, and the evaluation period. 

The hypothesis

Defining a hypothesis is often a step that gets forgotten by marketers. Usually, they just wait until the end and then look to see if the test group beat the control group. However, setting a hypothesis of what you think will happen is key to data-driven decision making. Once the test is complete, being able to compare the actual results to your hypothesis will help you better understand how accurate you are with regards to guessing what might happen. Over time, you can learn if you are overestimating the impact or underestimating how bid changes might help your campaigns.

The experiment period

Within the experiment period, there are three different parts:

  1. The Baseline
  2. Ramp-Up and Test
  3. Cooldown 

The Baseline is for comparison purposes and no changes should be made to either group during this time. The Ramp-up and Test is when you actually kick off the experiment. The Cooldown is not a part of the testing evaluation period. This time is designated for you to disable optimization tool you are testing on both or one of the A/B groups.

The evaluation

Once the test has been successfully disabled you can move to the evaluation stage. Was the test statistically significant? What is your risk tolerance and confidence level? Marketers have moved to a new level of sophistication and must feel confident that new bid strategies have a mathematically proven statistic before they widely adopt a new digital strategy.

How are your A/B tests going?

With over a decade in performance-based advertising, Kenshoo provides SaaS solutions for measuring your true marketing incrementality. If you are interested in learning more about how Skai can assist in identifying the true impact of your digital strategy, please contact us.

Learn More About Skai Search

Related Posts

  • What’s Going to Happen with Multi-Touch Attribution in a Cookieless World?
    Read More
  • The Evolution of Search Engine Marketing Software to Total Search Intelligence
    Read More
  • Putting Incrementality Measurement Into Action: How to Figure Out What to Test
    Read More
  • A Day in the Life of a Measurement Marketer Using Incrementality Testing (vs. Multi-Touch Attribution), Part 2
    Read More
  • Sign with the left arrow saying old way and yellow right arrow pointed that says new way.
    The Post-Attribution Era Has Begun: How to Transition Your Business to Cookieless Marketing Measurements
    Read More
  • Maximize Microsoft Automated Bidding Performance with Skai’s Budget Navigator 
    Read More
  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share via Email
  • Copy Link
    Copied!
Tags: Data, Data & Insights, Incrementality, Optimize, Plan, Research, Search, xTLx

Subscribe to Updates

Media that matters.
Marketing that works.
© 2023 Kenshoo, Ltd. All Rights Reserved.
Privacy Policy. Cookie Policy. Recruitment Privacy Policy.
  • Connected Data
    • Market Intelligence
    • Our Approach
    • By Need
    • By Solution
  • Connected Strategy
    • Dynamic Marketing Mix
    • Budget Forecasting
    • Strategic Consulting
  • Connected Media
    • Overview
    • Retail Media
    • Paid Search
    • Paid Social
    • App Marketing
    • Auditing
    • Expert Services
  • Measurement
    • Incrementality
    • Experiments
    • Cross-Channel Attribution
  • Resources
    • Blog
    • Glossary
    • Case Studies
    • Training & Enablement
    • Developer Hub
Privacy Preference

We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.

Privacy Preference

Save All

Save

Accept Only Essential Cookies

Manage Cookie Preferences

Cookie Details Privacy Policy Imprint

Privacy Preference

Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

Save All Save Accept Only Essential Cookies

Back

Privacy Preference

Essential cookies enable basic functions and are necessary for the proper function of the website.

Show Cookie Information Hide Cookie Information

Name
Provider Owner of this website, Imprint
Purpose Saves the visitors preferences selected in the Cookie Box of Borlabs Cookie.
Host(s) .skai.io, skai.io
Cookie Name borlabs-cookie
Cookie Expiry 1 Year
Name
Provider Owner of this website
Purpose This cookie stores selections made by the user in the Accessibe tool in order to maintain those settings on future visits. These cookies help us make our website compliant with our obligations under US law.
Privacy Policy https://accessibe.com/privacy-policy
Cookie Name acsbState, acsbReset
Cookie Expiry n/a
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name wordpress_sec_,wordpress_test_cookie,wp-postpass_*, wordpresspass_*, wordpressuser_*
Cookie Expiry Session / 1 Year

We use these cookies to enhance functionality and allow for personalisation, such as live chats, videos and the use of social media.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Owner of this website
Host(s) .chilipiper.com, skai.chilipiper.com
Cookie Name fs_uid, CHILI_PIPER_CLUSTER, guest-session, _sp_ses*, _sp_id*
Cookie Expiry Session / 2 Years
Accept
Name
Provider Owner of this website
Host(s) .comeet.co, www.comeet.co
Cookie Name visid_incap_, nlbi_#######, incap_ses_, referrer22_00a, incap_ses_1364_2167377
Cookie Expiry Session / 1 Year
Accept
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name moduleFormPardotDownload
Cookie Expiry 30 days

Statistics cookies collect information anonymously. This information helps us to understand how our visitors use our website.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for website analytics. Generates statistical data on how the visitor uses the website.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name _ga,_ga_*,_gat,_gat_*,_gid
Cookie Expiry 2 Months
Accept
Name
Provider Hotjar Ltd., Dragonara Business Centre, 5th Floor, Dragonara Road, Paceville St Julian's STJ 3141 Malta
Purpose Hotjar is an user behavior analytic tool by Hotjar Ltd.. We use Hotjar to understand how users interact with our website.
Privacy Policy https://www.hotjar.com/legal/policies/privacy/
Host(s) *.hotjar.com
Cookie Name _hjClosedSurveyInvites, _hjDonePolls, _hjMinimizedPolls, _hjDoneTestersWidgets, _hjIncludedInSample, _hjShownFeedbackMessage, _hjid, _hjRecordingLastActivity, hjTLDTest, _hjUserAttributesHash, _hjCachedUserAttributes, _hjLocalStorageTest, _hjptid, _hjSessionUser_2229986, _hjIncludedInPageviewSample, _hjIncludedInSessionSample, _hjAbsoluteSessionInProgress, _hjFirstSeen
Cookie Expiry Session / 1 Year

Marketing cookies are used by third-party advertisers or publishers to display personalized ads. They do this by tracking visitors across websites.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Linkedin
Cookie Name lidc, li_gc, lang, AnalyticsSyncHistory, UserMatchHistory, li_sugr, bcookie, TDCPM, TDID, bscookie, ln_or
Cookie Expiry Session / 1 Year
Accept
Name
Provider Skai
Accept
Name
Provider 6sense
Cookie Name _gd_session, _an_uid, _gd_visitor, _gd_svisitor, 6suuid
Cookie Expiry Session / 400 Days
Accept
Name
Provider Pardot
Purpose Cookie name associated with services from marketing automation and lead generation platform Pardot. The visitor value is the visitor_id in your Pardot account. This cookie is set for visitors by the Pardot tracking code.
Host(s) .pardot.com, pi.pardot.com, skai.io
Cookie Name pardot, visitor_id*, lpv*
Cookie Expiry Session / 10 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for conversion tracking of Google Ads.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name IDE, 1P_JAR, NID, SOCS, CONSENT, AEC, _gcl_au, OTZ, test_cookie
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Cookie by Facebook used for website analytics, ad targeting, and ad measurement.
Privacy Policy https://www.facebook.com/policies/cookies
Cookie Name _fbp,act,c_user,datr,fr,tr,m_pixel_ration,pl,presence,sb,spin,wd,xs
Cookie Expiry Session / 1 Year

Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Wistia
Host(s) .wistia.com
Cookie Name cb_anonymous_id, _sp_ses.2b40, _li_dcdm_c, __hssrc, _gcl_au, _clsk, hubspotutk, _sp_id.2b40, __hssc, __hstc, _uetsid, _uetvid, _gid, _ga, _ga_GQR109DZ3Y, _lc2, fpi, _ex-pricing-cta, _fbp, cb_group_id, cb_user_id, _clck
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Used to unblock Instagram content.
Privacy Policy https://www.instagram.com/legal/privacy/
Host(s) .instagram.com
Cookie Name pigeon_state
Cookie Expiry Session
Accept
Name
Provider Openstreetmap Foundation, St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS, United Kingdom
Purpose Used to unblock OpenStreetMap content.
Privacy Policy https://wiki.osmfoundation.org/wiki/Privacy_Policy
Host(s) .openstreetmap.org
Cookie Name _osm_location, _osm_session, _osm_totp_token, _osm_welcome, _pk_id., _pk_ref., _pk_ses., qos_token
Cookie Expiry 1-10 Years
Accept
Name
Provider Twitter International Company, One Cumberland Place, Fenian Street, Dublin 2, D02 AX07, Ireland
Purpose Used to unblock Twitter content.
Privacy Policy https://twitter.com/privacy
Host(s) .twimg.com, .twitter.com
Cookie Name __widgetsettings, local_storage_support_test
Cookie Expiry Unlimited
Accept
Name
Provider Vimeo Inc., 555 West 18th Street, New York, New York 10011, USA
Purpose Used to unblock Vimeo content.
Privacy Policy https://vimeo.com/privacy
Host(s) player.vimeo.com
Cookie Name vuid
Cookie Expiry 2 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Used to unblock YouTube content.
Privacy Policy https://policies.google.com/privacy?hl=en&gl=en
Host(s) google.com
Cookie Name CONSENT
Cookie Expiry 6 Month

Borlabs Cookie powered by Borlabs Cookie

Privacy Policy Imprint