• DE&I Commitment
  • Careers
  • SIGN IN
  • Book a Meeting
    Book a Meeting
  • Company
    Learn a little more about us, our values, and our team
    Our Values
    We don't just talk the talk; we live by our core values
    About Us
    Empowering brands to realize their potential with data, insights, and technology
    Leadership
    Meet our leadership team
    Secure Data Architecture
    Our promise of data security and privacy. We keep your data safe from publishers, competitors and bad actors
    Corporate Social Responsibility
    We are committed to making a positive impact on our communities and our planet
  • Platform
    Plan, manage, optimize, and measure your campaigns with our omnichannel platform
    Omnichannel Advertising
    A platform that connects all walled garden media for more effective campaigns
    Data Connectivity
    Make data-driven decisions as you plan and strategize
    Product Capabilities
    Take your campaigns to the next level by enhancing your platform capabilities
    See the industries we serve
    Learn how our customizable solutions can help with your unique needs
    Explore our partner integrations
    See the media, retailer, and data partners we work with
  • Case Studies
  • News and Events
    Check out recent announcements and see what we’re up to
    Newsroom
    Check out our recent media mentions and announcements
    Events
    Join us for our next conference or webinar
    Omnichannel platform launch
    Read the press release to see how we're helping marketers win the walled gardens
    Cannes Lions 2023
    Don't miss your chance to meet with Skai leadership in Cannes
  • Resources
    From new releases, to industry trends and best practices, Skai has you covered
    Marketer Insights
    Read the latest blog posts and thought leadership from Skai and other industry experts
    Skai Research
    Explore our studies, reports and whitepapers so you can keep up on the latest industry trends
    Quarterly Trends Report
    See the key digital marketing trends from Q1 2023
    The Breakthrough
    The only resource hub built for marketers in the new era of omnichannel
Back to Blog

The Road to a Test-and-Learn Organization Part 1: Learn from the Netflix Testing Process

Tom Affinito, VP Corporate Development @ Skai™

December  19, 2019

Most marketers know that being more data-driven is the (not so) secret ingredient to success, but often struggle with how to do this consistently. In a recent article, Data-Driven Marketing: The Secret is a Test-and-Learn Culture, I discussed how hard data-driven marketing is and that it can only really be done well when the entire organization is committed—from top to bottom—on elevating the use of data as the key to best-in-class decision-making.

Testing

For marketing leaders, there might not be a more important mission than setting up their teams to be successful data-driven marketers. But, operationalizing a dedicated data & insights organization is not so simple. You can’t just flip a switch. A strong foundation with a solid test-and-learn approach is required. 

To offer a blueprint on how to proceed, I outlined in my last post the Four Pillars of a Test-and-Learn Organization:

  • Test-and-Learn Organization Pillar #1: An organization’s testing process must include policies to support test prioritization, execution, and management.
  • Test-and-Learn Organization Pillar #2: An organization’s learning process is focused on actionability and decision-making
  • Test-and-Learn Organization Pillar #3: A test-and-learn culture that aligns the team and enforces accountability
  • Test-and-Learn Organization Pillar #4: A clear vision for the types of strategic & operational objectives to be tackled

Even armed with these four pillars, turning your team into a true test-and-learn organization won’t be an easy road to take—but it’s definitely the right path to choose.

To better understand these four pillars, we will highlight them in separate posts.

Today, we’ll focus on Pillar 1, The Testing Process and use Netflix’s excellent internal testing methodology as an example of how this can be done well. 

Netflix’s commitment to testing excellence required it to reorganize teams

Netflix is a leader in demonstrating how a test-and-learn mindset informs better business decisions through an accurate understanding of customer insights and preferences. In less than a decade, Netflix has moved from a fledgling service mailing DVDs to one of the most admired tech companies in the world. 

To broaden the impact of test-and-learn processes, the ability to propose ideas for measuring has to be available to everyone across the organization. But most marketers are not trained in data analysis nor experienced in creating measurable hypotheses on a daily basis.

Most organizations try to solve this by creating an analytics center of excellence (COE) that every team can tap into. Even though a dedicated analytics COE is a good start, the downside is that most of the organization won’t learn the principles of how to extract insight from data, but rather rely on others to come in and do it for them.

For Netflix, being more data-driven was such an important initiative that they tore down siloed business intelligence groups and instead embedded data scientists, data engineers, and data analysts inside each business unit. 

Now every business team has dedicated specialists who focus on the data and the opportunities to improve data-driven decisions. Now every business team can actively propose ideas for experimentation and can share the best applications for newly discovered insights.

(from “Enabling a Culture of Analytics”)

Netflix had so many tests that they created a system to prioritize

Testing as a primary approach to decision-making is such a major focus that sometimes there are simply too many tests to run. Experiments can be disruptive to the daily flow of business, require too many resources, or overlap with other tests and taint the final results. Another consideration is which tests to do first as the insights learned from some can be used to optimize future tests for even better insights. So, all of these tests—however helpful and valuable they maybe—can also be chaotic to manage.

How did Netflix solve this issue of testing prioritization? With machine-learning.

As explained in the Netflix technology blog: 

“With a collection of tests that, by nature, are time-consuming to run and sometimes require manual intervention, we need to prioritize and schedule test executions in a way that will expedite detection of test failures… In our quest to be objective, scientific, and in line with the Netflix philosophy of using data to drive solutions for intriguing problems, we proceeded by leveraging machine learning.”

While not everyone can build their own machine-learning system to evaluate which tests can be run in parallel and which must be run before others, the principal factor behind test prioritization can be understood by everyone: focus on those tests which will reveal insights about the needs and pain points of customers. 

This focus is the foundation of agile market positioning and lean business strategy, and Netflix’s solution helps to keep teams from spending too much time on testing internal improvements at the expense of understanding their core audience.

Netflix’s testing processes drive actions and accountability

To increase the benefits of experimentation, Netflix critically manages both test execution and the deployment of insights. “The Netflix Marketing team embraces experimentation to identify the best marketing tactics for spending paid media dollars…” says the technical team. “Our teams use experimentation to guide their instincts on the best performing campaigns.” 

This requires oversight to guarantee experiments successfully run to completion or are quickly cancelled if they run into quality issues. Experts look at the incoming signals to ensure that the volume of data is high enough to yield the required accuracy levels. Deviations from predicted responses are examined for local market interference from regionally competitive product or service offers. Seasonality impacts and influences are monitored to correct expectations for daily baselines.

Additionally, when the experiment has completed, the results are evaluated for business impact. For failed tests, what is the next hypothesis to be tested? For successful tests, how should marketing processes be optimized as a results of the new insights?

Netflix marketing analysts take every new result from a marketing test and examine whether that learning can improve each and every stage of their campaign management systems.

Five sets of campaign management systems are the most common targets for benefiting from the results of these marketing experiments:

  1. Media planning systems which are responsible for automating the workflows used for buying paid media and unlocking efficiencies in those flows (media planner)
  2. Creative management systems that help with creative development & localization and assembly of ads from the creative assets.
  3. Campaign management systems which are responsible for marketing campaign creation and execution 
  4. Advertising insights systems which are responsible for collecting analytics and insights into how our campaigns are performing on ad platforms like Google 
  5. Ad budget optimization systems that enable changing budget on live campaigns in order to optimize our marketing spend

Conclusion: excellence over a range of testing processes is required to maximize the impact of data-driven insights

The strategic approach used by Netflix relies on

  • Accurate and insightful data-driven decisions,
  • Supported by a broadly enabled test-and-learn culture that
  • Democratizes hypothesis making
  • Prioritizes test execution, and
  • Manages execution and insights across the business teams.

Netflix is an exemplar of how a test-and-learn culture guides business. Instead of taking chances with blind strategic bets, Netflix has instead focused on decision effectiveness, and become a market leader by maximizing opportunities from newly discovered consumer insights.

Netflix is so committed to testing, that it even has a Netflix Research blog. “Data-driven decision making and the Netflix culture of experimentation extend from our data scientists through all levels of the company to Reed Hastings himself. Hypotheses and results are subject to rigorous analysis plus robust and open debate from a wide variety of viewpoints. Our executives make time to understand experimental methods and test results, and their close interaction with data scientists helps ensure that our decisions are sound from both statistical and business perspectives.”

For marketing and business leaders reading this post, the road to data-driven maturity won’t be easy, but it offers the surest approach to success.

Book a Meeting of Skai

Related Posts

  • Test and Learn Marketing Can Guide Your Way Through Even the Toughest Market Conditions
    Read More
  • Putting Incrementality Measurement Into Action: How to Figure Out What to Test
    Read More
  • Team members collaborating at the desk
    Data-Driven Marketing: The Secret is a Test-and-Learn Culture
    Read More
  • Team collaborating on a table looking at pie charts and data and statistics
    3 Marketing Measurements Before the Campaign? Yes, You Must Do This
    Read More
  • Online/Offline Marketing Measurement Isn’t As Impossible As It Once Was
    Read More
  • The Next Wave of AI Data Analytics Will Revolutionize Marketing
    Read More
  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share via Email
  • Copy Link
    Copied!
Tags: Data & Insights, Incrementality, Measure, Optimize, Plan, Research, xTLx

    Subscribe to Updates

    Media that matters.
    Marketing that works.
    © 2023 Kenshoo, Ltd. All Rights Reserved.
    Privacy Policy. Cookie Policy. Recruitment Privacy Policy.
    • Connected Data
      • Market Intelligence
      • Our Approach
      • By Need
      • By Solution
    • Connected Strategy
      • Dynamic Marketing Mix
      • Budget Forecasting
      • Strategic Consulting
    • Connected Media
      • Overview
      • Retail Media
      • Paid Search
      • Paid Social
      • App Marketing
      • Auditing
      • Expert Services
    • Measurement
      • Incrementality
      • Experiments
      • Cross-Channel Attribution
    • Resources
      • Blog
      • Glossary
      • Case Studies
      • Training & Enablement
      • Developer Hub
      • The Breakthrough
    Privacy Preference

    We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.

    Privacy Preference

    Save All

    Save

    Accept Only Essential Cookies

    Manage Cookie Preferences

    Cookie Details Privacy Policy Imprint

    Privacy Preference

    Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

    Save All Save Accept Only Essential Cookies

    Back

    Privacy Preference

    Essential cookies enable basic functions and are necessary for the proper function of the website.

    Show Cookie Information Hide Cookie Information

    Name
    ProviderOwner of this website, Imprint
    PurposeSaves the visitors preferences selected in the Cookie Box of Borlabs Cookie.
    Host(s).skai.io, skai.io
    Cookie Nameborlabs-cookie
    Cookie Expiry1 Year
    Name
    ProviderOwner of this website
    PurposeThis cookie stores selections made by the user in the Accessibe tool in order to maintain those settings on future visits. These cookies help us make our website compliant with our obligations under US law.
    Privacy Policy https://accessibe.com/privacy-policy
    Cookie NameacsbState, acsbReset
    Cookie Expiryn/a
    Name
    ProviderOwner of this website
    Host(s)skai.io
    Cookie Namewordpress_sec_,wordpress_test_cookie,wp-postpass_*, wordpresspass_*, wordpressuser_*
    Cookie ExpirySession / 1 Year

    We use these cookies to enhance functionality and allow for personalisation, such as live chats, videos and the use of social media.

    Show Cookie Information Hide Cookie Information

    Accept
    Name
    ProviderOwner of this website
    Host(s).chilipiper.com, skai.chilipiper.com
    Cookie Namefs_uid, CHILI_PIPER_CLUSTER, guest-session, _sp_ses*, _sp_id*
    Cookie ExpirySession / 2 Years
    Accept
    Name
    ProviderOwner of this website
    Host(s).comeet.co, www.comeet.co
    Cookie Namevisid_incap_, nlbi_#######, incap_ses_, referrer22_00a, incap_ses_1364_2167377
    Cookie ExpirySession / 1 Year
    Accept
    Name
    ProviderOwner of this website
    Host(s)skai.io
    Cookie NamemoduleFormPardotDownload
    Cookie Expiry30 days

    Statistics cookies collect information anonymously. This information helps us to understand how our visitors use our website.

    Show Cookie Information Hide Cookie Information

    Accept
    Name
    ProviderGoogle Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
    PurposeCookie by Google used for website analytics. Generates statistical data on how the visitor uses the website.
    Privacy Policy https://policies.google.com/privacy?hl=en
    Cookie Name_ga, _ga_*, _gat,_gat_*,_gid, GOOGLE_ABUSE_EXEMPTION, AEC, __Secure-ENID, SOCS, CONSENT
    Cookie ExpirySession / 2 Years
    Accept
    Name
    ProviderHotjar Ltd., Dragonara Business Centre, 5th Floor, Dragonara Road, Paceville St Julian's STJ 3141 Malta
    PurposeHotjar is an user behavior analytic tool by Hotjar Ltd.. We use Hotjar to understand how users interact with our website.
    Privacy Policy https://www.hotjar.com/legal/policies/privacy/
    Host(s)*.hotjar.com
    Cookie Name_hjClosedSurveyInvites, _hjDonePolls, _hjMinimizedPolls, _hjDoneTestersWidgets, _hjIncludedInSample, _hjShownFeedbackMessage, _hjid, _hjRecordingLastActivity, hjTLDTest, _hjUserAttributesHash, _hjCachedUserAttributes, _hjLocalStorageTest, _hjptid, _hjSessionUser_2229986, _hjIncludedInPageviewSample, _hjIncludedInSessionSample, _hjAbsoluteSessionInProgress, _hjFirstSeen
    Cookie ExpirySession / 1 Year
    Accept
    Name
    ProviderOwner of this website
    PurposeVWO is AB Testing software that we'll use to test different elements of the website - form placement, nav, different page layouts, etc.
    Privacy Policy https://vwo.com/compliance/cookies-notice/
    Host(s).skai.io, dev.visualwebsiteoptimizer.com
    Cookie Name_vis_opt_exp_{CAMPAIGNID}_goal_{GOALID}, _vis_opt_test_cookie, _vis_opt_exp_{CAMPAIGNID}_combi, _vis_opt_exp_{CAMPAIGNID}_exclude, _vis_opt_exp_{CAMPAIGNID}_split, _vis_opt_s, _vis_opt_out, _vwo_uuid, _vwo_uuid_{CAMPAIGNID}, _vwo_ds, _vwo_sn, _vwo_uuid_v2, _vis_opt_exp_{CAMPAIGNID}_combi_choose, _vwo_referrer, _vwo, _vwo_global_opt_out, _vwo_ssm, uuid
    Cookie ExpirySession / 10 Years

    Marketing cookies are used by third-party advertisers or publishers to display personalized ads. They do this by tracking visitors across websites.

    Show Cookie Information Hide Cookie Information

    Accept
    Name
    ProviderLinkedin
    Cookie Namelidc, li_gc, lang, AnalyticsSyncHistory, UserMatchHistory, li_sugr, bcookie, TDCPM, TDID, bscookie, ln_or
    Cookie ExpirySession / 1 Year
    Accept
    Name
    ProviderSkai
    Cookie Nameken_gclid
    Accept
    Name
    Provider6sense
    Cookie Name_gd_session, _an_uid, _gd_visitor, _gd_svisitor, 6suuid
    Cookie ExpirySession / 400 Days
    Accept
    Name
    ProviderPardot
    PurposeCookie name associated with services from marketing automation and lead generation platform Pardot. The visitor value is the visitor_id in your Pardot account. This cookie is set for visitors by the Pardot tracking code.
    Host(s).pardot.com, pi.pardot.com, skai.io
    Cookie Namepardot, visitor_id*, lpv*
    Cookie ExpirySession / 10 Years
    Accept
    Name
    ProviderGoogle Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
    PurposeCookie by Google used for conversion tracking of Google Ads.
    Privacy Policy https://policies.google.com/privacy?hl=en
    Cookie NameIDE, 1P_JAR, NID, _gcl_au, OTZ, test_cookie
    Cookie ExpirySession / 400 Days
    Accept
    Name
    ProviderOwner of this website
    Cookie NamePHPSESSID, sid, okto-autoposter-XXX, oktgid, oktsid, oktlogid-XXX, oktolead-XXX-Email, oktolead-XXX-Company, oktolead-XXX-FirstName, oktolead-XXX-LastName, oktolead-XXX-Phone, oktolead-XXX-Country, oktolead-XXX-Url, oktolead-XXX-State, oktolead-XXX-Comment, optout
    Cookie ExpirySession / 2 Years
    Accept
    Name
    ProviderMeta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
    PurposeCookie by Facebook used for website analytics, ad targeting, and ad measurement.
    Privacy Policy https://www.facebook.com/policies/cookies
    Cookie Name_fbp,act,c_user,datr,fr,tr,m_pixel_ration,pl,presence,sb,spin,wd,xs
    Cookie ExpirySession / 1 Year

    Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

    Show Cookie Information Hide Cookie Information

    Accept
    Name
    ProviderWistia
    Host(s).wistia.com
    Cookie Namecb_anonymous_id, _sp_ses.2b40, _li_dcdm_c, __hssrc, _gcl_au, _clsk, hubspotutk, _sp_id.2b40, __hssc, __hstc, _uetsid, _uetvid, _gid, _ga, _ga_GQR109DZ3Y, _lc2, fpi, _ex-pricing-cta, _fbp, cb_group_id, cb_user_id, _clck
    Cookie ExpirySession / 400 Days
    Accept
    Name
    ProviderMeta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
    PurposeUsed to unblock Instagram content.
    Privacy Policy https://www.instagram.com/legal/privacy/
    Host(s).instagram.com
    Cookie Namepigeon_state
    Cookie ExpirySession
    Accept
    Name
    ProviderOpenstreetmap Foundation, St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS, United Kingdom
    PurposeUsed to unblock OpenStreetMap content.
    Privacy Policy https://wiki.osmfoundation.org/wiki/Privacy_Policy
    Host(s).openstreetmap.org
    Cookie Name_osm_location, _osm_session, _osm_totp_token, _osm_welcome, _pk_id., _pk_ref., _pk_ses., qos_token
    Cookie Expiry1-10 Years
    Accept
    Name
    ProviderTwitter International Company, One Cumberland Place, Fenian Street, Dublin 2, D02 AX07, Ireland
    PurposeUsed to unblock Twitter content.
    Privacy Policy https://twitter.com/privacy
    Host(s).twimg.com, .twitter.com
    Cookie Name__widgetsettings, local_storage_support_test
    Cookie ExpiryUnlimited
    Accept
    Name
    ProviderVimeo Inc., 555 West 18th Street, New York, New York 10011, USA
    PurposeUsed to unblock Vimeo content.
    Privacy Policy https://vimeo.com/privacy
    Host(s)player.vimeo.com
    Cookie Namevuid
    Cookie Expiry2 Years
    Accept
    Name
    ProviderGoogle Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
    PurposeUsed to unblock YouTube content.
    Privacy Policy https://policies.google.com/privacy?hl=en&gl=en
    Host(s)google.com
    Cookie NameCONSENT
    Cookie Expiry6 Month

    Borlabs Cookie powered by Borlabs Cookie

    Privacy Policy Imprint