The Road to a Test-and-Learn Organization Part 1: Learn from the Netflix Testing Process

Most marketers know that being more data-driven is the (not so) secret ingredient to success, but often struggle with how to do this consistently. In a recent article, Data-Driven Marketing: The Secret is a Test-and-Learn Culture, I discussed how hard data-driven marketing is and that it can only really be done well when the entire organization is committed—from top to bottom—on elevating the use of data as the key to best-in-class decision-making.

For marketing leaders, there might not be a more important mission than setting up their teams to be successful data-driven marketers. But, operationalizing a dedicated data & insights organization is not so simple. You can’t just flip a switch. A strong foundation with a solid test-and-learn approach is required. 

To offer a blueprint on how to proceed, I outlined in my last post the Four Pillars of a Test-and-Learn Organization:

• Test-and-Learn Organization Pillar #1: An organization’s testing process must include policies to support test prioritization, execution, and management.
• Test-and-Learn Organization Pillar #2: An organization’s learning process is focused on actionability and decision-making
• Test-and-Learn Organization Pillar #3: A test-and-learn culture that aligns the team and enforces accountability
• Test-and-Learn Organization Pillar #4: A clear vision for the types of strategic & operational objectives to be tackled

Even armed with these four pillars, turning your team into a true test-and-learn organization won’t be an easy road to take—but it’s definitely the right path to choose.

To better understand these four pillars, we will highlight them in separate posts.

Today, we’ll focus on Pillar 1, The Testing Process and use Netflix’s excellent internal testing methodology as an example of how this can be done well. 

Netflix’s commitment to testing excellence required it to reorganize teams

Netflix is a leader in demonstrating how a test-and-learn mindset informs better business decisions through an accurate understanding of customer insights and preferences. In less than a decade, Netflix has moved from a fledgling service mailing DVDs to one of the most admired tech companies in the world. 

To broaden the impact of test-and-learn processes, the ability to propose ideas for measuring has to be available to everyone across the organization. But most marketers are not trained in data analysis nor experienced in creating measurable hypotheses on a daily basis.

Most organizations try to solve this by creating an analytics center of excellence (COE) that every team can tap into. Even though a dedicated analytics COE is a good start, the downside is that most of the organization won’t learn the principles of how to extract insight from data, but rather rely on others to come in and do it for them.

For Netflix, being more data-driven was such an important initiative that they tore down siloed business intelligence groups and instead embedded data scientists, data engineers, and data analysts inside each business unit. 

Now every business team has dedicated specialists who focus on the data and the opportunities to improve data-driven decisions. Now every business team can actively propose ideas for experimentation and can share the best applications for newly discovered insights.

(from “Enabling a Culture of Analytics”)

Netflix had so many tests that they created a system to prioritize

Testing as a primary approach to decision-making is such a major focus that sometimes there are simply too many tests to run. Experiments can be disruptive to the daily flow of business, require too many resources, or overlap with other tests and taint the final results. Another consideration is which tests to do first as the insights learned from some can be used to optimize future tests for even better insights. So, all of these tests—however helpful and valuable they maybe—can also be chaotic to manage.

How did Netflix solve this issue of testing prioritization? With machine-learning.

As explained in the Netflix technology blog: 

“With a collection of tests that, by nature, are time-consuming to run and sometimes require manual intervention, we need to prioritize and schedule test executions in a way that will expedite detection of test failures… In our quest to be objective, scientific, and in line with the Netflix philosophy of using data to drive solutions for intriguing problems, we proceeded by leveraging machine learning.”

While not everyone can build their own machine-learning system to evaluate which tests can be run in parallel and which must be run before others, the principal factor behind test prioritization can be understood by everyone: focus on those tests which will reveal insights about the needs and pain points of customers. 

This focus is the foundation of agile market positioning and lean business strategy, and Netflix’s solution helps to keep teams from spending too much time on testing internal improvements at the expense of understanding their core audience.

Netflix’s testing processes drive actions and accountability

To increase the benefits of experimentation, Netflix critically manages both test execution and the deployment of insights. “The Netflix Marketing team embraces experimentation to identify the best marketing tactics for spending paid media dollars…” says the technical team. “Our teams use experimentation to guide their instincts on the best performing campaigns.” 

This requires oversight to guarantee experiments successfully run to completion or are quickly cancelled if they run into quality issues. Experts look at the incoming signals to ensure that the volume of data is high enough to yield the required accuracy levels. Deviations from predicted responses are examined for local market interference from regionally competitive product or service offers. Seasonality impacts and influences are monitored to correct expectations for daily baselines.

Additionally, when the experiment has completed, the results are evaluated for business impact. For failed tests, what is the next hypothesis to be tested? For successful tests, how should marketing processes be optimized as a results of the new insights?

Netflix marketing analysts take every new result from a marketing test and examine whether that learning can improve each and every stage of their campaign management systems.

Five sets of campaign management systems are the most common targets for benefiting from the results of these marketing experiments:

1. Media planning systems which are responsible for automating the workflows used for buying paid media and unlocking efficiencies in those flows (media planner)
2. Creative management systems that help with creative development & localization and assembly of ads from the creative assets.
3. Campaign management systems which are responsible for marketing campaign creation and execution 
4. Advertising insights systems which are responsible for collecting analytics and insights into how our campaigns are performing on ad platforms like Google 
5. Ad budget optimization systems that enable changing budget on live campaigns in order to optimize our marketing spend

Conclusion: excellence over a range of testing processes is required to maximize the impact of data-driven insights

The strategic approach used by Netflix relies on

• Accurate and insightful data-driven decisions,
• Supported by a broadly enabled test-and-learn culture that
• Democratizes hypothesis making
• Prioritizes test execution, and
• Manages execution and insights across the business teams.

Netflix is an exemplar of how a test-and-learn culture guides business. Instead of taking chances with blind strategic bets, Netflix has instead focused on decision effectiveness, and become a market leader by maximizing opportunities from newly discovered consumer insights.

Netflix is so committed to testing, that it even has a Netflix Research blog. “Data-driven decision making and the Netflix culture of experimentation extend from our data scientists through all levels of the company to Reed Hastings himself. Hypotheses and results are subject to rigorous analysis plus robust and open debate from a wide variety of viewpoints. Our executives make time to understand experimental methods and test results, and their close interaction with data scientists helps ensure that our decisions are sound from both statistical and business perspectives.”

For marketing and business leaders reading this post, the road to data-driven maturity won’t be easy, but it offers the surest approach to success.

Book a Meeting of Skai