• MarTech Today
  • Sections
    • Advertising
    • Marketing
    • Content
    • Social
    • Commerce
    • Sales
    • Analytics
    • Management
    • All
    • Home
  • Follow Us
    • Follow
  • MarTech Today
  • Ads
  • Marketing
  • Content
  • Social
  • Sales
  • Analytics
  • Mgmt
  • More
  • Events
    • Follow
  • SUBSCRIBE

MarTech Today

MarTech Today
  • Advertising
  • Marketing
  • Content
  • Social
  • Commerce
  • Sales
  • Analytics
  • Management
  • All
  • Events
  • Newsletters
  • Home
Martech: Analytics & Data

Subscribe to MarTech Today to receive news and insights of where marketing, technology, and management converge.

Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.


How Netflix’ quasi-experiments let it go beyond the limitations of A/B testing

What to do when a marketing problem doesn't lend itself to traditional means of testing? Discover Netflix' approach.

Pamela Parker on March 18, 2019 at 1:22 pm
  • More

Even if you seek to make every aspect of your marketing data-driven, you likely still run into situations that defy traditional testing. Perhaps you’re using non-addressable media, so you aren’t able to isolate and randomize the audiences to which you’re messaging. Or maybe it just isn’t practical to roll out a dramatic redesign of your website to only half your audience.

Such are the problems for which Netflix employs quasi-experiments — tests that aren’t as easy to conduct, but which nevertheless produce actionable business intelligence.

At our MarTech Conference in San Jose in April, Mike Pow, senior product manager of the experimentation platform at Netflix, will share how the company conducts quasi-experiments and how it’s rolling them out more widely. We spoke to Pow ahead of the show to get a sneak peek into this fascinating topic.

Typically, marketers looking for guidance on a way forward will employ an A/B test — rolling out a change to one group while keeping the second static. But to be successful, the A and B groups in the test need to be both similar and randomized, and there are times when that just isn’t possible or practical.

“There are a few types of problems in which you want to use quasi-experiments,” said Pow. “One is when you can’t apply treatment at the user level.” An example would be if you’re buying radio or other media where you can’t precisely control the distribution of your messaging.

“Another category would be when you don’t want to apply the treatment at the user level,” he continued. “For example, if you’re trying to measure word of mouth effect, then necessarily, you’re trying to see how groups of people will receive treatment and then interact and talk about your product. And so, therefore, you would necessarily need to deliver your media at a regional level…. and then you can see how buying patterns change as a result of your treatment.”

Pow explains that quasi-experiments are typically randomized — as much as is possible, anyway — by choosing two geographical areas where the overall audiences are similar, then using one as the control group and the other as the treatment group.

“This is just another technique in which companies can use to learn, which is core to what Netflix does,” said Pow. Netflix is known for having a very experimental company culture, using data to drive recommendations as well as the creation of its original content.

“Netflix shows us how, these days, just about anything can be predicted with incredible accuracy – even those things thought previously impossible,” writes author Bernard Marr, the author of Big Data and Big Data in Practice.

Quasi-experiments at Netflix began a couple of years ago. At that time, these projects were managed via ad-hoc standalone spreadsheets, Pow said, “which made the media buying communication, and also the subsequent analysis, super chaotic and really burdensome to repeat.”

That’s why Netflix wrote software to allow the learnings gained in designing, running and analyzing the results of quasi-experiments to be saved and built upon.

“As our volume of quasi-experiments ramped up, we decided to build out a platform with quasi-specific capabilities,” said Pow, explaining that the tool allows the team to properly support them through the entire testing lifecycle of planning, design, execution and analysis.

To learn more about Netflix’ quasi-experiments, the results they’ve achieved and how they’re operationalized, register for the MarTech Conference taking place April 3-5 in San Jose. Mike’s talk will be at 11:50 a.m. on Thursday. 

More on the cutting edge of martech

  • Register now for the MarTech Conference - April 3-5, San Jose, Calif.
  • There are 6 challenges every data-driven marketer faces. Here's how to manage them.
  • How National Public Radio reinvented itself for an on-demand audience
  • MarTech Overtime: How B2B marketers can reach buying committees at their target accounts


About The Author

Pamela Parker
Pamela Parker is Senior Editor and Projects Manager at Third Door Media's Content Studio, where she produces Martech Intelligence Reports and other in-depth content for digital marketers in conjunction with Search Engine Land, Marketing Land, MarTech Today and Digital Marketing Depot. Prior to taking on this role at TDM, she served as Content Manager and Executive Features Editor. Parker is a well-respected authority on digital marketing, having reported and written on the subject since its beginning. She's a former managing editor of ClickZ and has also worked on the business side helping independent publishers monetize their sites at Federated Media Publishing. Parker earned a masters degree in journalism from Columbia University.

Related Topics

Channel: Martech: Analytics & DataMarTech ConferenceMarTech Speaker SpotlightPredictive Marketing

Subscribe to MarTech Today to receive news and insights of where marketing, technology, and management converge.

Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.


We're listening.

Have something to say about this article? Share it with us on Facebook and Twitter.

ATTEND OUR CONFERENCES

April 15-17, 2020: San Jose

October 6-8, 2020: Boston

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • How Good Partnerships Should Reduce Your Workflow
  • Digital Insights 2019: How marketers confront the obstacles of digital customer engagement
  • 14 Industry Experts on the Future of Attribution
  • The Essential Guide to Protecting Your Ad Spend from Invalid Traffic
  • How to Turn Leads into Repeat Customers
See More Whitepapers

Webinars

  • The Fantastic 5: Emerging strategies to drive conversions & growth
  • CCPA Goes Live in Weeks: Is your marketing measurement ready?
  • 4 Ways to Get Started With Agile Marketing
See More Webinars

Research Reports

  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Enterprise SEO Platforms
  • Call Analytics Platforms
See More Research
Sidebar Inbox Deliverability 2019
Sign up for the NEW MarTech Today.
Martech Today
Download the Martech Today app on iTunes
Download the Martech Today App on Google Play

Channels

  • Advertising
  • Marketing
  • Content
  • Social
  • Commerce
  • Sales
  • Analytics
  • Management
  • Home

Our Events

  • MarTech West
  • MarTech East

Resources

  • White Papers
  • Research
  • Webinars
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Advertise
  • Marketing Services
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • iOS App
  • Google Play

© 2019 Third Door Media, Inc. All rights reserved.