How Netflix’ quasi-experiments let it go beyond the limitations of A/B testing

Even if you seek to make every aspect of your marketing data-driven, you likely still run into situations that defy traditional testing. Perhaps you’re using non-addressable media, so you aren’t able to isolate and randomize the audiences to which you’re messaging. Or maybe it just isn’t practical to roll out a dramatic redesign of your website to only half your audience.

Such are the problems for which Netflix employs quasi-experiments — tests that aren’t as easy to conduct, but which nevertheless produce actionable business intelligence.

At our MarTech Conference in San Jose in April, Mike Pow, senior product manager of the experimentation platform at Netflix, will share how the company conducts quasi-experiments and how it’s rolling them out more widely. We spoke to Pow ahead of the show to get a sneak peek into this fascinating topic.

Typically, marketers looking for guidance on a way forward will employ an A/B test — rolling out a change to one group while keeping the second static. But to be successful, the A and B groups in the test need to be both similar and randomized, and there are times when that just isn’t possible or practical.

“There are a few types of problems in which you want to use quasi-experiments,” said Pow. “One is when you can’t apply treatment at the user level.” An example would be if you’re buying radio or other media where you can’t precisely control the distribution of your messaging.

“Another category would be when you don’t want to apply the treatment at the user level,” he continued. “For example, if you’re trying to measure word of mouth effect, then necessarily, you’re trying to see how groups of people will receive treatment and then interact and talk about your product. And so, therefore, you would necessarily need to deliver your media at a regional level…. and then you can see how buying patterns change as a result of your treatment.”

Pow explains that quasi-experiments are typically randomized — as much as is possible, anyway — by choosing two geographical areas where the overall audiences are similar, then using one as the control group and the other as the treatment group.

“This is just another technique in which companies can use to learn, which is core to what Netflix does,” said Pow. Netflix is known for having a very experimental company culture, using data to drive recommendations as well as the creation of its original content.

“Netflix shows us how, these days, just about anything can be predicted with incredible accuracy – even those things thought previously impossible,” writes author Bernard Marr, the author of Big Data and Big Data in Practice.

Quasi-experiments at Netflix began a couple of years ago. At that time, these projects were managed via ad-hoc standalone spreadsheets, Pow said, “which made the media buying communication, and also the subsequent analysis, super chaotic and really burdensome to repeat.”

That’s why Netflix wrote software to allow the learnings gained in designing, running and analyzing the results of quasi-experiments to be saved and built upon.

“As our volume of quasi-experiments ramped up, we decided to build out a platform with quasi-specific capabilities,” said Pow, explaining that the tool allows the team to properly support them through the entire testing lifecycle of planning, design, execution and analysis.

To learn more about Netflix’ quasi-experiments, the results they’ve achieved and how they’re operationalized, register for the MarTech Conference taking place April 3-5 in San Jose. Mike’s talk will be at 11:50 a.m. on Thursday. 

About The Author

Pamela Parker

Pamela Parker is Research Director at Third Door Media's Content Studio, where she produces Martech Intelligence Reports and other in-depth content for digital marketers in conjunction with Search Engine Land, Marketing Land, MarTech Today and Digital Marketing Depot. Prior to taking on this role at TDM, she served as Content Manager, Senior Editor and Executive Features Editor. Parker is a well-respected authority on digital marketing, having reported and written on the subject since its beginning. She's a former managing editor of ClickZ and has also worked on the business side helping independent publishers monetize their sites at Federated Media Publishing. Parker earned a master's degree in journalism from Columbia University.