How NBC Used Product Intelligence to Drive Retention by 2x

Before Amplitude, NBC used a product analytics tool that often failed to deliver the insights it needed to make critical product decisions. As a result, the company leaned on its data science team for experiment results, cohort analysis, and user flows. With multiple divisions and organizations to support, their response time was often several days or even a week. Amplitude, which enables product teams to do everything from creating cohorts to analyzing retention, has cut those wait times to just a few hours.

“Now we can do this first level of data science ourselves, which is awesome,” said Jay Khavani, Director of Product Analytics, “We don’t have to go to the data science team for each and everything. It’s more like we’re in self-serve mode.”

Jay estimated that Amplitude has given back data analysts 25 percent of their day, QA engineers 10 percent and data engineers 5 percent to focus on more complex problems, personalized discovery algorithms, and optimizations; work that can be a competitive differentiator for NBC.

With its easy-to-access, real-time insights into user behavior, Amplitude has also allowed NBC to build what Eddie Lee, NBC Entertainment’s Senior Vice President for Product and Technology, called a “scalable framework for experimentation.” Amplitude allows the company to immediately understand test performance, and identify opportunities for additional tests using Cohorts, Conversion Drivers, and Engagement Matrix.

Eddie pointed to a test around a segmented homepage experience for Vizio TVs. When Amplitude’s Funnels and AB Test View showed the changes boosted viewership by 10%, the company rolled them out to all its viewers. NBC also found success in its experiments in tailoring its app homepage to user history. The result? Day 7 retention was up 2x. (Both experiments also leveraged Amplitude’s powerful Cohorts tool, which makes it easy to identify and target user segments.) Likewise, testing yielded a video preview feature that boosted video start conversions by 36%.

Of course, quick and accurate testing also means there’s less risk in trying new things, even when they don’t work out. When NBC found that a video-rating feature didn’t move any key metrics, they backed off wider implementation, saving time and money.

Amplitude has also made these sorts of findings more accessible—and actionable. “It’s become easier for product managers to share data and insights with other folks, whether that means engineers or a design team,” Josh said. He noted that this process is helping the company realize its goal of using data to drive its decisions. Even better, NBC has been able to create powerful narratives around this data using Amplitude’s Notebooks tool, which allows users to frame data in context and can be easily shared with other stakeholders. “The Notebooks feature has allowed us to very quickly synthesize findings and then present them to executives,” Josh said, which helps boost buy-in at the highest levels.

Beyond its immediate benefits, Amplitude’s greatest appeal was its potential, NBC said. It was clear that the platform was innovating fast enough to keep up with the company as its needs evolved. That conclusion was enough to move the company to switch from its previous product analytics tool.

“Our business is evolving and whatever analytics tools we use needs to evolve with the times and keep up,” said Josh. “We just felt that Amplitude was better positioned to do that.”