The best Product teams are obsessed with building experiences that delight and deliver value to their customers. Businesses that are smart about using quantitative and qualitative means to understand their customers’ needs build happy, loyal user bases and outperform their competition. In today’s digital age, it’s this kind of impact-driven product strategy that is the foundation of systematic growth.
Because of this, Product Managers are always trying to develop a deeper understanding of their users’ product journey, identify new opportunities to offer value, and continuously improve product engagement. Today, we’re spotlighting how Amplitude’s Impact Analysis helps teams do just this.
Untangling behavior is hard
We don’t need to tell you that understanding how and why customers behave a certain way is hard, even with a behavioral analytics solution in place. Behavior is complex and causality is notoriously difficult to prove.
Every week, our customers in the B2B, B2C, and media industries came to us with questions like:
- “How does our product’s new, faster search capability affect how often users share content with their colleagues?”
- “How does surfacing a new FAQ page affect how often users contact customer support?”
- “How does viewing the season preview of a new show affect how often customers watch video content?”
These PMs and Analysts have hypotheses about about user interactions within their product, especially those related to critical conversion and monetization behaviors. But they struggle to validate and measure the extent of these relationships in order to take action on that information.
That’s why we built Impact Analysis.
Impact Analysis is a unique Amplitude chart that helps teams understand how users engaging with a feature for the first time impacts how they behave in the product over time.
Our hope is that Impact Analysis helps people better understand the relationships between different user experiences by providing them with the means to: 1) identify opportunities to boost engagement and 2) evaluate and broadcast the impact of new or changed features on critical product behaviors.
How to create a new Impact Analysis chart in Amplitude.
Identifying opportunities for impact
Good PMs usually have some intuition about how behaviors are related in their product. They might want to test out their hypotheses immediately by jumping into a series of experiments—but this isn’t always easy or fast. Experiments still require planning, resources, and buy-in. Product teams can build conviction for their experiments more easily if they have data to back their hypotheses.
Impact Analysis gives PMs the ability to directionally validate hypotheses they have about how two user behaviors are related, before launching experiments to rigorously test their ideas.
Do it in Amplitude
Let’s say you are a product manager at AmpliTunes, a music and video streaming app. Since 80% of AmpliTunes’s revenue comes from in-content ads, your team’s current North Star is “number of songs or videos played.” This key engagement behavior is both a leading indicator of revenue and the value that users get from the product.
After conducting some user interviews, you notice that some people “favorite” content in order to revisit it at a later time. This makes you wonder if there might be a relationship between users favoriting content and playing content. Before setting up an experiment to test this, you use Impact Analysis to validate your hypothesis.
The chart below shows the average number of times users play content per week, before and after they favorite content for the first time.
Impact Analysis chart showing how users performing the Favorite Song or Video action for the first time in the last 12 weeks impacts the average number of times they perform the Play Song or Video action.
“Week 0” indicates the week users did the Favorite Song or Video action for the first time. Before Week 0, users play content an average of only three times a week. But after Week 0—that is, after users first favorite content—this jumps to an average of about eleven times a week.
This analysis backs your initial hypothesis with data—there could, in fact, be relationship between “favoriting” and playing content. This report could help you rally your team around your hypothesis and motivate them to experiment with new engagement strategies involving the “favoriting” feature.
(For full details on how to set up this report in Amplitude, see our documentation.)
Broadcasting product impact
Companies like Airbnb, Uber, and Netflix don’t have a magic silver bullet for growth—they build systematically, looping continuously through four building blocks of growth: generating hypotheses, experimentation, acting on their learnings, and broadcasting impact.
Impact Analysis helps teams validate hypotheses for improving user engagement and build a case for experimentation. Just as importantly, it helps close the learning loop by giving teams a powerful way to visualize the impact that the features they’ve rolled out has on other engagement behavior.
“Impact Analysis has been a huge boon for our team, helping us identify how our users behave before and after our engagement campaigns go live. With it, we can easily see the direct results our efforts have on our users and which bring us the largest return on investment. Recently, it has helped us make key decisions about our high-level strategy and what in-app events we will continue to emphasize moving forward. The chart has become one of our new staple tools and an instrumental part of our analysis framework.” Riley Pannkuk, Product Manager II, DoubleDown Interactive
Do it in Amplitude
As an AmpliTunes PM, last quarter, you rolled out a feature that allows users to follow automatically generated playlists or those created by friends. You can broadcast the impact of this new feature on user engagement by using Impact Analysis.
The chart below shows the impact that the feature rollout had on two important behaviors.
Impact Analysis chart showing how users performing the Follow Playlist action for the first time in the last 12 weeks impacts the average number of times they perform the Play Song or Video action (blue) and the Search Song or Video action (pink).
The blue curve shows the average number of times users play a song or video per week before and after they follow a playlist for the first time. The pink curve shows the average number of times users search for a song or video before and after they follow a playlist for the first time.
The average number of times that users listened to content jumped from 7-8 times per week to around 12 times per week after they followed a playlist for the first time. Similarly, the average number of times that users searched for content jumped from around 6 times a week, to more than twice that number after they followed a playlist for the first time.
This analysis shows that the new Playlist feature impacts not only how often users play content (AmpliTunes’s North Star) but also how often they search for content.
Next steps
For teams focused on iterative product development, Impact Analysis is a valuable tool to use at both the start and end of the learning loop. Teams can validate hypotheses to better understand the effects between user behaviors; they can also see the impact of product updates on engagement behavior.
While Impact Analysis does not replace experimentation for determining causality, it’s a useful tool to get a quick pulse on whether or not you are building something that your customers value.
Want to see it in action? Here’s a quick demo that shows you how to build an Impact Analysis report in Amplitude. (And for more details, take a look at our documentation.)
