In the last post of the Product Instrumentation Success series, we explored how to take a mixed-pattern approach to deciding what to measure. The next tip is about domain awareness—specifically customer domain awareness. This is where many teams go wrong.
Story time! About a year ago, I ventured onsite to do a North Star workshop at a highly successful B2B SaaS company. After a great workshop—bonus, we actually figured out a good North Star—I sat down to review their data. My jaw dropped. They had thousands and thousands of events. Bad. The events had names like: ProductsController.identify_product and LeftNav.Item[4].ButtonText=”Help” and Click.MidPane.H2.Span.Blah (yup, they stuffed CSS into the event name).
Effectively instrumenting their product would have required under thirty events and would have almost certainly covered their North Star Metric. But here I was with 3,000 events and couldn’t even answer a basic question.
This approach is more common than you would think, and it comes from a good place: a desire to save time and work scalably. But it makes the classic mistake of ignoring the language, context, goals, and mental models of the customer (see domain driven design which attempts to address this issue).
The issue is not unique to developers. It can happen when we get too fixated on the current interface. For example, I recently met with a team that was nearing completion of a major redesign. The conversation went like this:
Me: “Great job with the redesign. I am super excited to see how it performs!”
Them: “Oh yeah. That. Um, so are we, but we’re having trouble.”
Me: “Oh no. What happened? Can’t you compare it to the old version?”
Them: “No. Things have changed too much.”
Me: “What do you mean changed? Have you added functionality?”
Them: “No, it was a straight redesign.”
Me: “OH! Wait. Did you pack too much interface knowledge into the event names?”
Them: “Bingo. Lesson learned!”
Here’s what had happened. Instead of thinking like a customer and naming things based on their goals (e.g. Careers – View Latest), they had named them based on their knowledge of the interface (e.g. Click Careers Tab Left Nav). Customers don’t care whether something is on the left nav. And they don’t really care about clicking tabs (especially since 50% of people are tapping). They care about viewing the latest job openings.
Had the team instrumented their product with the customer domain in mind, they would have been able to track improvements by using the exact same event names with the interface specifics (e.g. Version X vs. Version Y) moved into the event properties, which are useful for capturing contextual information.
The reality is that user goals tend to stay fairly consistent. The details of how the goal is achieved in your product may change, but the goal—in the customer’s words—stays stable. Which is why #3, starting with a workflow and/or customer narrative, is a feasible approach IF you make sure to describe it from the perspective of the customer.
Using your product interface to jog your memory is a great idea, but always remember to step back and describe events—the things that happen that we care about—in a way that is agnostic to the interface. Similarly, looking at the code to figure out what might happen is helpful. The important step is to then ask… “So how does the customer experience and describe this?”
If one thing sticks about this tip, let it be this: to future-proof events, consider the customer domain. Fewer customer-centric events beat out more interface-specific or code-specific events any day.
Stick to plain-language, customer domain-focused nouns, verbs, adjectives, and adverbs. Events should be understandable purely from the event name, event properties, and user properties.
The full series:
Part 1: Measurement vs. Metrics
Part 2: Use a Mixed Pattern Approach to Instrumenting your Product
Part 3: Keeping the Customer Domain Front and Center
Part 4: Learning How to “See” Data
Part 5: The Long Tail of Insights & T-Shaped Instrumentation
Part 6: Asking Better Questions
