Your Dashboards are Lying to You
When I join a new team, one of the first things I do as a PM is to check out the product’s dashboard. How many users do we have? What do we know about their usage patterns? What are we trying to optimize? What are we worried about?
At first I felt a bit silly asking, like I ought to know already. I sort of assumed everyone else did. After all, the team had a dashboard with a bunch of product metrics. Shouldn’t I be able to figure it out for myself? But I’ve found it surprisingly rare for teams to have clear answers. And part of why is because the dashboards lying.
A clue that your dashboards might be lying is if you find yourself having conversations like these:
“What’s the definition of Active User in this chart?”
“Hmm, I’ll have to check with Steve. I think he instrumented this”“Why did all of the metrics drop by ~10% in September?”
“Oh, that must have been when we changed how we did logging for compliance. No actual change in user behavior”“Why do we have so many users in Bangladesh?”
“Uhhh, I don’t think we do. Those numbers are probably a click farm skewing our stats”“This slide looks great — we grew 40% last year! 40% of what?”
“I don’t remember. I suppose linking to a screenshot of the right graph from the dashboard would have been helpful”
Getting answers like this isn’t a sign that your team is hopelessly dysfunctional or disorganized. The reality is just that building and maintaining good dashboards is harder than most people realize. It’s hard, but it’s also essential to understanding your product and your users, which is key both to innovating and to catching regressions.
Teams need to define metrics and how to compute them. Logging systems and retention policies change over time. Teams need to define what constitutes bogus data and filter it out so that meaningful changes are visible. All of this context (definitions, rationales, dates) commonly lives ephemerally as institutional knowledge. But for anyone new to interpret or make inferences based on the dashboard, this context needs to be documented and easy to reference from the dashboard.
How is your dashboard?
Go ahead, grade yourself.
F: No dashboard
D: Dashboard exists, but no one checks it because no one really understands it
C: Dashboard exists, but only Steve can explain what the graphs mean
B: Most people on the team can access the dashboard and report values
A: At a glance, anyone can tell whether the top metrics are good or bad. The dashboard is organized and maintained, with alerts on the top metrics, links to metric definitions, and annotations on key events (rollouts, logging changes, etc.)
Getting a B is not bad — lots of teams have grade B dashboards — but it also might be the most dangerous state. People report values and make inferences and decisions without the context, based on a dashboard that was probably lying.
As a PM, I’m typically not the one who implemented the metrics, so I’m at the biggest risk of reporting nonsense from a grade B dashboard. If nothing else, I’ve learned to check with an engineer before making any grand proclamations about metrics.
But rollout monitoring and decision making would be so much easier with a grade A dashboard.
What do you want your metrics to be doing anyway?
A basic but underrated bit of prerequisite knowledge for interpreting graphs is knowing what they are supposed to look like. We’re used to graphs where good is up and to the right. That’s great for metrics like Daily Active Users and Revenue. But if Latency is going up and to the right, you have a problem. Big numbers are good for Users, but bad for Errors. For each type of metric, it helps to know what you’re looking for, what needs to be investigated, and when to panic.
A few tips for better dashboards
- Document all the context a viewer needs to know! Link out to where the logging is implemented in the code so that the documentation doesn’t get obsolete, but make sure to include a rationale as well
- Put graphs of top line metrics at the top
- Group together graphs that are supposed be going in the same direction. If one starts pointing the wrong way, it will be much easier to notice than if all the graphs were already pointing every which way
- Color code consistently. If paying users are purple and free users are orange in one graph, use those colors consistently across the other graphs
- For top line metrics, add bounds around the expected values and alerting for when a value crosses a bound. Then you’re not relying on someone opening the dashboard and noticing the issue. Just make sure the bounds are loose enough that you don’t over-trigger and start ignoring the alerts
Finding new insights
As a product manager, ideally you’re not just understanding what the team has already captured in their dashboard, but you’re also uncovering new insights. Come up with hypotheses about what different groups of users might be doing (like “paying users are active more days per week” or “younger users are more likely to be paying”). See if any of the existing graphs in your dashboard can be sliced to answer your question. Just remember that the dashboard might be lying!
If you can’t find what you need in the dashboard, often the next step is to investigate the logs. In a future post, I plan go into how to analyze log data to uncover trends and make the patterns pop.
