A successful Product Manager needs to be part salesman, part creative, part team-builder, part motivator, and full-time decision-maker.
However, we haven’t mentioned that this Product Manager also needs to be a very strong analyst for most companies.
The great Product Manager has a keen understanding of what separates trustworthy data from misleading data.
It doesn’t matter if you’ve ever written a SQL query, it doesn’t matter if you’ve never had a passion for discussing the differences between Elastic, BigQuery, and Postgres databases, you still need to be dangerous with data.
I’m pretty sick of the term “big data,” but today’s Product Manager, regardless of company size/vertical/position, needs to acknowledge the competitive landscape of data analysis.
Watch Out For “Vanity” Metrics
Vanity metrics include statistics like
- registered users,
- and raw pageviews
They are very easily manipulated and typically only move in one direction: up.
To a Product Manager, these metrics have very little value.
They’re great for PR and that’s about it.
Everybody knows the Product Manager who only presents these kinds of metrics. He/She may even *wow* a few of the more naive members of your company, but not you.
You know that the metrics you really care about are much deeper, much more significant and that vanity metrics just aren’t worth your time.
The worst part about reporting vanity metrics is that you’re often forced to continue to report them.
If you fail to keep reporting downloads to investors or the press, for example, they’ll assume the worst.
If you keep reporting downloads, you also strongly risk focusing your team on a metric that isn’t important for the product in your development process. Lose, Lose.
Key Performance Indicators aka “Real Metrics”
The Key Performance Indicators (KPIs) you choose to depend on your type of business and the stage of the growth cycle that you are in.
However, a few safe metrics to measure include
- active users (Daily/Weekly/Monthly),
- net promoter score (NPS),
- repeat engagement,
- cost per acquisition,
- cohort churn,
- and publish/key action rates.
Confused about these metrics or want to learn more? Check out the Kiss Metrics Blog for great, in-depth articles about each metric and how to best track them.
Additionally, I’d highly recommend reading Measure What Matters by John Doerr to learn about the transformational process of setting Objectives and Key Results (OKRs).
What Should I Measure?
If you’re a first time Product Manager, data can be very overwhelming.
Between Google Analytics, Heap, SQL queries, Optimizely, and qualitative data sources, you’re spending more time trying to access data rather than getting insights from it. Simplify your life and start with A-C-E goals.
Choose 3 Key Metrics (A-C-E)
1. An activity metric
- What is the key feature of your product?
- What do you want every new user to do?
- What user action makes you throw your hands up in the air and scream “hell yeah!”?
That is your activity metric.
At Pagemodo, the key activity metric was the use of the scheduled posts feature for Social Media.
At Upside Travel, it is purchasing the 2nd trip with them.
Choose an activity that highly correlates with your most successful users and focus on how you can increase the number and share of users who complete that behavior.
2. A customer satisfaction metric (ex. NPS)
Still don’t have enough customers to reach any sort of significance with an NPS survey?
That’s completely fine.
Reach out to every 5th or 10th new customer directly yourself after they have used your product.
Ask them to rate your product on a 1 – 10 scale at the end of your email/call/Intercom message.
Congratulations, you just started collecting satisfaction data.
3. An engagement metric
For most digital products, engagement is a key sign of success.
If your product is incredibly difficult for customers to live without, you’ve succeeded.
As a new Product Manager, you can either pick a count metrics like monthly active users (unique users in a calendar month) or a proportion metric like percent of customers with a second login (i.e. non-abandoners).
By tracking engagement rates closely, you’ll know if your product is consistently getting better at bringing users back for more.
We began this article by talking about the importance of a strong connection between a Product Manager and all available data surrounding his/her product or his/her new feature.
However, our discussion so far largely left out a significant portion of the analysis that needs to be completed by the Product Manager: a qualitative analysis.
Earlier, we briefly discussed the importance of qualitative analysis through the testing cycle, but we didn’t dive into the specifics on how a Product Manager can successfully adopt this essential counterpart to quantitative analysis.
1. Survey your users at key events
Did a user just complete a key “activity”?
Did a user just cancel their service from your account management page?
Make sure you survey them before they can fully complete the cancellation process. Ask them:
- Why would you like to cancel?
- Was there anything we could have done differently?
- Would you like to speak to someone? Sometimes this is all it takes to make a save!
Has a user been inactive for 30 days?
Make sure you have an automatic email and survey in place to re-engage these users or at least try to learn about why they stopped using your product.
2. Identify Promoters disguised as Detractors
When you get negative responses to your NPS survey (scores from 1-6), dive deep, and try to understand what led to their poor perception of your product.
Many times, you’ll find that your detractors are often promoters in disguise.
They liked features 1 and 2 but ran into a bug on feature 3, which destroyed their experience.
Engage these users and you’ll quickly build a population of people that aren’t afraid to give honest and open feedback on new features, even if it stings a little.
Remember that the Alpha population we worked on building in this article? These (hopefully temporary) detractors would be a great addition!
3. How does a Product Manager share or present data?
We, talk more about sharing data with the organization and externally here.
For now, let’s discuss sharing key performance data with your team only.
In this article, we talked about building the right team. Along with that building process, you should have built a detailed understanding of each of your team members.
- Are they excited to stand up against a challenge or do they need a pep-talk each time they try something new?
- If they see a disturbing change in trends, do they back-peddle or confront the challenge head-on in an effort to reverse it?
From this place of understanding, a Product Manager should know what type and what depth of data they can share with the team.
Are you a new Product Manager who doesn’t know your team well enough yet?
No problem, keep it simple, follow the ACE metrics for now.
As your team gets comfortable with these key performance indicators, you can expand to more niche analytics.
Consistency and focus are incredibly important for teams of all levels of experience and skill.
4. When is “Gut Feel” ok?
Quantitative and Qualitative Analytics are the best friends of a Product Manager building a new product, developing a new feature, or improving an existing product.
However, there are many situations when data either isn’t available, takes too long to access, or simply isn’t valuable.
In these situations, what should you as a Product Manager do?
An effective PM, in an absence-of-data environment, should use their best judgment and incorporate the following factors to make a decision:
- What does the customer experience look like with the different decision options?
- What are the risks of this decision?
- Who does this decision benefit/who does it impact negatively?
- What are the costs of the different options?
An essential part of making this “data-less” decision is building a foundation for future analytical confirmation or refutation of the “gut feel” decision.
By setting up this tracking, the effective PM will ensure that they either validate their decision or pivot quickly from it.