Data-Driven Product Management

I have been on a bit of a mission lately to counter the industry tidal wave of data-driven everything. It started with one of my more popular blog posts last year titled The Real Reason On-Premises Software is Lousy and continues with my recent article on Product Intuition where I claim that data-driven product development has jumped the shark.

Just today, on the popular Product Management group on Facebook, a poster asked the question “What’s the best piece of advice you’ve ever received as a PM?”. The third response was “Always support your decisions with data” garnering lots of Likes.

I believe that any approach taken to an extreme will eventually result in weak practices and inferior decision-making. That said, the reality is that everyone is talking about data-driven decision making because it is the right thing to do — most of the time.

What Is Data-Driven Product Management?

Data-Driven Product Management is the application of data-driven decision making to the field of product management. This conforms nicely with the rise in data that is available from digital products, massively scalable big data technologies, and the tools and techniques of modern data science.

Further, many product management best practices today, rely heavily on collecting data, analyzing it, and rapidly learning to decide what to do next. One significant area is Product Analytics, which is specifically about collecting data to understand how your users interact with your products in order to improve them and generate greater value for all stakeholders.

Another cloud business trend associated with this is a go-to-market strategy called Product-Led Growth (PLG) promoted by Open View Partners. In its essence, the idea is that by building the best possible product, you can drive more customer acquisition, retention, and referrals. PLG is often coupled with Freemium pricing models and demands constant, iterative, product improvements based on analyzing usage data.

The Challenge

Cloud-based software providers, regardless of what commercial and architectural model they leverage, have a clear advantage when it comes to product analytics. There are many techniques and vendor products that are designed to integrate with these cloud-based products, allowing vendors to collect and aggregate data for continuous learning of actual usage.

For on-premises software, especially within enterprise environments, software is typically installed on servers behind firewalls. Unlike cloud software, software installed on-premises is typically under the control of the customer. It is only at their discretion that the software gets updated. Combined, this means that it is difficult to centrally collect and analyze data AND to update the software rapidly based on learnings from that analysis.

While at times it seems that the whole world has moved to cloud-based software deployments, this is simply not the case. In reality, a significant amount of enterprise software is still delivered for on-premises installation. According to a 2019 report by Flexera, 28% of software vendors primarily monetize with on-premises software. 63% of vendors continue to have moderate to extensive on-premises deployments.

So, the challenge becomes how to derive Data-Driven benefits in the tricky and significant world of enterprise on-premises software?

Opportunities for On-Premises Providers

Product analytics, despite challenges, is possible and used by many on-premises software products today. While the vast majority of product analytics vendors ignore this more complex domain, there are some that support it such as Revulytics. You can also roll your own usage collection mechanism. Then, with customer opt-in, you can periodically send anonymized analytics data back to HQ for aggregated analysis.

What can this do for you and your clients?

Identify features that are no longer being used
Learn if new features are being used as expected
Learn usage patterns to improve the overall user experience
Understand the volume of user activity and data flow
Provide usage data back to the customers themselves
Provide better support services

These benefits can help you improve your product offering generally and engage with specific customers much more effectively. Cloud providers do this by default and it provides a competitive advantage. On-Premises providers need to do this to remain competitive.

Keys to implement:

Start small in your next release to learn how collecting data process will work
Consider both automatic periodic usage uploads and manually supported options
Consider sensitive data carefully, it is rarely needed to learn about product usage
Determine if the opt-out or opt-in approach is more viable in your market
Review with your legal counsel to ensure it is allowable under current agreements and any specific risk mitigations
Build out a central, secure, infrastructure to store and analyze the data. Be smart and use commodity tools for this work initially.
Communicate this plan to your customers, along with the benefits to them.

If you start small, even products deployed under multiple versions, over time, can be aggregated for analytics. Applying this capability as a maintenance release to older deployed versions will give you tremendous additional insights on lifecycle management as well.

Beyond Product Analytics

Product usage data is not the only data that product managers can make use of. There are a number of other opportunities to make more data-driven decisions a part of your product management process. Here are some common examples:

Market Research

When building the context for assessing market problems and opportunities there are a number of ways to collect data suitable for better decision-making.

Surveys – Collecting statistically significant and relevant research data through a well-designed survey is a time-tested standard. With the latest best practices, tools for designing, and analyzing this becomes much more effective. Further, through distribution tools like Facebook, LinkedIn, and AlphaHQ you can get very targeted in who you send these surveys to making your results even more meaningful and actionable.
Competitive Intelligence – Anecdotes from win-loss reports, Google Alerts, and ad-hoc competitor research is no longer sufficient. No there are a plethora of great tools continuously pulling together data on your competition from PR, product releases, customer feedback, pricing data, patent filings, HR changes, and more.
Marketing Event Insights – When running marketing events based on specific problems, the relative demand this generates can be evaluated compared to that of other marketing events.

Designing Solutions

This is where the magic happens and this is where the best advances have been occurring lately product management. Broadly speaking, these are practices born out of the Lean Startup movement. In short, Lean practices leverage the scientific method to confront the multitude of risks inherent in new software development – early and with minimal investment.

The approach is to develop a hypothesis based on a combination of existing data and intuition. Then, build an experiment to test that hypothesis, collecting resulting data to inform decisions on what to do next. These experiments can take many forms including: MVPs, surveys, prototypes, support data analysis, landing pages, concierge services, and many more.

Leveraging these techniques is possible regardless of whether your product is delivered on-premises or in the cloud. Each experiment offers the opportunity to capture data in a low-cost way before fully building out the wrong software.

When you build the wrong software there is an opportunity cost of what else could have been built that would have been more valuable. Moreover, especially with on-premises software, it creates a challenging technical debt. The worst such debt, in my view, is delivering a feature that only 1 customer is known to use and you maintain indefinitely.

Product Operations

The last example of enabling a more data-driven product management culture, is embodied in the form of a new role – Product Operations Manager (ProductOps). Organizations that strive to become more data-driven need data, tools, and practices in place to be successful. ProductOps is the emerging role that helps organizations improve and scale in their use of data-driven practices.

For more on ProductOps check out this article by ProductPlan.

The Bottom Line

Data-Driven Product Management is a critical success factor for Outstanding Product Organizations. A wide range of data-driven practices exists that should be considered to most efficiently find valuable market problems, build the best solutions, and bring them to market successfully.