By Keith Schulze and Kunal Tiwary
Today, organizations are increasingly recognizing the potential value of data – yet many fail to realize a return on investment from their data assets.
When it comes to developing data assets, many organizations take a ‘build it and they will come’ approach. While this philosophy may have paid off for Kevin Costner in Field of Dreams, organizations might not be so lucky. Because there is one inherent shortcoming in thisof the approach: it doesn’t consider the needs of data users.
What if we flipped the mindset, and consider some valuable user-centric lessons from our product teams? What if we managed data as a product – not just an asset? We’re seeing this shift in perception gain traction, allowing organizations to unlock more value from data projects.
Rethinking data
The data as a product mindset is one of the four principles of data mesh, a style of data management which decentralizes project architecture models. Data as a product treats the data users as customers, developing data products to bring them value and help them achieve their end goals. For example, if your customer’s end goal is to reduce churn rate by 10%, you will need to start with that goal and work backwards – developingand develop a churn forecasting data product that will meet this need. Thinking of data as a product means putting those user needs at the heart of their design. It’s designed to be shared – not controlled.
Zhamak Dehghani, author of Data Mesh, Delivering Data Value at Scale, and founder of the data mesh concept, describes it this way; “Data as a product is very different from data as an asset. What do you do with an asset? You collect and hoard it. With a product it’s the other way around. You share it and make the experience of that data more delightful.”
Product thinking requires a deep knowledge and understanding of your customer. Your teams can then build for real world problems – and continuously develop products that offer more value.
Build it right and they will come
Many data products fail because they are a solution in search of a problem – for example, ingesting a new dataset into the data platform because ‘someone’ will find it useful. Adding more data does not necessarily solve a customer’s problems – or provide them with value.
That’s why it’s so critical to start by knowing who your customer is and what is most valuable to them. What problems are they trying to solve? What’s at stake for them if they can’t use or access the data easily? Those customers might be internal or external – the key is to think beyond simply offering data sources, and expecting users to adapt or compromise the way they work to use it.
Unfortunately, there’s no silver bullet here. It takes time to understand your customers and their goals, and involves real-world testing and constant refinement. And once you’ve solved that for one group of customers, how do you scale and expand this? Can you make those products reusable, satisfying the needs of a broader range of customers?
At Thoughtworks, we have adapted the Double-Diamond design process model to make sure that we build the right thing and build it right. This starts with identifying what a customer needs. We use a structured discovery and inception process to uncover these requirements for any new data product. We then apply a set of well-understood practices and tools that are known to deliver high-quality software and data.
The Thoughtworks Double-Diamond design process model
Encouraging ownership and responsibility
If, in the more traditional mindset, projects end once a dataset or report is delivered, product thinking requires teams to retain ownership over a data product for its entire lifecycle. That means data product owners are responsible for evolving and adapting the data product to ensure it continues to meet the needs of the customer even as their requirements change.
Much like software products, data products also benefit from a responsible and accountable team who continuously improve performance and release new features in a safe environment. It also reduces the feedback loops needed to evolve or or enhance these products. It encourages direct communication between the producer and the consumer of data products – cutting out lengthy and convoluted central planning processes.
The owners of a data product are also accountable for maintaining agreed levels of service. This is important because without clear accountability, there might be complex processes and competing priorities to contend with when services go down.
For example, retail organizations use a number of metrics to facilitate demand planning (e.g. forecast accuracy, order fill rate). Different teams depend on these metrics to forecast and provision stock to meet the demand. Any delays or errors in reporting can have severe impacts to downstream business processes, leading to unhappy customers and a loss of revenue or a surplus of stock with a cost to business.
As a business evolves, there may be other demand planning metrics that would allow for more accurate forecasts; any delay in implementing these also means a sacrifice in potential profit. Businesses need to continuously evolve their demand planning process to use the most accurate metrics – and ensure that the metrics are reliable and high quality. Any error should be fixed promptly to minimize the impact on downstream consumers.
Having a team with clear accountability to develop, evolve and maintain these metrics as a product with high levels of service ensures that:
- You’re always providing downstream consumers with the most accurate metrics to facilitate demand planning, and
- You minimize the impacts of outages on other teams in the demand planning process.
What it takes to make the change
A mindset shift such as this often requires cultural and behavioral change as well. If your organization wants to reap the benefits of user-centric data products, you will need to move to a more product-centric, customer-focused culture – and build cross-functional teams to support this approach.
Moving away from teams aligned to archetypes or skill sets, to small product-oriented teams with tightly focused goals is one way to get there. These teams may require a blend of different capabilities – such as data engineers, data scientists, QAs and designers – to develop a product that meets the needs of customers.
Creating a culture where learning from failure is embraced and celebrated is also critical to the success of developing effective data products. Finding what doesn’t work, or where friction points lie, allows teams to adjust their thinking and approach for future projects – and continually improve products and customer experience along the way.
Putting data as a product into practice
An effective data product should be:
Discoverable
If you want a customer to use a data product, they need to be able to find it. Discoverability can take many forms, from a primitive list of datasets on an internal wiki system to a full-fledged data catalog. Irrespective of the implementation, catalogs should house important meta information about the data products such as their owners, source of origin, lineage, and sample datasets.Addressable
Data products should also have a single unique address where they can be found. This makes it easy for your customer to find them – and reduces the time your data teams spend on helping people locate them.Self-describing and interoperable
Data products should provide users and automated systems with metadata in a way that allows users to self-service. For example, a data product should expose metadata describing the data sources used to build the data product, and the schema and information about outputs of the data product.Trustworthy and secure
For a customer to use a data product confidently, the product should commit to an agreed level of trustworthiness (or quality). This means defining a set of Service Level Objectives (SLO) and measurable Service Level Indicators (SLI) upfront – and implementing automated mechanisms to test and report SLI metrics regularly.Secure and governed by a global access control
With the rapid rise in data breaches, it has never been more important to secure your data products and build in security. While registered data sets should be automatically discoverable to all customers, they should not be accessible by default. Users will need to request access to each dataset they need, with data owners granting or denying access individually, using federated access controls.
Reaping organization-wide benefits
Adopting a data-as-a-product mindset is an organization-wide exercise – it demands a shift in not only perspectives but also in culture and practices. It is certainly worth the effort. The principles of product thinking allow you to develop multiple data products that can be used within the organization, and ultimately help you form an effective and streamlined network of data products. And when it becomes embedded in your enterprise, it helps raise the bar for tech teams – supporting them to always think about creating value and working towards outcomes for every user.