We recently hosted an insightful discussion on data mesh with Zhamak Dehghani and Dave Colls. The audience asked some great questions during the session around a few key topics. One of our presenters, Dave Colls, has captured these in this summary.
Data mesh & enabling organisational change
Questions came up on the organisational change component of data mesh and Conway’s Law, including scaling specialised skills like data science.
The change in architecture and organisation are two interdependent and mutually reinforcing changes - one enables the other. Consider the Inverse Conway Maneuver for data mesh. The “bottleneck on the human side” can be (partially) addressed with platform thinking - how do people with rare and in-demand skills use technology to scale their impact and enable others to do some of the work they do, within guardrails and with support when necessary? For data science skills, for instance, consider a science platform along the lines of What I Talk About When I Talk About Platforms. See also Data mesh: it's not just about tech, it's about ownership and communication.
Sharing data across domains
People asked about domain ownership of data products, how disparate data is combined in analytics, how to manage dependencies on other teams, and conformed data.
Data products are absolutely intended to be combined, and used by multiple consumers for various different purposes with low friction - this is the essence of the mesh, and of managing data as a product. Note that domain-oriented operational software products or services are combined in a similar manner to deliver customer experiences.
Data mesh proposes source-oriented and consumer-oriented and shared domains data products. Consuming analysts should be able to combine multiple data products in scenarios from exploratory data science research to reporting - with different requirements of source data and derived data products in these various scenarios. The data product affordances help analytics consumers find, assess and use data.
“Conformed” data becomes one way to look at the data for one set of consumers, rather than the only way, and conformance is achieved when a source domain is authoritative, or federated computational governance addresses conformity for specific consumption needs.
Measurement & observability
People asked about how the mesh itself requires and supports observability.
Observability is a key -ility for data mesh - some of the data product affordances and federated computational governance features require observability. In a complementary fashion, data generated by providing observability can be considered a source data product and computed fitness functions can be considered as derived data products. A data mesh platform team could indeed use their own products (“dogfooding”) to understand the performance of the mesh. The good news is that modern tooling provides patterns to support this. See an example in Observability for Everyone
Implementation on lake or modern warehouse infrastructure
People asked about creating separate domains within a central environment.
The primary aim of data mesh is to decentralise the ownership of data, while recognising the benefits of shared self-service infrastructure for foundational storage and computation needs. Data lakes and modern warehouses have been evolving in the right direction in this regard. But you should consider a proposed lake or warehouse against the self-service platform considerations - does it satisfy these? - as well as the other data mesh principles, as we’ve often found centralised data environments lead to more centralised ownership of data than we’d like.
From small organisations to spanning organisations
People asked about data mesh in small organisations and across organisations.
There is a scale below which reliance on a central team isn’t a major bottleneck, but as every business relies more and more on enabling its teams with technology, data and analytics, thinking about the decentralisation of these capabilities remains a priority. Data mesh is fairly new to federations of organisations, but we’ve talked to many who see the potential to facilitate scaling data inside and outside organisations with a similar pattern.
If you’d like to watch the full webinar, click here to view the recording.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
