In the five years since Thoughtworks first introduced data mesh, we’ve seen how effective it can be at breaking down silos and barriers to innovation and collaboration, enabling seamless data sharing between diverse domains.
Data mesh has made it much easier to access and share data and data products within organizations. However, there’s increasing demand for organizations to share data with one another to realize mutual value, advance their industries, and drive progress toward critical societal goals. So basically, we are talking about a data mesh that spans across multiple organizations.
One example is Catena-X, a collaborative body that connects automotive manufacturers, suppliers and service providers and enables the secure exchange of data. It aims to harness the value of industry-wide data to help address challenges that a single OEM cannot solve alone, such as supply chain volatility and the move to a circular economy.
There’s also growing political momentum for inter-organizational data sharing. The European Commission’s data strategy calls for the creation of Common European Data Spaces that will eventually be connected to form a single market for data access and reuse across Europe. The Horizon Europe program, for example, is focused on the open sharing of research data, knowledge and tools to help researchers address global challenges such as climate change, loss of biodiversity, and the aging population.
Initiatives like these have the potential to deliver huge value for the organizations that participate in them — and for society as a whole. However, enabling secure, seamless data sharing between organizations brings many new challenges. Challenges that we believe data mesh could help to solve.
The challenges of inter-organizational data sharing
Organizations that have seen success with data mesh are accustomed to making decisions about the privacy-utility tradeoff for accessing and sharing sensitive data. The creation of inter-organizational data spaces makes these decisions even more critical.
Conventional data governance frameworks are generally focused on intra-organizational factors and aren’t designed to address the complexity and scale of governing inter-organizational data sharing. Robust frameworks are needed to ensure effective data governance and a shared understanding of individual organizations’ and users’ responsibilities. It’s also vital to have the ability to trace lineage throughout the data sharing environment.
Clearly, privacy is a major concern when sharing data outside the boundaries of the organization. In shared data spaces, safeguarding personally identifiable information (PII) while maintaining its utility will require a combination of privacy-preserving techniques, including differential privacy, data masking and attribute-based access control.
Organizations and collaborative bodies must also consider the licensing aspects of participation in shared data spaces, applying measures to ensure data can be used and processed by other parties, but cannot be copied.
For organizations to discover and unlock the value of shared data, it must be adequately documented and described by the data owner. Effective metadata management is also essential to ensure all data space participants can access accurate, high-quality data. Data quality must be specified and monitored across the environment to ensure all participants can realize value from shared data.
Some vendors are attempting to address these challenges by offering data ‘clean rooms’ that allow organizations to share privacy-preserved data at scale. However, a data mesh approach could allow organizations not just to share data, but to share data products that enable all participants to accelerate the realization of meaningful business value.
Extending the data mesh to data spaces
Applying the principles of data mesh to shared data spaces (or even building inter-organizational data meshes) can help organizations make data and data products available for widespread reuse — securely and efficiently.
By enabling zero-copy use by authorized entities over encrypted, authenticated interfaces, authorized participants can securely share privacy-preserved information while ensuring owners retain full control over their data.
The federated governance of data mesh demands that participant organizations set and uphold strong standards for data quality and usage. At an inter-organizational scale, it’s also vital to enable automated governance to ensure data security and integrity. Manual governance at such a scale would be prohibitively expensive and ineffective, with limited traceability of interventions.
In practice, organizations should take their established data product approach and extend it toward external use. By creating data products with multiple output ports, organizations can apply separate privacy-utility tradeoffs for internal and external ports.
For example, a data product could have four internal output ports that offer role-based and attribute-based access control, and an external port that adds differential privacy and data masking to further protect PII and other sensitive data.
Such an external port could be an Asset Administration Shell (AAS), which would help mitigate interoperability issues. Catena-X member organizations, for instance, use AAS to share digital twin data more efficiently, even though the input data may be in a proprietary format. In some cases, it may also be useful to register external output ports in data marts to enable specialist teams to access specific capabilities.
Unlock the data sharing opportunity
Data sharing can help organizations advance the state of the art in their industry, solve seemingly intractable problems, and generate significant business value. However, creating data sharing environments that specify, monitor and maintain the right levels of privacy and utility can be a complex undertaking. We have worked with multiple organizations to facilitate inter-organization data meshes and can share our experiences.
Organizations that have developed internal data meshes have an opportunity to extend them beyond the boundaries of the business and partner with other organizations to realize mutual value - or even straight up monetize this data. With their experience in applying data mesh principles internally, such organizations are ideally placed to accelerate the secure, compliant exchange of valuable data and data products — and propel the next wave of innovation.
