A new solution using Data Mesh is helping enterprises improve and accelerate insights.
For decades, enterprises have invested heavily in their data architectures. Many have poured substantial resources into creating architectures designed to help them rapidly transform ever-increasing amounts of data into actionable insights.
Often these investments often don’t deliver the promised value. Just 13% of organizations excel at delivering on their data strategy, according to research by MIT Technology Review and Databricks earlier this year. Only 26.8% of firms report successfully embedding a strong data culture across their organization, reported a study by NewVantage Partners last year.
For many enterprises, the centralized data architectures they have chosen, such as data warehouses and data lakes, are at the root of ongoing problems. Long data onboarding times, analytical bottlenecks, and overstretched and centralized teams, together with data quality issues and discovery challenges, can all be unwanted side-effects of these architectures.
Crucially, domain teams can find themselves having difficulties using the data products that are produced — with this key goal going missing in the rush to onboard and process data. Meanwhile, developing capability in domain teams, which is essential for creating value, can get overlooked when centralized architectures are used.
Increasingly, enterprises are looking for more flexible solutions. This is where Data Mesh comes in.
What is Data Mesh?
Data mesh is a decentralized approach to data architecture, originally defined by Thoughtworker Zhamak Dehghani. In a Data Mesh, data doesn’t sit together in a centralized pool. Instead it is broken down into distinct ‘data products’ that are owned and managed by the domain teams closest to them.
The four foundational principles of Data Mesh, as defined by Zhamak, are:
Domain-oriented decentralized data architecture. In a Data Mesh, data is owned and controlled by the teams closest to it, removing the number of steps and handoffs between data producers and consumers
Data is managed as products. Bespoke products make data highly accessible to the teams that need it. This empowers teams across domains to self-serve and access whatever they need quickly and easily
Self-service data infrastructure. Data Meshes are built to enable self-service, and give teams the automated means to operationalize and extract value from data without the manual and hand-crafted assistance of centralized experts
Federated governance. Governance is automated at the platform layer, ensuring standards are upheld without impacting flexibility or limiting how individual domains can use data
What does that all mean for your organization?
As an architectural approach, Data Mesh is neatly aligned with the data goals that enterprises want to achieve today. It brings data producers and consumers closer together and empowers teams to self-serve and access highly relevant data products. So it’s well-placed to help companies create and embed agile, data-driven cultures of innovation and experimentation that extend across their organization.
Here are some of the transformational benefits that data mesh offers enterprises:
Making better-informed decisions, faster
In centralized data architectures, there are a lot of expertly, hand-crafted steps between the creation of data and the actions that result from it. Data is ingested or onboarded in bulk — steps that are often not visible to teams that need the data; even once data is available, teams may face long analytical lead times to translate it into insight.
With Data Mesh, a lot of those steps are removed — as in automated or rendered unnecessary. Domain teams onboard their own data, and manage their own data products. They know what data they have, and they’re free to operationalize it however and whenever they choose. This makes a strong contrast with the world of centralized data architectures, where there can be a tendency to produce standardized views of data, under that assumption that one size will fit all. With data mesh, domain teams can be empowered to pull customized views of data as they wish.
For enterprises, Data Mesh therefore drives a huge acceleration in decision-making. By enabling domain teams to operationalize and act on data faster, organizations can gain competitive advantages and extract greater value from the large volumes of data they gather and hold.
At one major financial services institution, Data Mesh architecture had a substantial impact on average times to value almost immediately. With access to domain-oriented data products and the freedom to operationalize data at speed, executives were able to ask more questions, get more reliable answers, and act on valuable insights faster than ever before. Domain teams were also able to build analytical data directly into their customers’ digital experiences, providing real differentiation in the market.
Creating truly data-driven cultures of innovation
One of the biggest advantages of a decentralized architecture like Data Mesh is that it puts the end users of data in control of how it’s managed and used.
In a Data Mesh, domain teams are in the driver’s seat. As the custodians and controllers of their own data products, they’re free to experiment with that data however they like. They can ask more questions, simulate more scenarios, and explore more data-driven moonshot ideas — the kinds of things that lead to lasting, meaningful innovations.
Every domain team is incentivized to ensure that their data products are as coherent and well-maintained as possible, as they directly impact that team’s analytical capabilities and outcomes. So, across an organization, that adds up to a culture where everyone across every domain is invested in data quality, experimentation, and pushing the boundaries of data innovation.
At Saxo Bank, Data Mesh played a significant role in the organization’s journey to becoming a data-driven open banking institution, working in partnership with Thoughtworks. The implementation of Data Mesh principles alleviated challenges around data visibility, quality, and access, and empowered teams not only to move their open banking objectives ahead but also to continuously improve upon them.
Supporting AI and machine learning initiatives
AI and machine learning have quickly evolved from highly-sophisticated specialist technologies into essential capabilities applied across all levels of the modern enterprise. To deliver value, both need two things; high-quality, relevant data sets, and innovative minds that can identify powerful use cases for them.
When domain teams are in control of their own data products across a Data Mesh, those teams will naturally start to build and maintain the kinds of data sets needed to fuel game-changing AI and ML use cases.
Plus, because the domain teams are the custodians of that data, there are far fewer barriers preventing them from experimenting with AI and bringing powerful new use cases to life. The Data Mesh becomes an enabler of AI and ML innovation, with teams even having the freedom to create data products specifically for AI and ML use — making the capabilities accessible to more teams and across more domains than ever before.
Transformation starts with a winning business case
Together, those benefits form the foundation of a robust business case for Data Mesh. They’re widely applicable and relevant, but they’re far from the only advantages that Data Mesh can deliver. The approach also lends itself well to helping organizations:
Improve data quality and governance, and even automate many elements of governance and compliance using purpose-built data products
Respond faster to emerging regulations thanks to the improved visibility, quality, and governance models enabled across the Data Mesh
Create or participate in data product marketplaces and securely share data products — or even collaborate to co-create products — across organizations
Identify more opportunities across your enterprise data with more eyes on the data that matters — and more teams incentivized to explore every potential use case for it
However, it’s worth keeping in mind that any business case you create for Data Mesh needs to be highly tailored to the challenges your organization is facing. Chances are, some of the benefits we’ve highlighted will resonate more clearly and feel more exciting than others. And it’s those areas where you need to focus.