Three big reasons to embrace Data Mesh in financial services

By Isa Goksu, director of technology,  financial services, UK

 

Not too long ago, a new generation of centralized data architectures — including data warehouses, data lakes, and cloud data lakes — looked like it was going to solve all of the data challenges organizations had battled with for decades. 

 

While those architectures have proven extremely useful for things like accelerating analytics, today we know that, in many cases, they’re far from perfect. From processing bottlenecks and data quality issues to the inability to turn data into value at speed, fundamental issues persist in centralized architectures.

 

Now, that’s giving rise to new, decentralized architecture approaches, like Data Mesh. Data Mesh seeks to maintain the visibility and governance advantages delivered by centralized models, while cutting processing times and helping teams get far greater value from the data held within the organization.

 

The Data Mesh approach is built on four key principles:

 

1. Decentralization of data ownership and architecture

    

2. Domain-oriented data served as products

    

3. Self-service data infrastructure as a platform to enable autonomous, domain-oriented data teams

    

4. Federated governance to enable greater interoperability

 

For financial services organizations, those principles make Data Mesh an appealing prospect. The idea of an architecture approach that can support stronger data governance while also accelerating how teams turn data into value has naturally piqued the interest of many technology leaders in the space.

 

But gaining those benefits takes a lot of work and commitment across an organization. Data Mesh isn’t an instant fix for the common data challenges such as data quality, system of record and data lineage issues experienced in financial services, but it does enable a fundamental shift in how data is managed and operationalized, to/which can help solve some of the industry’s most critical data-related issues.

 

Shifting to Data Mesh isn’t for everyone, but here are three big reasons why adopting the approach might be worth the effort for your organization.

 

Reason #1: Solving persistent data quality challenges

 

For a long time, financial services organizations have tried to resolve underlying data quality problems by implementing solutions like Master Data Management. While these solutions can help resolve some of the symptoms they don’t do a very good job of addressing the problem itself — so most institutions still face the same fundamental data quality challenges they’ve faced for decades.

 

That isn’t because the solutions lack capability. It’s because, in many cases, data quality problems aren’t caused by data sources or structures — they’re caused by processes and cultures. Whether it’s rigid processes, bloated bureaucracy, or siloed operations, these are problems that tech capabilities alone cannot hope to solve.

 

Instead, they demand a shift in thinking — and that’s where Data Mesh comes in. 

 

Data Mesh is a movement towards a microservices mentality that encourages domain-driven thinking. In a Data Mesh architecture, domain teams are close to the data products they depend on and they have direct control over the quality of those products. Teams across the business all become data product owners and are incentivized to capture, input, and manage data more responsibly, embedding a culture of data quality at every level.

 

That has a massive impact on an organization’s ability to innovate and experiment with data-rich use cases that demand an exceptionally high level of data quality. Things like Artificial Intelligence and Machine Learning, for example, demand clean, high data quality to deliver meaningful value. The domain-oriented structure and quality-centric culture enabled by Data Mesh provides the ideal foundation for that.

 

Reason #2: Enhancing data strategy with product thinking

 

Large financial services organizations gather and store huge quantities of ‘dark data’ — data that’s gathered for a hypothetical purpose, but never operationalized. In most cases, that data sits within the enterprise, but most teams in the organization don’t even know it exists, so nobody puts it to use.

 

The existence of dark data was one of the biggest reasons why many CIOs and CTOs created the kinds of centralized data warehouses and data lakes that are common across the financial services industry today. In theory, in a structure like that, all data is exposed and accessible to all. But, in practice, key elements of data governance and stewardship are still missing. The data might be part of a pool that anyone can see and explore, but if it’s marred by issues like unfamiliar formats or cryptic naming and labeling conventions, it isn’t truly in plain sight.

 

Data Mesh can play a valuable role in helping to solve that problem, by improving the visibility of data and enabling stronger data stewardship between domains. It supports a structure and culture where everyone understands what each domain wants to achieve using data, and products can be built to meet that specific purpose — operationalizing data wherever it is.

 

That ‘product thinking’ approach also helps build a stronger collective knowledge of how data is being used across the organization. Teams can look at the products created by other domains, collaborate with the product owners, and see how they could iterate on those capabilities, or operationalize similar datasets in new ways to achieve something completely different. 

 

It’s a culture of collaboration and autonomy. Products are visible to all, and all data can be operationalized with ease through the creation of domain-specific products, but domain teams are also empowered to drive their own outcomes.

 

Plus, because data is owned and managed by domain teams, they’re incentivized to manage it responsibly. They’re directly responsible for ensuring that their own data products are served with clean, reliable data which creates stronger data stewardship and governance that benefits everyone.

 

Reason #3: Improving security, fighting fraud, and streamlining compliance

 

Improving the visibility, management, and governance of your data environment doesn’t just help your teams operationalize data — it can also bring significant security and compliance benefits.

 

In the European Union, GDPR has exposed just how much work financial services organizations need to do to build up a complete view of their data - to share how it’s being used and who’s using it with regulators.

 

Using a Data Mesh approach, organizations can build bespoke data products specifically for compliance, enabling teams to respond to specific regulatory requirements with just a few clicks, instead of spending hundreds of hours compiling unreliable reports. That also means that compliance products have dedicated owners within the organization, so there’s a clear point of contact and responsibility when new demands emerge.

 

This approach enables organizations to work much more closely with regulators and collaborate with them as customers. Organizations can have proactive conversations with regulators to understand what they need, and quickly tailor a data product to meet that requirement. Then, when a regulator needs visibility of that information, the organization can give them exactly what they want immediately.

 

As regulatory requirements evolve, organizations can stay ahead of the game by creating new data products as soon as they’re needed. In evolving areas like cryptocurrency trading, for example, we’re set to see an explosion in regulation that organizations will need to be ready for. Data Mesh helps ensure that whatever questions regulators want to ask of your data, you can provide them with the right answers, fast.

 

The same principles can also be applied to security and anomaly detection. Using Data Mesh, systems like fraud detection don’t need to connect to numerous other systems and extract the same data every day. Instead, organizations can build domain-oriented data products that anomaly detection experts can use to create better models and outputs. 

 

Experts across your organization can build stronger models for everything from intrusion detection and prevention to attack surface detection. From visualizing attack trees to spotting emerging fraud trends, Data Mesh supports tailored visibility of just about anything an organization needs to monitor.

 

The benefits are clear. But Data Mesh isn’t a quick fix.

 

Data Mesh isn’t just a new approach to data architecture; it’s a completely new way of thinking about, governing, managing and operationalizing data. It’s not going to solve all data-related challenges overnight. It’s a change in approach that demands discipline and long-term commitment before it can be applied at scale to address those challenges.

 

It represents a significant increase in autonomy and responsibility for teams across the business. And for it to be effective, people need to rise to that new responsibility. 

 

The shift to Data Mesh also demands a significant change in thinking among technology decision-makers. For years now, organizations have taken steps to eradicate any duplication of effort across their data architecture. Data Mesh, as a decentralized approach, can start to reintroduce some duplication of effort. Teams shouldn’t be put off by that, as most resource-intensive work remains centralized within the model, but it can often become a point of friction that turns leaders away from the model before its value is fully realized.

 

Data Mesh isn’t for everyone. It’s a challenger model, built for organizations that want to change the inefficiencies and issues that have arisen in centralized data structures. Like any other challenger approach, it demands that the teams involved can think critically about the status quo, move away from today’s measures of success, and visualize a better future where everyone has access to (and responsibility for) the data products they need to succeed.

 

In financial services, the teams that can adopt those behaviors and facilitate shifts of that magnitude will be able to solve some of the industry’s most pressing data challenges. But it’s important to recognize the commitment required to get there and the scope of change the move to Data Mesh demands.

 

If you’d like to learn more about how you can apply the Data Mesh approach in financial services, and discover how Thoughtworks can help you navigate the challenges of applying the model, please get in touch with us at contact-uk@thoughtworks.com.