At Thoughtworks, we're noticing a trend towards using real-time and batch data to achieve business objectives. It is common for organisations to use multiple separate tools to meet the specific needs of different teams for batch and real-time. However, when crucial business use cases, such as creating customer golden records or running a campaign, need both data ingested via real-time and batch, then several challenges like accuracy, consistency, complexity of heterogeneous data, volume, performance and reconciliation can arise.
Here is one example of a data platform which could overcome these challenges as it enforces the same rules for both streaming and batch. This architecture ensures that real-time and batch data use the same data stores, shared artefacts, schema management and governance. Another advantage of this architecture is that it could be designed to be mostly configuration driven.
If you’re interested in discussing how we could help you with these challenges, or want to chat about how you’re tackling them, we’d love to hear from you!
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
