Apache Kafka is an open-source streaming platform frequently used by companies looking to develop innovative digital services that rely on large, real-time data sets.
Kafka became popular because of its performance and ability to scale. It’s often used by companies that have adopted event-driven architectures and that are building real-time applications — ones that are time-sensitive and deal with a continuous stream of data.
A popular data streaming platform that provides a scalable and fault-tolerant infrastructure to build systems adopting an event-based architecture and real-time applications.
If you need a platform that can rapidly and reliably deal with large volumes of event data — that is data associated with business events — Kafka is a solid choice.
The way Kafka works is markedly different from similar so-called message brokers — systems built to orchestrate the processing of that preceded it. So when your teams start using Kafka, there is a learning curve to use and operate it.
Kafka is used by many companies that deal with web-scale data volumes, including LinkedIn, Uber, Twitter and Spotify and as the backbone for organizations’ "data-in-motion" needs.
What is it?
Kafka is an open-source steaming platform frequently used by companies looking to develop innovative digital services that rely on large, real-time data sets.
Kafka became popular because of its ability to scale. It took a novel approach to message brokering and is useful for storing and processing data both historical and real-time work.
As enterprises look to develop innovative digital services, they increasingly want access to a wide variety of real-time data. Kafka provides a platform to integrate that data: it can reliably cope with vast volumes of data and is fast and resilient.
What’s in for you?
Kafka can play an important role in the development of new digital services and products that take advantage of web-scale volumes of real-time data.
Its ability to scale, its speed and its reliability are all important qualities in a message streaming platform.
It’s also flexible, allowing you to build different kinds of systems on top of its infrastructure — from messaging to event-driven architectures or real-time streaming apps. The ecosystem around Kafka is very mature and it works with any cloud provider or even on-prem if needed.
What are the trade offs?
Adopting Kafka may require a shift in mindset for staff used to working with other messaging brokers. And not all of the products and services you’re working on will need vast quantities of real-time data — and there may be more suitable tools for those types of projects.
Kafka requires non-trivial infrastructure to operate and the cost to build or operate this might not be justified if there’s a simpler solution that can be used. Some ‘Kafka-as-a-service’ providers are starting to emerge because of the complexity in learning to operate Kafka.
How is it being used?
Kafka was originally developed at LinkedIn to enable it to scale as its user base — and the complexity of dealing with all of those interconnected users — increased.
Kafka rapidly gained popularity with web-scale businesses, including Spotify, Twitter and Uber. It has subsequently been embraced by many Fortune 500 companies.
It is a proven technology for those organizations that need to be able to rapidly deal with large volumes — and often real-time — data.
Would you like to suggest a topic to be decoded?
Just leave your email address and we'll be in touch the moment it's ready.