Event sourcing is the process of ensuring that all changes to application state are stored as a sequence of events. But how do you manage this in Kafka? Let’s find out in this article to get you started with the concept of real-time data streaming, something our team over at Axual is an expert in and can help you with.
Getting started with Event Sourcing and Kafka?
Event sourcing is the process of ensuring that all changes to application state are stored as a sequence of events. These events can be queried and be used to log to reconstruct past states thereby allowing automatic adjustment of the state to align with retroactive changes. The idea behind event sourcing is that it makes sure every change to the state of an application is recorded in an event object. Beyond that, it also makes sure that the event objects are stored sequentially. These events can be aggregated into different application states.
What is Kafka? Simply put, Kafka is a distributed streaming platform used for publishing and subscribing to streams of records. It is designed to help applications to process records in real-time. It can also be used as fault-tolerant storage.
Using Kafka for Event Sourcing
There are three basic ways to store events in Kafka. They include the following:
- One, single partitioned Kafka topic per aggregate
This provides a guarantee that events will be stored in order and the reading state of an aggregate is very simple as it’s more or less like reading a topic from offset 0. However, this strategy is not feasible; since each user in the system needs to create a topic, there will ultimately be too many topics. Besides, any aggregation, for example, indexing all the users in a search engine, will be very hard as such a task requires the consumption of huge topics.
- One partitioned Kafka topic per aggregate type
Here, the required number of partitions depends on a specific number of aggregate types. This means that aggregate ID is important as events are produced. This strategy also guarantees that events will be stored in order. However, unlike the first strategy discussed, this one is highly feasible as it only reads partition where aggregate instance or type is stored. With this strategy, it is possible to scale and partition each aggregate type stream separately.
- One, single highly partitioned Kafka topic for all aggregate types.
Here, all events are stored in a single topic. This has a great advantage as it is possible to have a global view of all the events in a single topic. It guarantees that events will be stored in order and it is highly feasible just like the second strategy.
Kafka was designed as a data streaming engine. However, with capabilities for partitioning, streaming APIs, state stores, replicated topics, etc., it can be used to implement an event sourcing system. The three strategies to do this have been explained in this article while it’s noted that two of these strategies are greatly feasible. Here at Axual, we have expertise in using Kafka for event sourcing. Therefore, you do not need to look any further, we have best-in-class solutions for you. Get started now!
Download our whitepaper
Want to know how we have built a platform based on Apache Kafka, including the learnings? Fill in the form below and we send you our whitepaper.