Do we need Event Sourcing?

-



Before we start, I would definitely recommend getting familiar with the definition and benefits of Event Sourcing. Sara did an amazing job in explaining what Event Sourcing is in more detail and comparisons. Let's briefly touch upon some Event Sourcing aspects that are going to be important for this article. 

In order to validate a business rule and/or make any decision in our software we need to create a model representation of our domain. For the purposes of this article, I'm going to call this model the Decision Model. There are different ways of storing this model durably. 
The first one is called state-stored, which means that we are storing only the current state of the Decision Model. 
Another approach is to store the Decision Model as a series of events that describe facts that happened in our system - we say that this model is event-sourced. Now, when we need to load an event-sourced model from the storage, we must read all events related to it. The most suitable storage for events is the Event Store (another concept You should get familiar with). 

Some frown on the fact that we need to load a lot of events in order to reconstruct the Decision Model. However, they see the benefits of Event Sourcing. This gave me an idea to explore whether we are able to achieve all the benefits of Event Sourcing while using a state-stored Decision Model.

In my opinion, the best way to approach this challenge is to zoom into the benefits of Event Sourcing and see whether we can obtain the same, but without Event Sourcing. Buckle up!

Benefit: Transactionality

Event Sourcing solves the transactionality challenge by using a single resource - the Event Store. In this scenario, the Decision Model is stored in the Event Store as a series of events, and it is also loaded as a series of events. 

The same challenge could also be solved by having a single resource that can store the current state of the Decision Model and historical events in an ACID way. Here, any storage that guarantees transactionality between (1) storing the current state of the Decision Model and (2) storing events will serve the purpose (see Diagram 1). 

Unfortunately, not a single Event Store (at least to my knowledge) is able to store events and a complex Decision Model. However, any Relational Database (RDBMS) could provide capabilities to support this use case.

Storing current state of the Decision Model and historical events in the same Storage
Diagram 1



Benefit: No Information Loss and Auditability

Alongside storing the current state of the Decision Model, we are storing events (in the same transaction) that led to this state. Obviously, no information is lost - we can audit our system and we can learn from the history. 

But, not using the proper Event Store will lead us to trouble when the number of events increases. RDBMS doesn't perform well when the history table grows to several billions of rows. Adding a new event will slow down the update of the Decision Model which will increase the latency and reduce the throughput of our system in the end.

Although I mentioned that RDBMS doesn't perform well with large history tables, certain RDBMS solutions can be optimized for this use case. We must keep in mind that this optimization is not trivial and affects how we are going to index this large table, how we are going to back up data, how we are going to use JOINS, etc. So, not impossible, but definitely not trivial. Anyway, let's move on and see whether we can solve this issue differently.

We can solve this issue by limiting the size of the history we store in RDBMS - we can move older events to more suitable storage, some form of Event Log for example (see Diagram 2).

Moving Events to Event Log
Diagram 2

In this scenario, we are still persisting the current state of the Decision Model in the RDBMS Storage, but we are storing only recent events in the limited history table. Older events must be moved to the Event Log in order to protect us against Information Loss. For this, we are going to assign an asynchronous process (let's call it the Event Moving Process) to move events from the RDBMS to Event Log. 
If we can guarantee that adding events to the Event Log is an idempotent operation, the job of the Event Moving Process is not so complicated (to some extent). However, this idempotency is not widely supported among Event Log implementations in the industry, which means that our Event Moving Process might not be so easy to implement. It would have to take care of Event Log idempotency. 

The complexity of implementing this approach should not be underestimated!

Benefit: Event Replay

Having events durably stored gives us the opportunity to go back in time and replay (some of) them so we can recreate certain View Models or create new ones. This gives us quite a relief while we are developing View Models since any information we miss in the View Model can easily be added because the information is present in the form of stored events.

By solving the issue of non-performant writes (limiting the size of history and moving older events to the Event Log), we solved the issue of non-performant reads - Event Log is much more suited for reading as well. And we need performant reads for Event Replays. The Event Log is eventually consistent, which is a fact we just have to accept.

Conclusion

In situations where the number of events in your system is not high, using the RDBMS only can fit the purpose. Keep in mind that event-driven systems typically tend to grow very quickly in the number of events. As soon as the history table grows to some extent, RDBMS will not fit the purpose anymore. 
Evolving this architecture by introducing the Event Moving Process is a completely valid solution. But, You should really ask yourself: Is it really simpler than using Event Sourcing backed up by the Event Store?

Event Sourcing is much easier to reason about since we are dealing with a single resource - the Event Store. We don't need to take care of moving events or optimize it for large amounts of data - the Event Store is built for these purposes. There are techniques that can help us prevent loading too many events. The first one is to snapshot the Decision Model from time to time and load only the events between the last snapshot and now. The second one is using the Dynamic Consistency Boundary, which in combination with snapshotting will reduce the number of loaded events even further.

Location: Novi Sad, Serbia

Comments

Post a Comment

Popular posts from this blog

Lock-free Exclusive Processing

-
Image
The problem that I'd like to explore in this article is related to multithreaded processing, and it goes something like this:  Design a component that executes tasks one after the other - there is no overlapping in processing. Tasks could be submitted by different threads. Of course, the most obvious solution would be to synchronize the execution of tasks. Each thread would try to acquire the lock on this object if possible, otherwise, it will block and wait for the lock to be released. While waiting for this lock to be released, the thread cannot do anything else. This approach creates a lot of contention and besides that, acquiring and releasing the lock is quite an expensive operation. So, it is not performant as well. However, we must admire the simplicity of the solution. Is there a more performant solution that does not block the thread which wants to execute the task?  Firstly, let's decouple task submission and task execution - a queue sounds like a perfect fit for the
Read more

Ordering in Event-Sourced Systems

-
Image
In a utopian world, Event Store will write events to the disk in the exact same order in which they have happened in real life. However, computers are not so good at observing things by themselves (yet), we (still) must tell them what to do. Hence, we must provide a way to deterministically order events inside the Event Store.  A naive approach would be to attach a timestamp to each and every event and use it for ordering. The reason for its naiveness lies in the difficulty of synchronizing clocks across different machines in a distributed system. As a consequence, we mustn't rely on timestamps for ordering . Since we also cannot impose ordering of events sent by different Event Store clients among them, the only thing left is to accept the ordering of received events by the Event Store. In other words, the order in which Event Store receives events is going to be the order of events written to the disk. This means that two events could have happened in one order in reality, but t
Read more

Myths about Distributed Business Transactions

-
Image
Nowadays, most systems are distributed in one way or the other - just accessing a database via HTTP makes the system distributed. Not so rarely, there is a message broker in the mix which makes a good soil for Distributed Business Transactions. There is a lot of mystery about them, they get confused with ACID transactions easily. I'm going to address common myths and misunderstandings about Distributed Business Transactions and provide some options on how to handle multiple independent resources. Before we start, let's try to understand what a Distributed Business Transaction is. Distributed Business Transactions Distributed Business Transaction is a business transaction that spans at least two independent  transactional resources.  One example would be a database update and sending a message via a message broker (see Diagram 1 ). Those two resources have their own transaction management and are completely independent. Another example would be updating two different files on a
Read more