Event Sourcing

Talks

Event Sourcing

Bryan Reinero

#domain-driven-design

Event Sourcing

by Bryan Reinero

Event sourcing, as presented by Bryan Reinero at GoRuCo 2015, introduces a paradigm shift in how we manage and store the state of domain objects in transaction processing. Unlike traditional methods that persist the current state of an object, event sourcing records an immutable log of changes, or events, to the database. From this log, an object’s state can be reconstructed at any moment by replaying the events sequentially. This talk emphasized the importance and maturity of this concept by outlining its various applications, especially in system reliability and historical data tracking.

Key Points Discussed:

Definition of Event Sourcing: Event sourcing focuses on persisting the history of changes (events) rather than the current state of the data, making it easier to recover from data loss or corruption.
Example of a Banking App: Bryan illustrates event sourcing with a banking application where transactions are logged, allowing for a complete history of actions (e.g., transactions between users A and B) rather than just current balances.
Security and Recovery: With event sourcing, if an attack or data corruption occurs, the system can regenerate current states by replaying events up to the point of failure, enhancing security and traceability.
Historical Context: He also points out that event sourcing is not a novel concept; its roots can be traced back to historical practices such as the dual-entry accounting system introduced by Luca Pacioli and records from Augustus Caesar.
Modern Applications: Event sourcing is likened to version control systems like Git and is utilized in databases, such as MongoDB, where operations are logged to maintain data consistency across nodes.
Challenges of Event Sourcing: Bryan addresses potential performance issues, noting that deriving the current state from a full event log every time can be inefficient. He suggests creating snapshots to optimize this process.
Command Query Responsibility Segregation (CQRS): The talk discusses how CQRS complements event sourcing by separating write and read operations, which allows for scaling and improved performance, particularly for read-intensive applications.

Conclusions:

Bryan concludes by expressing enthusiasm for further discussions on event sourcing and encourages attendees to reach out with questions. The key takeaway is that event sourcing represents a powerful yet historically established approach to data management that enhances system reliability, security, and flexibility in application design.

00:00:14.170 Oh, everyone! My name is Bryan Reinero, and I work as a software developer advocate at MongoDB. Another thing about me today is that it’s my birthday! So, standing here means my wish came true. Let's talk about event sourcing: what it is, how it can be ubiquitous, and how I can assert that you've never heard of it before. More importantly, why might you want to use it? Well, let's dive into an example: let's build a banking app.

00:00:30.850 If I’m building a banking app, I’m probably going to need to persist state in some form back on my data store. This seems like a pretty obvious problem to solve. I’m going to have users A and B, and they both have balances. The balance describes the current state of their accounts. When I want to do a transaction between the two accounts—let's say User A buys something for five dollars from User B—that’s easy. I just credit the account of User B while simultaneously taking away five dollars from User A. However, the problem is that if I’m thinking this way, I may already be doing it wrong.

00:01:06.580 Why? By design, I’m throwing away useful information. The only thing I’m persisting right now is the current state of the users. I’ve lost the history of what I’ve done or how they got there. So, if there’s any corruption or data loss, I can’t trace back to what happened. I can’t recover from an attack where someone changes my state. Instead, event sourcing was first proposed by Greg Young and Oody DeHaan, and it is often used in conjunction with Command Query Responsibility Segregation, or CQRS.

00:01:52.179 Event sourcing tells us to forget about the current state. We care about the current state, but we won't persist it. Instead, we will log events that change our state. In this case, instead of updating a user’s account directly, I’m updating the transactions that would change that account. I can derive my state by replaying the event log. What’s more, if I lose the current state, I can always regenerate it by replaying the event log. I can even look back and tell you what the world looked like at any point in time, allowing for temporal queries.

00:02:26.260 This is actually quite useful if I've learned something new about how users interact with my tool or website. I can go back and see how many people I could have reached in the past, provided I’ve kept that information in the log. Now, is this more secure? Consider if someone tries to attack me by sending erroneous data. No problem, because I have a log of the attack. I can replay up to the point where I detect the attack, and since I have a log of the attack, I can analyze it and protect myself in the future.

00:03:01.569 Now, for those of you who might think that this is a radical new idea, it’s not untested; it's surprisingly mature. This concept dates back to ancient history. For example, the 'Res Gestae Divi Augusti' translates to 'The Divine Deeds of Augustus,' which is Caesar’s record of payments to his Legionnaires. He recorded payments rather than maintaining a current state so he could know his accounts at any given time. This illustrates a form of event sourcing—durable storage of important information.

00:03:49.720 For those of you who appreciate technology explained in modern contexts, you might already guess that this pattern is used in version control systems, like Git. When I change something in a Git repository, I'm logging the changes in a way that allows me to always rebuild the current snapshot or revert back to a previous state. This also includes merges and branches that can be complex but powerful, particularly in continuous integration environments.

00:04:53.800 Similarly, databases also leverage event sourcing internally. In the case of MongoDB, we use replication. Our primary node writes to what's called the OP log, which records write operations from the database. Secondary nodes then follow the OP log to stay consistent with the primary. The neat thing about the OP log is that, similar to event sourcing, I can start at any point in the past and replay to achieve the current state.

00:05:12.220 There’s a challenge with event sourcing: if every time I want to derive my current status I have to go back through the entire log from the beginning, it can be time-consuming and may blow out my cache. The solution is to create snapshots at given intervals. When I need to go to a targeted state, I'll simply go back to the last snapshot before the target and replay the OP log from that point.

00:06:17.430 This approach actually has roots in the 14th century when a man named Luca Pacioli introduced dual-entry accounting. In 1495, he documented an accounting system where transactions were not just recorded as, say, 'I owe you,' but in a way that allowed for both accounts payable and receivable to be tracked. This innovative method supported explorations during the age that followed. We can correlate this dual-entry method with Command Query Responsibility Segregation (CQRS), separating commands that write operations from queries that read data.

00:07:02.270 CQRS inherently recognizes that the way we command—write operations—differs from how we read or query data. Given that read operations usually far outnumber writes, separating these operations allows for optimized scaling. This separation also implies that if I’m asymmetrically scaling my readers, I can use eventually consistent data sources, such as MongoDB, to handle loads distributed across secondary nodes. So that’s basically event sourcing and CQRS. I'm looking forward to discussing this with you further! And one last thing, you can reach me at @blimpyacht. I love reaching out and discussing these kinds of topics, so hit me with your best shot. Thank you!

GoRuCo 2015