Kafka

Summarized using AI

Events. Events. Events!

Anton Davydov • September 06, 2019 • Bangkok, Thailand

The video titled "Events. Events. Events!" presented by Anton Davydov at RubyConf TH 2019 discusses the concept of event sourcing and its implications for software architecture. The presentation highlights the shift from focusing on current state data in applications to the significance of capturing events to understand the application’s history and state at any given time.

Key Points Discussed:
- Introduction of Event Sourcing: Event sourcing revolves around the idea of logging events that happen in an application rather than merely tracking the current state. This allows for historical context and the ability to reconstruct states from events.
- Types of Events: Examples provided include commits in version control systems and user actions in e-commerce applications, both of which demonstrate the importance of tracking changes over time.
- Storing Events: Various storage options for events are discussed, including Redis, Kafka, and SQL databases. The nature of events must be immutable, meaning once an event occurs, it cannot be changed, but new events can be created to reflect updates.
- Data Evolution: Event schemas need to evolve over time while maintaining historical integrity, allowing for backward-compatible changes.
- Projections: The concept of projections is introduced, whereby the current state can be derived from stored events through a predictable processing function. It addresses concerns over performance regarding the accumulation of events over time.
- Real-world Applications: Examples of event sourcing applications include financial transactions, version control systems, and audit trails. These help illustrate how detailed event logs can enhance debugging and state reconstruction.
- Challenges: The complexities of implementing event sourcing include debugging difficulties, version compatibility, and regulatory compliance related to data privacy.
- Advanced Techniques: Methods for real-time data presentation, like synchronizing databases and using WebSockets for immediate updates, are discussed.
- Implementation in Ruby: Libraries such as Rails Event Store and Sequent are recommended for implementing event sourcing in Ruby applications.

In conclusion, Anton emphasizes that while event sourcing can be complex, it brings significant advantages in managing data and understanding application functionality over time. Practicing these techniques, along with fostering a deeper grasp of business events, can make event sourcing a vital architectural strategy in software development.

Events. Events. Events!
Anton Davydov • September 06, 2019 • Bangkok, Thailand

We build different systems and usually think about the current state of our application. Usually, it’s data from our DBs. But what will happen if we start thing about the list of events which can help us calculate a state of the application for any time?

RubyConf TH 2019

00:00:07.170 Hello everyone! Hello Thailand! I want to thank you for having me here.
00:00:15.640 It's my second time in Thailand and my first time in Bangkok. Yesterday, we tried to install Party Jam.
00:00:23.260 I found it, but I'm not sure if I really want to install it. So, this is not my first trip to Asia.
00:00:31.960 This is why I really know what jet lag is. It’s tough because I have no idea whether to wake up at 4 a.m. or go to sleep at 3 p.m.
00:00:44.680 Yesterday, I was inspired by a team I really love. I tried to find something interesting on the Internet.
00:00:55.930 Unfortunately, I found nothing useful. I thought perhaps looking for 'Quest of Rubies' would be super complicated on Google, and I tried to find Ruby Nia but wasn't sure if it would be helpful.
00:01:09.119 So, okay, my name is Anton. I am from Moscow. You can find my contact information if needed.
00:01:15.130 I'm a top talent architect, and I can use this opportunity to share some insights. It's my favorite topic involving architecture.
00:01:32.920 Some people know me as the guy who works on Crimea and strong systems. I really love open source. Right now, I’m a core developer.
00:01:58.500 However, I know the theater world as well. Sometimes, I stream on Twitch. You might want to know that I really love stickers.
00:02:07.350 I have a collection of stickers, and I also love burritos. I even have a photo of myself in Minsk enjoying a burrito. I love that photo because I look like a person who could be on a Japanese TV show.
00:02:28.900 If you want to talk with me, we can discuss coffee, beer, or psychology. I have a Nintendo Switch, and I really love to draw images for our presentations.
00:02:49.480 The first drawing was done by me, and I also enjoy writing good stories.
00:03:04.090 I expected to be like a certain type of storyteller, but unfortunately, reality turned out to be a bit different.
00:03:18.820 However, I consider myself lucky because I got help from my friend Georgie. He is really quiet and sometimes asks insightful questions.
00:03:37.900 Today, the title of my talk is 'Events.' We will discuss these in general. In the real world, we encounter numerous events.
00:03:56.620 As developers, we often need to capture these events and take action. For example, does anyone know what a commit is?
00:04:11.230 Indeed, we can say that each commit represents something that happened. For instance, I have a dynamic contributions application, and we keep a list of commits.
00:04:28.810 Each commit contains a description of changes, metadata, and information about what happened.
00:04:43.540 If we open a commit from a repository, we see that it includes a description of changes, the associated files, and metadata about who made changes and when.
00:04:57.310 Another example of events is what happens when a user tries to purchase an item in an online shop application.
00:05:11.790 We need to track exactly what the user added to their order and what was deleted from it.
00:05:26.200 We have to sometimes work with deleted activity. My favorite topic is tracking deleted data in the database.
00:05:44.340 Unfortunately, in our applications, we mostly deal with the current state, meaning we focus only on what is happening at this moment.
00:05:57.310 In this case, we say our events reflect our actions, and the current state is our data in the database.
00:06:08.190 This leads us to the point that we often look at current data rather than considering how to store event histories.
00:06:24.880 So, you might be asking, what would happen if we changed our approach to start storing events instead of focusing only on the current data?
00:06:45.540 In this case, we would enter the realm of event sourcing. Today’s discussion covers both event sourcing and its significance.
00:07:04.080 The first rule of event sourcing is that nobody knows what event sourcing is. The second rule is that you must store what happened with the application.
00:07:32.110 This involves capturing events rather than simply storing the current state.
00:07:49.300 The first question you should ask yourself is where to store these events. You can keep a list of events or a chain of events.
00:08:12.770 In event sourcing, event storage can take various forms—it can be Redis, Kafka, MongoDB, or SQL databases.
00:08:33.720 Event storage has certain rules: the first is that an event must have occurred in the past.
00:08:52.850 We may not know how long ago it took place, but it’s important to remember that it happened.
00:09:07.400 The next rule is that events should be immutable. We cannot update, modify, or delete events once they occur.
00:09:20.370 Instead, we can only create new events to adjust for changes or updates.
00:09:41.740 Let’s consider a simple example of event storage: when a user is created, we log that event. Later, if we want to update the user's information, we don’t change the initial event but create a new one.
00:10:05.800 The next challenge in event storage is ensuring that we have valid data in each event.
00:10:21.570 This means validating data against our database before we store an event.
00:10:41.040 The second rule of event sourcing is data evolution. This implies that events must adapt over time without altering the history.
00:10:58.250 Documentation must remain unchanged while allowing increased complexity in the application.
00:11:15.110 Data evolution requires us to accommodate different event schemas concurrently.
00:11:33.070 It's crucial to remember that the application could experience significant changes over time.
00:11:53.290 Now, let’s move on to how we can derive the current state from an event store.
00:12:12.580 In event sourcing, we use projections to calculate specific states from the event history.
00:12:29.240 Projections act as functions that take the list of events and an initial state, then compute a specific state.
00:12:43.180 In this context, the projection function is inherently a pure function, and its behavior is predictable.
00:13:03.100 To illustrate, after a user is created, we can accumulate events and check if the user has been updated.
00:13:22.670 The concern, however, is that recalculating the current state every time can slow down performance.
00:13:37.320 Especially when events can span 2 or 10 years, handling accumulation can become a hefty task.
00:13:56.950 In Ruby or other languages, performance issues may arise, but there are ways to resolve this.
00:14:09.960 We could introduce abstraction with colored streams to better manage smaller sets of events.
00:14:26.020 For instance, with two different event sources, we could create specific streams for individual sources.
00:14:45.840 This would allow us to only process relevant events associated with each aggregate.
00:14:59.470 Additionally, we could implement snapshots of the current data state to improve retrieval speed.
00:15:14.800 These snapshots could be taken at intervals, allowing us to quickly access recent states from a database.
00:15:34.360 The general advantage here is that it enables us to leverage different kinds of databases for caching as well.
00:15:45.910 For instance, relational databases serve effectively for searching and indexing large datasets.
00:16:04.340 Conversely, document-oriented databases might be more useful for presenting detailed content.
00:16:20.190 Now let’s delve into real-world examples of event sourcing, particularly in e-commerce systems.
00:16:38.450 During order and checkout flows, it's crucial to track what actions were taken, making it easy to debug and recalculate state.
00:16:59.200 Event sourcing proves valuable in domains dealing with money, as it allows for detailed auditing of transactions.
00:17:15.580 It is also applicable in version control systems, like Google Docs, where changes can be rolled back.
00:17:31.920 Additionally, tracking systems often utilize event sourcing to calculate statistics about occurrences.
00:17:47.800 Regarding frameworks like Redux, it's important to note that event sourcing and event-driven architectures differ.
00:18:03.170 In event sourcing, we store events that occurred in the system, while in event-driven architecture, events happen and impact all components.
00:18:20.160 A good example of event sourcing can be seen in blockchains, which serve as excellent references for how this architecture functions.
00:18:36.800 Now, let’s explore the pros and cons of event sourcing. First, communicating with domain experts is easier.
00:18:54.650 We can clearly understand business events since detailed logs can be generated automatically.
00:19:12.640 Event sourcing allows us to recover lost system states whenever needed, but this comes at a cost.
00:19:30.170 It also shifts developers' mindsets; we no longer think in terms of table-based storage but rather of changes in the system.
00:19:47.890 However, event sourcing has drawbacks; it can be complicated to implement and is not popular among some developers.
00:20:01.380 Additionally, debugging can be challenging since tracing through numerous events may become cumbersome.
00:20:17.220 Version compatibility remains a concern, particularly if migrations require previous versions to be maintained simultaneously.
00:20:33.030 In Europe, data privacy regulations also impose restrictions on event management and deletion.
00:20:49.800 Now, let’s address some advanced topics in event sourcing.
00:21:07.400 When we want to display data as quickly as possible, we can apply two techniques for real-time data presentation.
00:21:29.970 The first involves synchronizing the client database with the event database as events occur.
00:21:47.490 The second strategy involves using WebSockets or long-polling to keep clients updated on changes in real time.
00:22:07.620 Another common question is how to ensure relevance ordering in a multi-application environment.
00:22:26.840 We may need to implement distributed locks or other strategies to solve issues of event collisions.
00:22:45.290 It’s also essential to think about how to deactivate or delete related events, particularly concerning data privacy.
00:23:05.800 Thus, encryption plays a role in preserving sensitive information while allowing for valid deletions.
00:23:20.100 If we have event lists, lineage is essential in efficiently retrieving information, like specific user activities.
00:23:38.440 We can achieve this by using techniques such as indexed queries and read models.
00:23:56.060 Finally, let’s talk about implementing event sourcing in Ruby. The most popular libraries available include Rails Event Store.
00:24:12.680 Also, Sequent offers a comprehensive list of features related to event sourcing.
00:24:25.900 Additionally, you can build your components if you desire a custom experience, leveraging libraries such as Kafka.
00:24:39.760 In regulated industries, incorporating event sourcing guarantees better control over changes, enabling audits and validations.
00:25:02.040 As I share this knowledge, using concepts from workshops and stickers has been instrumental in conveying intricate details.
00:25:24.540 I also created an experimental repository demonstrating how to implement these techniques effectively.
00:25:45.820 In conclusion, while event sourcing can be complex, it provides significant advantages for long-term data management.
00:26:07.100 The key is to understand the business events and leverage these techniques to handle complexity in distributed systems.
00:26:28.560 Ultimately, though complicated, mastering event sourcing can lead to better data management and clearer insights into application histories.
00:26:46.550 I believe event sourcing will become even more relevant in the future, and I encourage you to explore this area further.
00:27:06.460 Thank you all for your attention, and I'm happy to take any questions!
Explore all talks recorded at RubyConf TH 2019
+6