00:00:23.910
Hello everyone! My talk today is about applying Event Sourcing and CQRS together on Rails applications. I will start with a kind of round trip, showing you how each piece of this looks.
00:00:31.099
First, let me introduce myself. I am the CTO of a Berlin-based fintech startup, and I am currently working at ZenCargo. I have a strong interest in Domain-Driven Design, Event Sourcing, CQRS, functional programming, and various technologies. Over the years, I have utilized a variety of programming languages and tools.
00:00:45.180
This is the agenda for today. I will cover several topics that I would like to mention. I won’t go into too much detail since there are many aspects to discuss, but I will show you some real code examples and how we deal with these concepts in practice.
00:01:12.720
First of all, I want to reiterate the importance of Event Storming, which was mentioned yesterday. To be honest, I believe this is one of the most important parts of implementing Domain-Driven Design in your company. You can attempt to perform Domain-Driven Design, but without genuine cooperation from the business side, it won't work at all. The best way to make this successful is to invite business stakeholders to collaborate with you directly.
00:01:42.950
To facilitate this collaboration, you need simple tools that they can use. If you show them UML diagrams, they probably won’t find it enjoyable, and there will be a disconnect because we operate on a technical level while they use their own domain language. The idea behind Event Storming is to utilize simple tools like sticky notes in various colors, combined with a large space, such as a wall covered with paper. Everyone has their own pen and sticky notes, which have different meanings. The most important sticker represents the events.
00:02:17.880
You begin the workshop by explaining its purpose. You present a timeline and select one domain expert to provide an example of an event that occurred in the business process. We won't focus on the technical details during this part; instead, we are interested in the business processes. Once they provide an example, you can ask questions about what happened before, afterward, or concurrently, which allows you to gather all the events on sticky notes.
00:02:48.940
As you collect these event sticky notes, more and more insights are recorded. Everyone works in parallel so they don't interrupt each other, and they can add as many notes as they want. If there are questions about open topics, they can use different colors for these questions or notes, which can be set aside for later discussion without disrupting the flow of thought for others who are formulating their domain in sticky notes. When you have numerous event sticky notes, you will slowly start to identify your main subject areas.
00:03:40.660
Once you have identified the main subjects, you can start to arrange the cards and think about the sequence and relationships between each event. You can dive deeper by adding more questions to help you understand the conditions surrounding each event in your business process. You can also use different colors for external services or to express UI needs if necessary, as there are no limits here. However, it's essential that everyone agrees on the meanings of the colors and the corresponding concepts.
00:04:55.750
This slide shows an example of how the result looks after a simple session where you have added a multitude of sticky notes, each representing different questions, on the wall. The reality is that you cannot cover everything in just one step. Event Storming is inherently iterative; you will need multiple iterations to reveal a comprehensive understanding of the domain. You have to keep in mind that it's vital not to force people into lengthy sessions, as they may become unmotivated and start to overlook crucial aspects.
00:05:44.660
As mentioned earlier, it is also essential to ensure people step outside their comfort zones. They must interact, and you, as the facilitator, need to observe them. Some individuals will readily share extensive insights, dumping all their knowledge about the business process onto the table, while others may try to hide in the background and avoid engagement. You'll need to adapt your approach accordingly to encourage participation from all types of personalities.
00:06:49.610
The result of such Event Storming sessions can look like what was mentioned yesterday. We utilize a real-time board to allow everyone to work in parallel. Each participant has an open screen, and this methodology works remarkably well remotely. However, it can be challenging to gauge their reactions, but it prevents interruptions as everyone takes notes in one dedicated space, and then they can later rearrange them appropriately.
00:07:31.300
Using different colors to indicate which questions have already been answered is also beneficial. You can choose to keep them for reference rather than removing them, allowing you to track what was previously questioned, which might be useful later. Event Storming is a good alternative to the traditional approach. I’ve frequently seen requirement sheets that span multiple pages detailing expectations and deliverables from our work. However, the issue is that if I were to take those requirements and randomly select five different stakeholders from the room, they would likely produce five different implementations.
00:08:51.280
This discrepancy arises because everything comes down to perception regarding how the requirements are understood. Therefore, if you have universally accepted conventions in place, it’s manageable; however, if people subscribe to different frameworks and libraries, the implementation nightmare ensues.
00:09:27.800
If you need more information about Event Storming, I suggest revisiting the book on the topic. There are great resources available, including multiple presentations from Alberto Brandolini regarding this aspect of Domain-Driven Design. I won’t delve too deeply into it since it has been covered extensively already. I recommend exploring Google for the topic.
00:09:59.790
In the realm of Domain-Driven Design, I would like to briefly touch on Bounded Contexts. My understanding of Bounded Context is that it is not just about encapsulating business logic and separating it from other logic; it’s also about creating boundaries that ensure singular meaning for the same concept.
00:10:26.920
For instance, in an application that deals with B2B and B2C customers, there are two distinct types of customers, and they will never reside in the same Bounded Context. This way of thinking ensures that there is only one interpretation of certain entities within the bounded context.
00:10:59.960
We are using the Grace Event Store, and I would like to share some rationale behind choosing this platform. First off, it has excellent documentation, making it user-friendly for newcomers. It is actively maintained and well-supported by the community, which is crucial for ongoing development.
00:12:10.280
One of the advantages I found while collaborating with the team is that after submitting a pull request, they promptly grant me access to manage my implementations without micromanaging, which fosters autonomy. Today, I’d also like to discuss how to describe the domain. While you might have numerous sticky notes, events alone do not encapsulate everything within your domain.
00:13:03.960
For this purpose, we utilize Dry Types, Dry Struct, and Drive Validations. We are gradually moving away from Drive Validations, mainly because these libraries tend to be highly opinionated, often restricting implementations to a singular approach that may not support other frameworks or libraries. This example illustrates how different domain types can be amassed to effectively express what resides in your domain.
00:14:10.590
The beauty of Dry Types is that they behave somewhat like static types, allowing you to define constraints such as Union Types. By leveraging Dry Types, you can encapsulate the values in your domain by defining them within a Dry Type. One practical application is creating your constructor, for essential types like money, which wraps existing types in Dry Types for future reuse.
00:15:02.420
This second example demonstrates a domain that showcases different types and constraints from another application. Here is an overview slide that showcases all of the components I will explain today. Event Sourcing is not an entirely straightforward concept; it requires a solid understanding before diving in.
00:15:29.990
Each piece of the software plays a role in a broader context, and I will also explain the modifications we made to libraries provided by our K&C team tailored to our specific needs. If you want additional insights into Event Sourcing, there were many excellent questions and answers discussed today. However, again, I urge you to revisit the presentations by experts to gain varied perspectives on this subject.
00:16:12.930
Let’s start with the user interface. We developed a component that behaves like a comment form, which also has dimensions of regular forms. I would argue that comments are distinct entities in their nature. We created this segment of the software to operate externally as a standard ActiveModel form object.
00:17:03.460
The appealing aspect of this is that you can integrate any nested Dry Struct type within it, enabling the creation of any complex form using standard fields. An important feature of the comment form is that you can define multiple commands based on the command form. For instance, if a large form contains numerous fields and only the 'new password' is entered, that command associated with changing the password will be triggered.
00:17:52.010
By detecting which fields were altered in the command forms based on the input provided by users—regardless of possibly numerous other commands present—you can accurately determine which action to dispatch through your system. There’s also a method to define weak fields; some fields may appear in various commands, and you wouldn’t want to trigger command dispatch when referencing just a related item.
00:18:23.920
In our implementation, we utilize Dry Structs to define our commands. One aspect is creating a schema type that means if you submit this as a command input and do not provide necessary information, nothing will happen. We ensure validation integrity, especially for events, by storing comprehensive information and avoiding loss of vital details.
00:19:47.920
For instance, our controller using Event Sourcing looks like this. We invoke an update utilizing all patterns that match specific commands and provide context such as the user's email, location, or IP address. The Save method for commands is fairly straightforward; it simply verifies that validations pass before we dispatch the necessary commands that detect changes within the form.
00:20:51.600
Now about validations: in prior implementations, we relied on Drive Validations defined on the commands. However, we've switched to defining these on the command form because command validations pertain to the user input process rather than core business logic. Hence, this allows you to map multiple nested validations, combining types and value objects as needed. You have access to all standard validations from ActiveModel and ActiveRecord.
00:21:45.720
Let’s discuss command handlers. This example illustrates most of our command handlers, detailing aggregate information along with aggregate ID. The aggregation helper assists in creating the comment aggregate and aggregates its past events, ultimately returning an instance that we utilize later for methods on comments.
00:22:52.100
Here’s how the helper looks: first, it performs the aggregate class creation by passing the aggregate ID as a finished value. We then yield the already hydrated aggregate instance since it initializes the stream name and retrieves all past events from the store. This way, you can execute any business method on it. There are multiple definitions of what an aggregate or aggregate root represents.
00:23:59.890
From an event sourcing standpoint, the aggregate root embodies the primary object tied to executing your business logic, maintaining transactional boundaries, and housing all serialized events in the event store. In the more domain-driven design context, the aggregate root serves as the public interface for the entire tree of interrelated objects.
00:25:00.780
For instance, if you possess an invoice model, you will have various components inside, including payment details, addresses, etc. You may directly modify these items with a raceway, but with an aggregate root, you should invoke methods that comprise the public contract instead of altering properties directly. This systematic approach fosters disciplined interactions.
00:25:59.640
Here's a brief illustration of how a typical constructor of an aggregate might look: two categories of methods exist. One category comprises command methods that accept command requests, while the other consists of callback methods designed to modify the state of the aggregates. This fundamental distinction exists because you want to avoid executing code that leads to side effects while rebuilding the aggregate from scratch.
00:27:34.760
An example of command form and command methods demonstrates how we validate business conditions while applying events to the aggregate instance. After fulfilling the necessary criteria, the subsequent callback method is executed with the event that was applied, while the state of the aggregate mutates accordingly. If an error occurs, we never reach that point, implying the aggregate won’t persist.
00:28:23.480
The aggregate maintains data around all yet-to-be-published events that were applied since the last retrieval. In our exploration of patterns, we researched the concept of nested aggregates having entities which can grow unwieldy over time.
00:29:32.760
This raises the question: how to split them effectively? One approach is implementing nested aggregate roots that include multiple aggregates. However, I’m not entirely fond of this methodology. Instead, I believe in extracting code into strategies and policies, allowing for segmentation of logic that can be individually tested without creating an instance of the aggregate itself.
00:30:18.150
Here’s an example illustrating how to apply a book builder and strategy to the aggregate, enabling you to reuse this pattern effectively. Speaking of aggregate roots, you can manage states independently without engaging in event sourcing, merely by holding the root object.
00:31:11.390
For example, if you're utilizing Rails' Active Record, the foundational rules still apply where you maintain public contracts for all linked objects. However, a challenge arises in Ruby due to the lack of interfaces that aid in scoping. When providing instances of aggregates, one can still invoke destructive methods on them or manipulate connections directly.
00:32:03.740
So, I conducted research, developing a straightforward wrapper that provides a semblance of interfaces in Ruby. This enables you to define methods clearly, which allows you to specify objects that implement particular interfaces. As a return, even if additional methods exist, only those defined in the interfaces will remain accessible.
00:33:24.329
Let’s talk about operational changes in a system. It’s never true that we prepare systems and make no alterations; we continually reveal domain complexities and initiate multiple changes along the way. The most effective strategy for aggregates involves establishing a small mixin to define aggregate names.
00:34:16.560
In case we need to refactor the aggregate for improved naming, we can maintain the original name to build the stream name without altering the event store, which should inherently be immutable. Moving on to Sagas and Process Managers, it's often the scope of an aggregate isn’t wide enough to facilitate cooperation. That’s where Sagas and Process Managers come into play.
00:35:27.390
Sagas represent long-running processes, such as purchase verifications, whereas Process Managers coordinate the work of multiple aggregates to achieve overarching business objectives. Here’s an implementation sample of how we organize Sagas within our system. Similar to aggregates, we also conceived Sagas in the form of aggregate roots, containing events detailing the saga's progression.
00:36:30.970
I recommend revisiting Robert’s previous talk; it thoroughly dissects the intricacies of Sagas and their relationship to business logic. Now let’s delve into the subject of events. Not all attributes are simple values; some represent value objects encasing additional information, enabling their incorporation into a Rice Event Store.
00:37:21.770
We developed our class to utilize Dry Types and exemplify event versioning. Below are examples of how we employ the Ruby Contracts library to validate infrastructure code, ensuring it adheres to constraints that prompt immediate failures when objects do not conform to defined contract methods.
00:38:21.740
One notable feature of Ruby Contracts is the inclusion of a simple pattern-matching implementation. We recognized the necessity to support three different use cases. Implementing this without pattern matching proved problematic; it muddled the code. Thus, we opted to divide it into three methods to enhance clarity and understandability. This does come with certain penalties; being another tool means Ruby Contracts are disabled in production, but they remain active in testing and development environments.
00:39:15.740
This leads to a key point: our events have strict schemas with default settings, which means if one forgets to provide critical fields, failures will occur immediately, allowing early recognition of mistakes. Regarding event versioning, we found a simplistic method for upcasting events from one version to another.
00:40:10.360
This approach accommodates necessary changes since we may adjust our understanding of the domain. It's crucial to define a version number within events; by default, we begin with version one. You must equip every event with the corresponding callbacks to facilitate migrations.
00:41:13.159
Significantly, you should refrain from relying on external states within these callbacks since view models can shift, refresh, or become deleted entirely. This loss of information can incur severe complications when updating an aggregate.
00:42:08.480
Additionally, when transitioning event names, if classes were defined earlier, Ruby’s handling of those names in the database increases complexity as maintaining old class definitions becomes burdensome. That’s why we designed an event mapping to connect old class names with new definitions.
00:42:46.078
Working with Rails and Rice Event Store introduces multiple configurations, such as event and command handlers. If you do not manage the configuration correctly, reloading the application could lead to multiple subscriptions to the same events, causing confusion and bugs. Below is a configurational example we implemented, outlining event handler strategies.
00:43:43.240
This illustrates one method of organizing your event handlers and how events can be de-normalized. The de-normalizers allow you to gather all events published after they have been applied to the aggregates. Consequently, you build your read models against those events.
00:44:43.410
Utilizing event sourcing enables substantial decoupling between the actual state held in aggregates and how the data is represented in your application. You can have different read stores that are optimized for specific views, circumventing the necessity to default to ActiveRecord or Postgres.
00:45:08.580
It is important to recognize that when crafting read models, those may not encompass all data needed to render a particular page. It may result in needing to append fields later. The best strategy for this is often to clear out existing information and rebuild the view model from the past events while incorporating any newly added elements.
00:45:52.410
Let’s break down event handlers and de-normalizers briefly. Event handlers are designed to take the series of events emitted after being applied. The de-normalizers listen for these events, enabling the real-time update of read models. This decoupled architecture carries the benefit of allowing for numerous read models, essential for efficiently handling various data requirements.
00:46:35.620
One challenge I encountered when utilizing the Rice Event Store revolves around error-handling strategies. If an error emerges in any handler, it can disrupt the chain of subsequent events that may retrace other events. If the information track is lost due to one broken link in the chain, I'll be unable to reveal subsequent events.
00:47:16.640
This necessitated developing a strategy that defines how errors should be managed. My focus would enable notifications via tools such as Airbrake, but other event handlers would still execute uninterrupted.
00:48:04.250
When creating view models, these may very well just be simplistically constructed ActiveRecord models with additional abstraction layers, adhering to the CQRS model. The queries we apply can aggregate data from different sources, not strictly confined to ActiveRecord, potentially involving ElasticSearch for specific data searches.
00:48:55.840
In reflecting on the learning process derived from implementing these methodologies, I recognize that the complexity of grasping Event Sourcing requires commitment. Numerous resources and presentations exist, alongside several books about these topics; however, each source contributes uniquely to the intricate understanding of the subject matter.
00:49:44.340
Collecting and categorizing diverse information into a mind map has been beneficial. It consolidates information, allowing for real-time referencing of use cases and patterns identified during the learning process.
00:50:14.410
This approach has become crucial when discussing possible improvements and prototyping new features with my team. Relying solely on the latest articles can lead to confusion and disconnection from past knowledge.
00:50:55.550
This is an overview of our current infrastructure code, specifically developed for utilizing the Rice Event Store. While you don’t require every single aspect I presented, these patterns emerged throughout the development, instilling best practices we learned along the way.
00:51:31.080
I have access to our collaboration tools with the Rice Event Store team, which has catalyzed many discussions around the various components discussed here.
00:51:50.140
Thank you for your time, and I’m open to any questions.
00:52:05.010
I have two questions. First, could you estimate the percentage of your time spent maintaining this infrastructure? What percentage of it is complete?
00:52:41.130
It appears I didn’t preemptively incorporate signal repairs. When we began our implementation, a few key questions and issues arose that would need addressing within the infrastructure code.
00:53:50.530
For example, developing the simple form as a command took around two to three days to bring the basics online. Integrating ActiveModel functionality—like the second version—was substantially faster, taking only a day and a half. This work involved pairing which reinforced the importance of collaborative brainstorming.
00:54:47.930
That’s how we ensure use cases are clear, validate tests, and check that all functionalities are functioning as intended. Thank you!
00:55:54.860
Do we have any other questions?
00:56:20.510
When dealing with similar architectures, I’ve noticed how a simple task—like adding releases—can generate a lot of boilerplate.
00:56:56.070
This can include commands, command handlers, event handlers, and other elements, leading to overwhelming complexity.
00:57:37.040
Have you considered using tools, such as generators or scaffolding to streamline this process?
00:58:06.860
Regarding Domain-Driven Design, many in the Java community have spoken about converting diagrams into code automatically, but I find that approach counterintuitive.
00:58:55.970
Being overly reliant on generated code can lead to difficulties in maintenance. Thus, having well-defined patterns and structures has multiplied the ease of onboarding new developers.
00:59:36.750
This structure grants clarity around how various components work together, which is the optimal way for any emerging developer to engage with a complex system.
01:00:22.610
In summary, delineating well-defined architectures and interfaces can facilitate coherent development practices that are robust over time.