Ruby Video
Talks
From ActiveRecord to Events
Emanuele Delbono
@emadb
#event-sourcing
#cqrs
#ruby-on-rails
#domain-driven-design
#activerecord
#software-architecture
#mongodb
Summarized using AI

From ActiveRecord to Events

by Emanuele Delbono

The video 'From ActiveRecord to Events' by Emanuele Delbono, presented at the wroc_love.rb 2014 event, explores the limitations of traditional ActiveRecord models in Rails applications and advocates for the adoption of Event-Driven Architectures, particularly Event Sourcing and CQRS (Command Query Responsibility Segregation).

Key Points:

  • Traditional ActiveRecord Limitations:

    • ActiveRecord tightly couples application models to database structures, complicating persistence as applications scale.
    • Inner workings of modern applications can become convoluted due to the mixing of data and behavior in domain models that lead to anemic entities.

  • Domain Model Challenges:

    • Difficulty in handling complex models: a large domain model can create a monolith that makes it hard to separate concerns.
    • The N+1 query problem arises during lazy loading, violating the Single Responsibility Principle as models handle both data persistence and domain logic.

  • Shift from CRUD to Business Operations:

    • Emphasizes the need to move away from traditional CRUD thinking and focus on user-driven business operations.

  • Command Query Responsibility Segregation (CQRS):

    • Proposes separating read and write operations into different models, allowing for performance enhancements and clearer business logic handling.
    • Describes a system with command handlers for processing business logic and separate read databases optimized for querying.

  • Event Sourcing:

    • Suggests moving from a state-based model to an event-based approach that captures changes as discrete events, enabling the reconstruction of past states and providing insights.
    • Illustrates Event Sourcing as a way to maintain historical records, resembling the function of a bank statement detailing all transactions.

  • Implementation Structure:

    • Highlights a design utilizing message buses for decoupling command handling from the application while ensuring events reflect changes accurately.
    • Discusses the storage of events in a dedicated database (e.g., MongoDB) and the synchronizing of read databases for efficient querying.

  • Complexity and Trade-offs:

    • While Event Sourcing provides numerous advantages such as clarity and encapsulation, it may introduce complexity, especially in simpler applications where the overhead may outweigh benefits.
    • Addresses challenges in managing long-lived objects by suggesting periodic snapshots to handle extensive event histories efficiently.

Conclusion:

Delbono's talk encourages developers to consider Event-Driven Architectures as powerful alternatives to the limitations of ActiveRecord, stressing that understanding how to separate business operations from database operations is crucial for building scalable and maintainable Rails applications.

00:00:13.839 Good afternoon, everybody! Thank you for joining. Let's start by asking how many of you are familiar with these books.
00:00:21.320 Oh nice, it seems most of you are! This book was published in 2002, and when I was a young developer, it contained many interesting patterns for enterprise applications.
00:00:30.080 One of the most interesting patterns inside this book is the Domain Model pattern.
00:00:37.040 This pattern teaches us that we should model our applications using objects that contain both behavior and data.
00:00:42.280 Until then, my applications were filled with anemic entities—objects that contained only data but no behavior.
00:00:49.280 After reading this book, I started to model an application, trying to apply the patterns I understood from it.
00:01:03.239 I began a new large application where I tried to define my domain models: a Customer can have an Invoice, which in turn can have items. A customer also has an Address, which is tied to the City, Price, Contract, and so on.
00:01:14.200 After a few iterations, I had created a very large domain model where all the objects were interconnected. It seemed awesome to me because I could start from the City and navigate through the Address to the Customer and retrieve all the contracts signed in a particular city.
00:01:32.479 I felt like I could do a lot of things with this extensive graph of interconnected objects.
00:01:39.200 However, the first problems arose when I tried to persist this graph to a database. At that time, the application was written in C# and I had to use an ORM like Hibernate to persist these object models to the database, which was quite painful. The problem was that this domain model was a monolith, with all the application's concerns bundled together in the same piece of code, making it difficult to separate them out into different concerns.
00:02:16.680 Around 2003 or 2004, a pivotal book was published. Who here is familiar with it? It’s a difficult book to understand, as I read it three times to fully grasp all the concepts it contained. If you read it after the book on Enterprise Application Architecture, you will see how it explains where I went wrong in my past designs. This book clarifies how the Domain Model pattern should be correctly applied in real-world applications.
00:02:56.400 During this talk, I want to show you two different approaches for our Rails applications. My name is Emanuele Delbono, and I come from Italy. I work as a software developer at a small company called C Plastico. Although I’m not a full-time Ruby developer, I enjoy developing in Ruby for fun in the evenings.
00:03:35.360 Let’s kick things off with the Lasagna architecture—does everyone here know about this architecture pattern? Being Italian, I know how to cook lasagna very well!
00:03:46.760 A Rails application can be likened to a lasagna; it has the view on top. The View communicates with the Controller, which ultimately interacts with the Model to persist data to the database. What's wrong with this arrangement? Actually, nothing—if the application is relatively simple, this architecture works just fine. However, if the application grows larger, and the model portion expands with numerous model classes, persisting data to the database becomes complex. This is why we use an ORM—in the case of Rails, that ORM is ActiveRecord.
00:04:31.800 Martin Fowler describes the Active Record pattern as an object that wraps a row in a database table. The concept is simple; you have one row in a database table corresponding to one object in your application. But if the model changes, the database changes, leading to tight coupling between the database structure and the domain model. In many cases, we have one class tying to one table, which means that our applications often reflect our database structure in memory.
00:05:14.840 This coupling creates problems when it comes to queries, as we face issues such as the N+1 problem during lazy loading. Moreover, we violate the Single Responsibility Principle because our models mix concerns related to both data persistence and domain logic. Therefore, our models fulfill two roles. In our Rails applications, we tend to think only in terms of the database; when we scaffold or project our models, our focus is almost exclusively database-centric.
00:06:37.160 Another problematic aspect is that we use the same model for both GET and POST operations—reading and writing data. For instance, during a GET operation, we might fetch all products from the database to return them to the view, but during creation, we employ the same model to create a new product in the database. As one prominent architect mentioned, a single model cannot adequately serve purposes for reporting, searching, and transactional behavior, yet that's exactly what we attempt to do.
00:07:38.880 This leads to different constraints between reads and writes. Reads are generally simpler; we just need to query the database and present the data to the view—sometimes performing minor formatting or internationalization. In contrast, writes are more complex as they require validation, authorization, and business logic to persist data.
00:08:13.799 The perspective of our users is also essential; they don't care about the underlying database mechanics. When a user clicks a button, they are interested in activating a new contract or performing a business operation. Such actions might involve sending emails, creating documents, or initiating workflows. However, developers—especially in Rails applications—tend to remain entrenched in INSERTs, READs, and CRUD operations.
00:09:01.600 My first piece of advice is to stop thinking in CRUD. We need to shift our mindset away from inserts, reads, deletes, and updates in the database because we’re building applications for users that require business operations.
00:09:36.360 This brings me to the first architectural pattern I will present today: Command Query Responsibility Segregation (CQRS). The idea is to treat reading and writing as two distinct concerns, resulting in two separate stacks—one for reading and another for writing. A possible implementation is thus: we have a presentation layer, such as our controllers, that sends commands to the green area.
00:10:11.040 Each command, which could be creating a contract or adding an item to a basket, is processed by a Handler.
00:10:18.200 The complex business logic is handled here, and the data is written to the appropriate write database. This database is then denormalized into a read database by a normalizing layer that prepares the data to be read as efficiently as possible. The result is a query service—very lightweight—designed solely for querying data in an optimal manner.
00:10:50.800 Here, we can employ one table for each view, where each view in our web application corresponds to a table. This means that reading operations become straightforward; we can use a simple SELECT * query or specify the required fields, with no joins or subqueries involved. Simplifying our queries makes reading data easier.
00:11:26.520 While this first step is conceptually straightforward, its implementation can be complex, particularly since we need to denormalize data and ensure that the two databases remain synchronized to avoid dirty reads.
00:12:09.320 Next, we often encounter a situation where data is stored in tables where each row represents the state of an object. For example, a basket might include two items, each represented in a table. The state of our object is defined by this table.
00:12:56.480 However, this presents a challenge because we only see the present state, missing the history that led to this status. It’s similar to receiving a bank statement that only contains totals with no details regarding how we got there—losing sight of all transactions.
00:13:24.320 Using this analogy, we might miss how our state changed yesterday, last week, and so forth. If we employ the same approach in our applications, we can shift our thinking towards events. Each change made to our system by a user can be treated as an event.
00:14:00.680 If we persist the events, we can reconstruct the state starting from them. For example, if a user adds an item, removes another, and adds yet another, we will know that their current basket contains those specified items, along with the historical actions of addition and deletion.
00:14:31.680 From a marketing perspective, being able to see what a customer adds to their basket and later removes can provide useful insights—helping us to strategize offers or discounts for items they show interest in.
00:15:02.680 This kind of architecture is known as Event Sourcing. As Martin Fowler noted, it captures all changes to an application state as a sequence of events. Instead of storing the actual current state of our objects, we record only the events.
00:15:41.720 This opens opportunities for various solutions, including the ability to travel back in time. We can reconstruct the state of our basket from yesterday, two weeks ago, or even a year back. Furthermore, we can simulate new events that may happen in the future, essentially projecting different scenarios.
00:16:22.360 The general architecture of event sourcing looks somewhat like this. It resembles CQRS and is a bit more complex. In this implementation, the presentation layer sends commands to a Handler, utilizing a message bus (like RabbitMQ or Redis) to decouple the Rails application from the command handling operations. We can have multiple instances handling groups of events.
00:17:03.200 The Handler pulls information from the event store, reconstructs the current state by reapplying past events, and calls methods using the Domain Model pattern. Notably, every method call produces new events that reflect changes.
00:17:49.840 These events are captured by a normalizer, which prepares the data for reading, while the query service reads from the database to expose data to the user. We essentially have an event store and the actual database.
00:18:35.280 Benefits of this architecture include encapsulation. We can model our domain without having to worry about storage since we only store events. Our objects typically do not have accessor methods; they contain methods that perform operations based on state. As such, our Domain Model becomes fully encapsulated.
00:19:31.680 With this architecture, managing storage becomes simple. In the forthcoming examples, I will use MongoDB for storing events. The overall performance and scaling are enhanced because we can spin up more instances of our handlers to process more commands in a given timeframe.
00:20:38.480 Additionally, testing becomes more straightforward since the units we test are the Domain Model objects themselves. This allows us to easily verify that the expected events are raised correctly. We can collect a considerable amount of information in our event store for future use, even retaining data that is not actively utilized but may become relevant later.
00:21:39.440 Integrating with other services also becomes easier, as we can use events for integration rather than relying solely on the database. For example, when needing to communicate with an invoicing system, we can create a consumer that listens for relevant events and executes necessary operations.
00:22:06.400 While there are numerous advantages, some cons do exist. Establishing this full-fledged infrastructure can be quite complex, especially for simple applications consisting of a limited number of classes. The costs associated with setting up messaging systems, two distinct databases (one for querying and another for event storage), can outweigh the benefits in less complex applications.
00:23:06.640 Handling long-lived objects can also be tricky, as they need to be reconstructed from extensive event histories, which can become time-consuming and costly.
00:23:50.840 To mitigate this, we often take periodic snapshots to contain the state of long-lived objects, which allows us to rebuild state from a specific point while keeping processing times manageable. We can then rebuild the entire state tree from the snapshot.
00:24:17.440 Lastly, the lack of a traditional database means that we need specialized tools to manage events. Unlike a conventional database where we can easily perform updates through SQL queries, we must build our own scripts and methods for dealing with events in the context of our application.
00:25:03.840 I have attempted to build this infrastructure into a sample Ruby on Rails application. It has been in production for nearly a year, and I aim to convert an existing C-based application into this architecture using Ruby. I believe Ruby, being an expressive and compact language, allows for simpler development than static languages like C or Java.
00:25:51.440 The mini application I created focuses on basket management, using Redis for communication, SQL for querying, MongoDB for event storage, and SQLite for the read database. The design follows Domain-Driven Design principles, using plain Ruby objects without infrastructure overheads.
00:26:36.880 As an example, when the controller receives a POST request to add an item to the basket, it simply sends a command of type AddToBasket with the correct parameters.
00:27:07.440 The Handler retrieves the current state of objects from the event store and applies the necessary changes. The command processing results in raising an event that informs our system of a change in the basket's state.
00:27:39.440 Once the basket processes the add item event, it collects uncommitted events before persisting them in the event store and notifying interested parties through the publishing of new events.
00:28:24.760 The event store is essentially a MongoDB list containing details of all the events related to the basket's state change, along with their corresponding metadata.
00:29:06.720 The normalizer listens for those events, updating the read database (SQLite) to ensure the data is readily accessible for displaying to the user.
00:29:55.480 The process we just discussed encompasses commands, event storage, handling commands, feeding into a normalizer, and finally presenting those states back to users. It might seem lengthy, but much of this is infrastructure code, which simplifies the addition of features as the infrastructure stabilizes.
00:30:42.080 Lastly, transitioning from thinking in CRUD to focusing on business operations is key. Users do not care about your database; we must understand the differences between reads and writes and separate the pertinent operations.
00:31:18.720 Domain models should be constructed with plain old Ruby objects, without the complexities associated with ActiveRecord or other base classes. CQRS and Event Sourcing prove valuable in complex enterprise applications, and Ruby offers expressive constructs that can enhance development efficiency.
00:32:12.640 You can find the code for the simple application I developed online. I welcome your feedback to improve the quality of the code. Feel free to approach me with questions or suggestions as we can explore the code together.
00:33:15.440 Thank you!
00:34:08.760 Questions? Hi! I find this architecture quite interesting, but I would like to know how you deal with atomicity and data consistency.
00:34:14.200 One thing I didn’t mention is that our system is eventually consistent. When a user clicks 'add to basket', it takes some time for the action to reflect accurately in the state of the basket.
00:34:27.040 The atomicity of operations is guaranteed by the Handler and the aggregate models, thus ensuring that actions are performed correctly.
00:35:00.080 If errors occur, we manage them by notifying users at the top of the interface that an operation failed.
00:35:31.040 The important part is that we never rewrite history; once an event has been written, it remains as part of the historical record of actions.
00:36:19.680 We can modify the event store when necessary, rerun all relevant events, and ultimately rebuild the state.
00:37:01.920 As we use a denormalized database design, data duplication is common. Still, it's essential to maintain the truth of our application's state through events.
00:37:45.120 I encourage open conversation—please feel free to ask about any other topics or clarifications.
00:38:19.800 Thank you once again!
Emanuele Delbono
@emadb
Edit
Explore all talks recorded at wroclove.rb 2014
+13