00:00:22.140
First of all, welcome and thank you for coming. My name is Guillermo Aguirre, and this is my first time attending and speaking at the conference. I might be a little nervous.
00:00:34.260
Thank you. Today, we'll be talking about my journey.
00:00:39.420
I will share my experiences while applying microservices patterns to a modular monolith. It will be more about my journey.
00:00:46.800
Here is my face, and a little helper to pronounce my name, hopefully that helps you.
00:00:53.820
These are my social media accounts; feel free to follow me. Probably the most interesting one is my GitHub. I still use LinkedIn, so you can connect with me there as well.
00:01:04.979
I was born and raised in Ontario, Uruguay, South America, and I've been a Rails developer for almost six years now.
00:01:10.260
I work at Rootstrap, a Latin American company with headquarters in Argentina, Colombia, and Uruguay.
00:01:16.439
We provide services ranging from staff augmentation to building products from scratch. Thank you very much to them for bringing me here.
00:01:29.280
To give you a brief overview of the agenda, we will start with my journey—basically, my experience and how this talk came to be. I'll discuss some of the things I've learned and where I am currently, in the project I am working on.
00:01:40.380
Next, we will introduce the domains we will follow throughout the presentation, which will set the stage for the demo. Then, I’ll introduce the concept of patterns; you probably already know about them, but if you don't, I will explain what they are, what they seek to solve, and how they apply to the example app.
00:01:58.380
I will conclude with a final demo to show that everything works well. My journey, as I mentioned, began six years ago when I started learning Rails and working with simple projects, simple monoliths, and MVPs.
00:02:18.540
Then, I moved on to larger projects, which came with different challenges and constraints. I hope to provide a good example of what I faced with these larger monoliths, and hopefully, you can relate.
00:02:39.000
After that, I transitioned into a microservices project. I've been bouncing between different architectures and have learned a lot through these experiences. Currently, I'm working on a modular monolith project.
00:03:05.640
As I mentioned, everything I initially worked with was a monolith. They are good and widely used; there is nothing wrong with them if they meet the needs of the product.
00:03:17.819
Monoliths provide a range of functionalities and concepts that are very familiar to smaller apps. However, as the product grows, so does the app, and consequently, the teams. We face several issues with scaling.
00:03:35.159
As I mentioned earlier, I started with simple MVP projects and later worked on larger models. My first experience was quite frustrating. Please raise your hand if you've encountered monolithic code bloat.
00:03:54.299
If you are unfamiliar with larger monoliths, it is common to see a user model bloated with 2000 lines of code. Larger monoliths tend to have more dependencies and tightly coupled code, leading to longer deployment times.
00:04:15.720
Sometimes, it would take 30 to 40 minutes just to deploy a simple change, and due to tight coupling, maintaining and extending such code becomes significantly more difficult.
00:04:41.400
So that was more or less my experience with monoliths, and it informed what I learned from that project. After that, I moved to microservices, believing that it would be a change of scenery where everything would be great.
00:04:59.900
In smaller cohesive teams, every member is familiar with the service they work on. This setup allows us to ship changes faster without worrying about other components.
00:05:06.979
However, the microservices project I worked on didn't exhibit any of these benefits. It was a stereotypical bad microservices project that had lots of data inconsistencies. The services didn't communicate well with one another, making issues in production quite common.
00:05:32.640
This resulted in our team frequently having to hotfix issues, which, as we all know, is not an ideal situation.
00:05:53.040
Microservices also come with their own disadvantages, especially in terms of infrastructure complexity. We moved away from the big monolith, but we transferred all that complexity to how everything communicates in deployment.
00:06:06.779
If we don't manage this step correctly, the architecture can become messy with many services interacting inconsistently. After that challenging microservices project, I am currently working on a project that is a monolith with multiple databases.
00:06:21.420
We are actively trying to modularize and migrate away from the problems associated with larger monoliths. I first learned about modularization by reading an article from Shopify a couple of years ago.
00:06:41.400
This article provided great examples of how to tackle the process and emphasized that you are not alone in dealing with larger monoliths. The article contained an image that I deeply related to, especially since I spent most of my career at that moment in the lower half, stuck in monolithic architecture.
00:07:02.940
Now, as I try to modularize the project, I am aiming to reach the upper half of the diagram. The central question is: how can we achieve a good modular monolith, avoiding the problems associated with traditional monoliths?
00:07:29.580
In the same vein, how can we ensure that we prevent the pitfalls of distributed systems while still achieving a certain degree of intrusion and modularization?
00:07:41.759
I realized that I was not the first person to have such an idea. It prompted me to think about cross-pollination between architectures. Why can't we borrow the best practices from microservices for use in our modular monolith?
00:07:54.539
It is likely that similar issues will arise in a modular monolith, so reusing our knowledge in these systems is vital.
00:08:04.560
That's the backstory on how this idea came to be. I couldn’t simply test everything in production, so I created a simplified example to mimic the scenarios I encountered in my project.
00:08:17.639
This example will be the focal point of today's presentation. The main concept revolves around a simple app that allows members to schedule meetings.
00:08:30.660
We have two domains: 'meetings,' which is focused on meeting management, and 'user access,' which handles user registrations, policies, roles, and so forth.
00:08:49.140
To keep things simple for this presentation, we'll focus only on user registration and member creation.
00:09:01.620
You might think that this is a simple example, and you’d be correct. We could diagram it quite easily, showing a simple monolith with a member model depending on two other models and a single database.
00:09:32.640
However, this layout doesn’t serve the purpose I aimed to achieve in mimicking a modular structure. So, let's take a step further and introduce one database per domain, resulting in separate 'meetings' and 'user access' databases.
00:09:51.180
Leaving behind one database per domain may seem beneficial, but comes with its own set of challenges. When using one relational database, we benefit from transactions ensured by the four ACID properties.
00:10:04.440
Atomicity, in particular, ensures that everything is treated as a unit, meaning that if a transaction includes multiple steps and fails at any point, it will rollback all changes, resulting in a consistent state. However, once we separate the databases, we lose this functionality of rolling back multiple changes as one atomic unit.
00:10:51.420
Thus, we must find ways to achieve atomic behavior across multiple databases. You may wonder if we need to handle this change and mimic the expected behavior required for successful transactions.
00:11:00.000
In my experience with the microservices project, the absence of proper transaction management led to consistency issues. Data discrepancies made it extremely difficult to debug problems.
00:11:20.220
A user could even receive an incorrect billing amount, forcing manual fixes in production—surely not ideal. Important flows could break if we expect data to be present in the database that is not there.
00:12:03.060
In my opinion, it is essential to handle distributed transactions and data consistency in such projects. I will focus on specific solutions for ensuring data consistency during the user registration process.
00:12:11.700
For this discussion, we will utilize established patterns: proven solutions for common problems. We will specifically look at microservices patterns.
00:12:36.360
I highly recommend reading 'Microservices Patterns' by Chris Richardson. I will borrow some definitions of patterns from this book.
00:12:53.880
Most examples in the book are in Java, which posed challenges as I prepared this experiment. The examples I needed were usually Java, JavaScript, Go, or other performant languages, which meant I had to adapt them for Ruby.
00:13:15.600
The patterns we will focus on include one database per service or domain, using a Saga to handle distributed transactions, and implementing a transactional outbox to atomically emit events.
00:13:37.560
What do these patterns entail? One database per service or domain isolates data structures between domains or services, which can be implemented using private tables, schemas, or actual databases.
00:14:02.700
A Saga manages distributed transactions, providing eventual data consistency across systems. Unlike the synchronous two-phase commit method for distributed transactions, a Saga is asynchronous.
00:14:41.640
Think of it like a flow of local services each having their local transactions. When they make a change, they emit an event, which gets processed by the following service in the chain. If any local transaction goes wrong, we have rollback mechanisms that allow us to revert changes.
00:15:14.520
However, emitting events presents its own challenges—what if the message bus is down and we lose critical events? We need a way to ensure updates to the state and publish events consistently.
00:15:36.360
We will employ the transactional outbox pattern to do this. The concept is to persist events while benefiting from the local transactions used in the Saga.
00:16:04.260
We can use local transactions to maintain a consistent state by ensuring that when a change happens, the corresponding event is also emitted.
00:16:20.460
More specifically, we'll insert a new entry in the user registration table while also adding a corresponding record in the outbox table related to that entry.
00:16:47.640
If the user registration fails for any reason, we can rollback everything and maintain a consistent state.
00:17:04.440
Next, we will have a message relay—essentially a worker that pulls information from the outbox table or reads the transaction logs, publishing the events to the message broker.
00:17:23.640
Now our application architecture will resemble something more complex, similar to the earlier diagram. Here, domains communicate indirectly via a message bus, identifying the same user across different databases with a shared identifier.
00:17:43.860
We now have our databases, domains, and systems set up. Let's follow the user registration flow as a practical example.
00:17:59.400
A user registers in our app, inputs their data, and gets sent a confirmation email. This email confirmation triggers the entire user registration Saga.
00:18:17.500
When the confirmation is received, we update the user registration's status to 'confirmed' and create an outbox record representing this event, which will be processed by the consumer.
00:18:49.440
The two domains will listen to this event: the user access domain will create a corresponding user entry, while the meetings domain will attempt to create the member.
00:19:03.960
In the ideal scenario, we want everything to be successful, creating a confirmed user registration record, a user record, and a member record.
00:19:35.760
Once the member is successfully created, we will emit events indicating the success. Finally, we'll move to the database to see what happens.
00:19:54.540
If everything worked perfectly, we should see records persisting, confirming a successful user registration and member creation.
00:20:11.400
Now let's see what happens if a failure occurs during the member creation step. I will utilize a feature flag to illustrate this failure.
00:20:51.180
With the feature flag disabled, the member creation will fail, and we will need to follow the rollback sequence.
00:21:11.100
I will walk through the process once again, sending a confirmation email and triggering the user registration to confirm.
00:21:30.480
When the confirmation is processed, both domains will respond: the user access domain creates the user, and the meetings domain tries to create the member. However, the member creation will fail, resulting in a rollback.
00:22:09.060
After encountering this failure, the user access domain will consume the event, recognizing the need to roll back changes to maintain consistency across the system.
00:22:43.560
Thus, the user registration will be brought back to a state of waiting confirmation, and we will also destroy the user record that was created before the rollback.
00:23:30.240
The outcome in the database should indicate that the user registration record remains in a state of waiting for confirmation, and no member record should exist due to the rollback process.
00:24:10.740
In summary, we aimed to create a modularized monolith while implementing transactional outbox patterns to ensure consistency. By pooling together distinct principles from various architectures, we can enhance how we manage our microservices.
00:24:49.740
As for the code implementation, I utilized Rails' multi-database support while using Kafka as the message relay, which connects to Kafka Connect for moving data among applications.
00:25:08.760
Additionally, I’ll showcase the registration process within the application, walking through the flow of user input and confirming that the events are being logged correctly.
00:25:26.880
During the chapter, we will see an email being sent, and we will observe the user creation event persisted in the outbox.
00:25:42.600
Currently, we should see the user registration waiting for confirmation status while we confirm the email.
00:26:03.360
As the registration is successfully confirmed, we will witness various processes operating, including events being captured across multiple domains.
00:26:25.560
Following this, we should check the database to track the outcomes of our events and confirm the records' status.
00:26:57.600
The events we registered should reflect in both the user registration and user access tables, with references displaying the same shared identifier.
00:27:29.880
In contrast, if everything operates as expected, we can safely submit member creation tasks and maintain flowing events through the outbox, maintaining the state of consistency.
00:28:10.740
After running through a successful user registration, we've experienced what should happen when all components align.
00:28:40.440
Now, if we enabled the feature flag to deliberately induce a failure within the system, we can observe the rollback sequence in action.
00:29:20.520
This should show the rollback leaving the database in a consistent state with the registration record set back to its waiting state.
00:29:41.760
In closing, this presentation aimed to tackle the complexities involved in user registration flows by leveraging modular architecture principles.
00:30:01.980
I hope this discussion offered valuable insights into how to resolve distributed transactions within modular monoliths.
00:30:24.540
If you're interested in exploring the code further, I’ll have a QR code on the next slide to access the open-source project repository.
00:30:45.180
Thank you for attending my talk. I hope you found it enlightening, and I'm happy to answer any questions you may have.
00:31:00.600
I look forward to engaging with anyone who has questions or feedback. Let’s create an environment of collaboration and continued learning.
00:31:29.640
Thank you once again, and I hope to inspire you to explore possible paths that can lead to efficient solutions.