00:00:09.030
Hello everyone! For RailsConf 2020, my name is Vladimir, and I'm going to talk about what lies between monoliths and microservices.
00:00:16.139
Rails applications tend to turn into giant monoliths, and keeping the codebase maintainable usually requires architectural changes.
00:00:21.540
Microservices? A distributed monolith is still a monolith. So, what about breaking the code into pieces and rebuilding a monolithic puzzle out of them?
00:00:26.550
In Rails, we have the right tool for the job: engines. With engines, you can break your application into components—the same way Rails combines all its parts, which are engines, too.
00:00:39.750
However, this path is full of snares and pitfalls. Let me be your guide on this journey and share the story of monolith engine-ification.
00:00:46.140
A typical Rails application is often doomed to become a monolithic monster. The question is: how do we prevent this?
00:00:51.840
We need to keep our applications maintainable as they grow, and make developers happy to work with them. One may suggest, 'Why not just split it into microservices?' But that hype train is starting to derail because we know that microservices usually end up becoming a distributed monolith, which is just as daunting to manage.
00:01:03.150
The issue with both monoliths and distributed monoliths lies in forgetting the core principle of separation. We need to split the application into logically independent units, components that are connected together, almost like a kind of bus.
00:01:14.460
The idea is to maintain loose coupling, which means that if one component is broken, it doesn’t affect the others. Refactoring becomes much easier this way, unlike with monoliths, where if there's a bug, fixing it can be slow and painful. We need a system where we can guarantee that a bug won't reappear in the next version.
00:01:30.950
Today, we are going to discuss component-based monoliths or modular monoliths. A particular example of such a model is Shopify.
00:01:38.540
Shopify has a great article describing their architecture, showcasing components with a gem-like structure inside their application.
00:01:47.040
However, the problem with Shopify's approach is that we cannot directly apply it to our applications because their solution hasn't been open-sourced yet. Yet, we can still borrow the idea, and that's what we'll do during this talk.
00:02:01.170
Let's also look at another Ruby framework that allows you to use component-based architecture. They call themselves Molly Thirst, which means you can start with a single component but add other components later as your application grows.
00:02:13.870
Another example is umbrella projects in Elixir, which actually date back to Perl, but they share the same idea: multiple logical applications working as a single system.
00:02:22.530
In Rails, we also have this concept, known as engines, and that's what we're going to discuss today.
00:02:36.640
Let me take a quick detour and talk about myself a little. How did I end up here, speaking about component-based architecture in Rails?
00:02:42.640
You can find me on GitHub under the name Falcon. I'm working at a company called Evil Martians, which has been off in Ramallah for over ten years. We have bases around the world with agents scattered across various continents.
00:02:54.060
You've probably heard of open-source projects from the Ruby world, development tools, and our beautiful blog with a series of technical posts.
00:03:02.490
Today, however, I'm not talking about that; I'm sharing the darker side. A large part of my everyday life is working on commercial projects, and about half of all our clients use Rails as their core technology.
00:03:13.010
Today, I want to share one particular case study: a story involving a project called Common and Cocaine. This is where we started actively using component-based architecture.
00:03:22.320
Let’s take a quick overview of what these services are and why we decided to use this architecture. Common assault started as a long-term Cleveland rental service working in major cities in the U.S. and Canada.
00:03:40.880
Before we joined their team, they only had a management application—managing properties, losses, billing staff, and so on.
00:03:47.540
Our task was to build a community portal for users, and one of the requirements was that we needed to do this within the same Rails app.
00:03:56.900
They didn’t want to go with microservices right away, and that was a wise decision, considering that most of the logic was implemented in this Rails monolith.
00:04:04.680
This was when the model started to risk becoming a monster if we didn't handle it correctly.
00:04:17.080
So, what exactly is a community portal? It’s a mix of something like Meetup.com, a chat platform, and other smaller features.
00:04:26.230
The components are almost independent of each other, even at a logical level, as they have well-defined boundaries.
00:04:35.630
That’s why, from day one, we started using namespaces to isolate the lexical and structural levels of all the code related to a particular feature or domain.
00:04:45.240
Even though we don’t have namespaces in Ruby, we can use modules to contain all the code. This is a very fast way of creating a pseudo-component-based architecture.
00:05:01.969
However, we cannot guarantee that code from one namespace doesn't access the code from a different namespace, which it shouldn’t.
00:05:10.440
This was pretty much how we established the MVP, the minimum viable product. It was an initial version, but then the requirements evolved.
00:05:23.649
It became clear that we were going to build some spin-off projects and other community portals with very similar functionality.
00:05:39.650
We thought we could reuse most of the feature code, but we needed a way to ensure isolation and portability.
00:05:49.530
That’s when we began to use engines and project gems. This became the final architecture for the project, which I will explain in more detail later.
00:05:56.900
One key point that motivated us to make this decision was the modular model of architecture. This is a strong starting point to learn about component-based architecture in Rails and identify potential initial problems.
00:06:06.800
It turned out that there were many more pitfalls, and that is what I would like to address today.
00:06:15.700
We’ll dive into the technical details of how to make engines work well in a component-based system.
00:06:24.270
As I mentioned, we used engines and also local gems, which we call ‘soul gems’—exclusive to this application, similar to a mono-repo without public publishing.
00:06:34.890
We refer to both concepts as engines, so let’s start with an engine. What is a Rails engine?
00:06:44.160
Simply put, it’s like any other gem that you can generate by running 'rails plugin new'.
00:06:50.500
However, let’s go a bit deeper. We need to understand what an engine is before diving into the details.
00:06:57.240
First, an engine is indeed a gem. You can build it and publish it like any other gem.
00:07:05.600
It has a gem specification that lists both its required dependencies and the lib folder that contains all the code.
00:07:12.850
What differentiates an engine from a typical gem is that it includes some Rails-specific entities.
00:07:20.400
It has your app folder with controllers, models, and other standard components.
00:07:28.680
The idea is that when you add the engine to a Rails application, its content is accessible through the outer loading mechanism.
00:07:37.440
You can just use these constants without having to require them explicitly.
00:07:45.450
Engines also come with their own test suites, allowing us to test them in isolation.
00:07:52.900
Additionally, they have their own class in the namespace that acts as a glue between the Rails application and the engine’s functionality.
00:08:02.160
You can add initializers, configurations, and other elements similar to those in a Rails application.
00:08:11.530
Lastly, you can add routes to the engine and use those routes in your main application.
00:08:19.750
Have you ever seen engines in real life? Of course, you have! These are some of the most popular engines that are part of Rails applications.
00:08:26.150
Rails indeed offers a component-based architecture with many full-featured engines, in addition to Rails itself.
00:08:35.120
One popular engine outside the Rails world is Devise. Many of us use Devise or its extensions to manage user authentication.
00:08:43.430
Thus, using engines can lead to happiness and help in building component-based applications.
00:08:50.800
However, the process of engine-ification in applications is not always straightforward.
00:08:58.200
Realizing what an engine is is just the beginning; it hides many complexities and problems that you are likely to face.
00:09:07.570
I won’t cover all of these issues today, but I will highlight some of the more interesting ones.
00:09:18.320
Let’s begin discussing the architecture of the apple head. Here are a few pictures of what the final result was.
00:09:25.740
We had an 'engines' folder with a few engine gems, while another folder consisted only of mount directives.
00:09:34.790
Our application.js file contained just these mount directives.
00:09:42.850
In addition, our gem files contained definitions pointing to our local engines, utilizing half of the gem specification.
00:09:52.370
The application diagram, as a components diagram, could appear as a unified structure, representing our approach.
00:10:03.190
The components implementing the business logic share similar naming conventions, often using a verb with a suffix.
00:10:10.920
We also employed a core engine, which included basic models like users and generic entities.
00:10:20.760
Another layer utilized this core engine as a dependency and included other layers, which also represented the engines.
00:10:29.560
We had two utility engines for search proxy and common events, which will be discussed later.
00:10:37.800
Finally, we formed one umbrella engine which integrated all the implementations.
00:10:47.590
Let’s begin investigating the journey from here. Our first stop is dependency management.
00:10:55.310
This is essentially the first problem you will encounter when using engines. There are many questions that arise when you start to extract functionality into isolated gems.
00:11:03.740
The first issue occurs when you have non-Rubygems dependencies. Each gem may need to load from GitHub, local sources, or other repositories.
00:11:11.640
You cannot specify these dependencies in the gem file, so you end up duplicating definitions in your root gem file.
00:11:18.030
This makes it difficult to maintain and synchronize dependencies, and it's easy to overlook some dependencies.
00:11:26.780
As a solution, we evolved a gem file structure that enhances functionality with bundler.
00:11:34.160
Every engine has a runtime gem file that includes non-Ruby gem dependencies required at runtime and another for development dependencies.
00:11:43.690
This way, we can ensure that our dependencies are synchronized cleanly.
00:11:51.850
Another concern is sharing common dependencies between engines without duplication.
00:12:01.430
The approach is similar: we maintain shared gem files for synchronizing Rails versions across engines.
00:12:08.630
In the root gem file, we reference the engine gem files and maintain them consistently.
00:12:17.090
But what about lock files? Should the versions in the root application and engines be the same?
00:12:25.990
Initially, we didn’t care about consistency, but we kept encountering inconsistencies.
00:12:33.540
We realized that most were either tied to development dependencies or batch version differences.
00:12:41.780
We started thinking about a better way to maintain synchronization, resulting in a unified lock file.
00:12:50.980
Now, instead of having separate lock files for engines, we use a root-level lock file, encompassing all engines.
00:12:59.880
This presents its limitations and potential complications, particularly in managing development and test dependencies.
00:13:08.770
We didn’t want development test dependencies to load in production.
00:13:17.530
Moreover, while testing engines, we often load everything from all applications, which is not ideal.
00:13:25.420
We resolved this using a helper method in the gem file, called 'component,' which serves two portfolios.
00:13:34.990
First, it adds a gem to the root gem file; the second part allows adding development dependencies under a named group.
00:13:42.890
This way, we ensure that the engine’s dependencies don’t pollute the main application.
00:13:50.360
Next, let’s discuss database management. What makes databases interesting within engines?
00:14:01.260
Well, we need a way to utilize migrations efficiently within the main app and how to handle seeds.
00:14:08.650
The standard way with migrations usually involves copying them into the main app's migrations directory.
00:14:18.260
However, in a component-based architecture, we would rather avoid the extraction and instead keep the codebase together.
00:14:27.240
We prefer to make the main app recognize engine migrations by tweaking two parameters in the Rails migration system.
00:14:35.480
Essentially, we need to point the main app to look for migrations in both its own directory and the engines' directories.
00:14:43.560
This allows us to conduct seamless migrations without copying any files.
00:14:56.740
The same approach can be applied for seeds, as engines already come with 'load_seed' methods that can be utilized.
00:15:03.360
This allows us to run the engine’s seeds within the main application effectively.
00:15:08.770
Now, let's address testing: How do we ensure engines are tested in isolation?
00:15:17.840
To do this, we rely on a tool called 'combustion,' which streamlines testing engines within their gem.
00:15:26.660
Combustion generates a minimal Rails app, enabling tests without unnecessary components.
00:15:36.700
Another relevant question is how to run engine tests on CI.
00:15:46.470
This architecture can complicate CI configuration, but on the other hand, it allows us to skip tests for unchanged engines.
00:15:54.960
Engines with unchanged code or dependencies can be safely skipped during testing, saving time.
00:16:02.640
To implement this, we developed a simple script that identifies dirty engines or changed dependencies.
00:16:12.490
The script builds an inverted index and checks for modifications compared to the previous state of the engines.
00:16:22.280
When integrating this into CI, it executes the script first; if it returns zero, we run the tests.
00:16:32.030
If it returns one, we skip the tests.
00:16:41.850
Next, we face the question of accessing entities across engines or the main app.
00:16:48.230
How do we deal with different entities across engines?
00:16:53.400
We came up with an idea called base and behavior convention that allows for flexible entity handling.
00:17:01.260
For example, an application controller in one engine can inherit from a configurable base controller defined outside.
00:17:08.860
This base controller can specify required behaviors, offering flexibility in your engine.
00:17:17.800
In practical terms, this translates to a custom application controller in the main app implementing additional behavior.
00:17:25.490
Another common scenario is extending models from other engines using concerns.
00:17:32.960
We utilize ActiveSupport's loading hooks to ensure that when a model is accessed, any registered concerns are included.
00:17:39.640
Now, let’s discuss communication between engines.
00:17:47.160
How do we manage notifications when a user registers in one engine that affects another?
00:17:54.020
In this scenario, we want newly registered users to join a default chat within the chat engine.
00:18:01.300
Here lies the crux: the registration process occurs in the connect by engine, while the chat logic belongs in another.
00:18:08.300
Our solution was to use an event-based system and a publish/subscribe mechanism.
00:18:15.740
Though many tools exist for these purposes, we chose Rails as our base framework, leveraging its features.
00:18:22.950
This integrates seamlessly with ActiveRecord, allowing synchronous and asynchronous event consumption.
00:18:31.770
The result is improved event-handling capabilities that are encapsulated within custom gems.
00:18:38.810
Moving on, let’s address the critical topic of keeping the main app intact while embracing microservices.
00:18:47.310
As we migrated towards a component-based system using engines, it raises the question of what should remain in the main application.
00:18:56.870
It’s impractical to transform the main app into a ‘zero code’ structure even when components are independent.
00:19:04.950
You still need configurations for the database, certain generative processes, and maybe other library integrations.
00:19:12.380
In ideal scenarios, you might function from a zero-code perspective, especially for applications built from scratch.
00:19:18.320
Still, common applications will tend to retain essential logic directly within the main application.
00:19:27.620
What definitely resides in the primary app includes feature/system tests. You should always write integration tests for the system as a whole.
00:19:36.240
Instrumentation, error handling, and basic configuration for the Rails framework must remain.
00:19:43.640
Lastly, let’s discuss gems and local gems, which we widely use to tackle different application problems.
00:19:52.990
By organizing code unrelated to business logic into reusable components, we enable efficient functionality across applications.
00:20:02.350
This becomes especially valuable with our event store gem, which initially served us internally before evolving into an open-sourced solution.
00:20:12.060
The benefits of local gems include not only better structure but also isolated tests that can significantly reduce CI build times.
00:20:20.920
That’s a quick overview of engines and gem self-rejection. If you noticed, we’ve only covered half of the map, but I hope I provided valuable insights.
00:20:29.780
In conclusion, using engines can prevent monoliths from becoming monsters. They come with plenty of benefits.
00:20:36.920
One significant benefit is the option to extract engines into microservices in the future.
00:20:44.930
It's essential to acknowledge that third-party gems often lack sufficient support for these practices.
00:20:50.680
Here’s a link to the resources, scripts, and other useful materials from this talk. Feel free to use them.
00:20:58.900
Thank you, and remember, don't let your monoliths turn into monsters!