Ruby Video
Vladimir Dementyev
Monoliths Between Microservices
Vladimir Dementyev
@palkan
#ruby-on-rails
#monolith
#microservices
#rails-engines
#software-architecture
#component-based-architecture
#dependency-management
Summarized using AI

Monoliths Between Microservices

by Vladimir Dementyev

In the talk "Monoliths Between Microservices" presented at RailsConf 2020, Vladimir Dementyev explores the challenges and strategies associated with maintaining a Rails application as it grows, particularly in managing the transition from monolithic to microservices architectures.

Key Points Discussed:

  • The Nature of Rails Applications: Rails applications are inherently monolithic, including diverse functionalities like APIs, background jobs, and utilities. This can lead to inefficiencies as the codebase expands.
  • The Misconception of Microservices: A typical approach suggests breaking an application into microservices, which often results in distributed monoliths—still cumbersome and difficult to manage, negating the advantages sought.
  • Separation of Concerns: Emphasizing the importance of logically separating an application into independent components, this separation eases maintenance and improves developer experience.
  • Component-Based Monoliths: Vladimir provides examples of successful modular architectures, including Shopify's approach and others like umbrella projects in Elixir. He highlights that while some models are not directly applicable to Rails, their concepts can be adapted.
  • Utilizing Rails Engines: Engines in Rails allow for the segmentation of code into manageable pieces—much like other modular frameworks. Each engine acts as a mini-application, providing a path to a more organized codebase.
  • Case Study - Common and Cocaine Projects: Vladimir shares insights from his work on these projects, explaining how their architecture evolved into a component-based system using Rails engines. He discusses the initial use of namespaces and the progression towards more isolated and portable code.
  • Challenges and Solutions: Throughout the presentation, he identifies common pitfalls—like dependency management and database migration—and offers intelligent solutions, such as utilizing a shared Gemfile structure to maintain dependencies synchronously across engines.
  • Testing and CI Complexity: The talk also touches on the complexities of testing in a component-based architecture, suggesting methods for isolating tests by engine and managing them effectively within a CI environment.
  • Communication Between Components: To facilitate interaction between engines, Vladimir introduces the idea of event-driven architecture, highlighting their use of publish/subscribe mechanisms for smooth operation between components.

Conclusion and Takeaways:

  • Engines provide a solid foundation for building component-based applications in Rails, allowing for better organization and potential future scalability to microservices.
  • Migrating from monolithic applications requires thoughtful consideration of architecture, dependencies, and isolation techniques.
  • While the transition presents challenges, with careful planning and the use of engines, developers can create robust applications that are easier to maintain and evolve over time.

Vladimir's talk serves as a guide for developers seeking to optimize their Rails applications by understanding and implementing component-based architectures.

00:00:09.030 Hello everyone! For RailsConf 2020, my name is Vladimir, and I'm going to talk about what lies between monoliths and microservices.
00:00:16.139 Rails applications tend to turn into giant monoliths, and keeping the codebase maintainable usually requires architectural changes.
00:00:21.540 Microservices? A distributed monolith is still a monolith. So, what about breaking the code into pieces and rebuilding a monolithic puzzle out of them?
00:00:26.550 In Rails, we have the right tool for the job: engines. With engines, you can break your application into components—the same way Rails combines all its parts, which are engines, too.
00:00:39.750 However, this path is full of snares and pitfalls. Let me be your guide on this journey and share the story of monolith engine-ification.
00:00:46.140 A typical Rails application is often doomed to become a monolithic monster. The question is: how do we prevent this?
00:00:51.840 We need to keep our applications maintainable as they grow, and make developers happy to work with them. One may suggest, 'Why not just split it into microservices?' But that hype train is starting to derail because we know that microservices usually end up becoming a distributed monolith, which is just as daunting to manage.
00:01:03.150 The issue with both monoliths and distributed monoliths lies in forgetting the core principle of separation. We need to split the application into logically independent units, components that are connected together, almost like a kind of bus.
00:01:14.460 The idea is to maintain loose coupling, which means that if one component is broken, it doesn’t affect the others. Refactoring becomes much easier this way, unlike with monoliths, where if there's a bug, fixing it can be slow and painful. We need a system where we can guarantee that a bug won't reappear in the next version.
00:01:30.950 Today, we are going to discuss component-based monoliths or modular monoliths. A particular example of such a model is Shopify.
00:01:38.540 Shopify has a great article describing their architecture, showcasing components with a gem-like structure inside their application.
00:01:47.040 However, the problem with Shopify's approach is that we cannot directly apply it to our applications because their solution hasn't been open-sourced yet. Yet, we can still borrow the idea, and that's what we'll do during this talk.
00:02:01.170 Let's also look at another Ruby framework that allows you to use component-based architecture. They call themselves Molly Thirst, which means you can start with a single component but add other components later as your application grows.
00:02:13.870 Another example is umbrella projects in Elixir, which actually date back to Perl, but they share the same idea: multiple logical applications working as a single system.
00:02:22.530 In Rails, we also have this concept, known as engines, and that's what we're going to discuss today.
00:02:36.640 Let me take a quick detour and talk about myself a little. How did I end up here, speaking about component-based architecture in Rails?
00:02:42.640 You can find me on GitHub under the name Falcon. I'm working at a company called Evil Martians, which has been off in Ramallah for over ten years. We have bases around the world with agents scattered across various continents.
00:02:54.060 You've probably heard of open-source projects from the Ruby world, development tools, and our beautiful blog with a series of technical posts.
00:03:02.490 Today, however, I'm not talking about that; I'm sharing the darker side. A large part of my everyday life is working on commercial projects, and about half of all our clients use Rails as their core technology.
00:03:13.010 Today, I want to share one particular case study: a story involving a project called Common and Cocaine. This is where we started actively using component-based architecture.
00:03:22.320 Let’s take a quick overview of what these services are and why we decided to use this architecture. Common assault started as a long-term Cleveland rental service working in major cities in the U.S. and Canada.
00:03:40.880 Before we joined their team, they only had a management application—managing properties, losses, billing staff, and so on.
00:03:47.540 Our task was to build a community portal for users, and one of the requirements was that we needed to do this within the same Rails app.
00:03:56.900 They didn’t want to go with microservices right away, and that was a wise decision, considering that most of the logic was implemented in this Rails monolith.
00:04:04.680 This was when the model started to risk becoming a monster if we didn't handle it correctly.
00:04:17.080 So, what exactly is a community portal? It’s a mix of something like Meetup.com, a chat platform, and other smaller features.
00:04:26.230 The components are almost independent of each other, even at a logical level, as they have well-defined boundaries.
00:04:35.630 That’s why, from day one, we started using namespaces to isolate the lexical and structural levels of all the code related to a particular feature or domain.
00:04:45.240 Even though we don’t have namespaces in Ruby, we can use modules to contain all the code. This is a very fast way of creating a pseudo-component-based architecture.
00:05:01.969 However, we cannot guarantee that code from one namespace doesn't access the code from a different namespace, which it shouldn’t.
00:05:10.440 This was pretty much how we established the MVP, the minimum viable product. It was an initial version, but then the requirements evolved.
00:05:23.649 It became clear that we were going to build some spin-off projects and other community portals with very similar functionality.
00:05:39.650 We thought we could reuse most of the feature code, but we needed a way to ensure isolation and portability.
00:05:49.530 That’s when we began to use engines and project gems. This became the final architecture for the project, which I will explain in more detail later.
00:05:56.900 One key point that motivated us to make this decision was the modular model of architecture. This is a strong starting point to learn about component-based architecture in Rails and identify potential initial problems.
00:06:06.800 It turned out that there were many more pitfalls, and that is what I would like to address today.
00:06:15.700 We’ll dive into the technical details of how to make engines work well in a component-based system.
00:06:24.270 As I mentioned, we used engines and also local gems, which we call ‘soul gems’—exclusive to this application, similar to a mono-repo without public publishing.
00:06:34.890 We refer to both concepts as engines, so let’s start with an engine. What is a Rails engine?
00:06:44.160 Simply put, it’s like any other gem that you can generate by running 'rails plugin new'.
00:06:50.500 However, let’s go a bit deeper. We need to understand what an engine is before diving into the details.
00:06:57.240 First, an engine is indeed a gem. You can build it and publish it like any other gem.
00:07:05.600 It has a gem specification that lists both its required dependencies and the lib folder that contains all the code.
00:07:12.850 What differentiates an engine from a typical gem is that it includes some Rails-specific entities.
00:07:20.400 It has your app folder with controllers, models, and other standard components.
00:07:28.680 The idea is that when you add the engine to a Rails application, its content is accessible through the outer loading mechanism.
00:07:37.440 You can just use these constants without having to require them explicitly.
00:07:45.450 Engines also come with their own test suites, allowing us to test them in isolation.
00:07:52.900 Additionally, they have their own class in the namespace that acts as a glue between the Rails application and the engine’s functionality.
00:08:02.160 You can add initializers, configurations, and other elements similar to those in a Rails application.
00:08:11.530 Lastly, you can add routes to the engine and use those routes in your main application.
00:08:19.750 Have you ever seen engines in real life? Of course, you have! These are some of the most popular engines that are part of Rails applications.
00:08:26.150 Rails indeed offers a component-based architecture with many full-featured engines, in addition to Rails itself.
00:08:35.120 One popular engine outside the Rails world is Devise. Many of us use Devise or its extensions to manage user authentication.
00:08:43.430 Thus, using engines can lead to happiness and help in building component-based applications.
00:08:50.800 However, the process of engine-ification in applications is not always straightforward.
00:08:58.200 Realizing what an engine is is just the beginning; it hides many complexities and problems that you are likely to face.
00:09:07.570 I won’t cover all of these issues today, but I will highlight some of the more interesting ones.
00:09:18.320 Let’s begin discussing the architecture of the apple head. Here are a few pictures of what the final result was.
00:09:25.740 We had an 'engines' folder with a few engine gems, while another folder consisted only of mount directives.
00:09:34.790 Our application.js file contained just these mount directives.
00:09:42.850 In addition, our gem files contained definitions pointing to our local engines, utilizing half of the gem specification.
00:09:52.370 The application diagram, as a components diagram, could appear as a unified structure, representing our approach.
00:10:03.190 The components implementing the business logic share similar naming conventions, often using a verb with a suffix.
00:10:10.920 We also employed a core engine, which included basic models like users and generic entities.
00:10:20.760 Another layer utilized this core engine as a dependency and included other layers, which also represented the engines.
00:10:29.560 We had two utility engines for search proxy and common events, which will be discussed later.
00:10:37.800 Finally, we formed one umbrella engine which integrated all the implementations.
00:10:47.590 Let’s begin investigating the journey from here. Our first stop is dependency management.
00:10:55.310 This is essentially the first problem you will encounter when using engines. There are many questions that arise when you start to extract functionality into isolated gems.
00:11:03.740 The first issue occurs when you have non-Rubygems dependencies. Each gem may need to load from GitHub, local sources, or other repositories.
00:11:11.640 You cannot specify these dependencies in the gem file, so you end up duplicating definitions in your root gem file.
00:11:18.030 This makes it difficult to maintain and synchronize dependencies, and it's easy to overlook some dependencies.
00:11:26.780 As a solution, we evolved a gem file structure that enhances functionality with bundler.
00:11:34.160 Every engine has a runtime gem file that includes non-Ruby gem dependencies required at runtime and another for development dependencies.
00:11:43.690 This way, we can ensure that our dependencies are synchronized cleanly.
00:11:51.850 Another concern is sharing common dependencies between engines without duplication.
00:12:01.430 The approach is similar: we maintain shared gem files for synchronizing Rails versions across engines.
00:12:08.630 In the root gem file, we reference the engine gem files and maintain them consistently.
00:12:17.090 But what about lock files? Should the versions in the root application and engines be the same?
00:12:25.990 Initially, we didn’t care about consistency, but we kept encountering inconsistencies.
00:12:33.540 We realized that most were either tied to development dependencies or batch version differences.
00:12:41.780 We started thinking about a better way to maintain synchronization, resulting in a unified lock file.
00:12:50.980 Now, instead of having separate lock files for engines, we use a root-level lock file, encompassing all engines.
00:12:59.880 This presents its limitations and potential complications, particularly in managing development and test dependencies.
00:13:08.770 We didn’t want development test dependencies to load in production.
00:13:17.530 Moreover, while testing engines, we often load everything from all applications, which is not ideal.
00:13:25.420 We resolved this using a helper method in the gem file, called 'component,' which serves two portfolios.
00:13:34.990 First, it adds a gem to the root gem file; the second part allows adding development dependencies under a named group.
00:13:42.890 This way, we ensure that the engine’s dependencies don’t pollute the main application.
00:13:50.360 Next, let’s discuss database management. What makes databases interesting within engines?
00:14:01.260 Well, we need a way to utilize migrations efficiently within the main app and how to handle seeds.
00:14:08.650 The standard way with migrations usually involves copying them into the main app's migrations directory.
00:14:18.260 However, in a component-based architecture, we would rather avoid the extraction and instead keep the codebase together.
00:14:27.240 We prefer to make the main app recognize engine migrations by tweaking two parameters in the Rails migration system.
00:14:35.480 Essentially, we need to point the main app to look for migrations in both its own directory and the engines' directories.
00:14:43.560 This allows us to conduct seamless migrations without copying any files.
00:14:56.740 The same approach can be applied for seeds, as engines already come with 'load_seed' methods that can be utilized.
00:15:03.360 This allows us to run the engine’s seeds within the main application effectively.
00:15:08.770 Now, let's address testing: How do we ensure engines are tested in isolation?
00:15:17.840 To do this, we rely on a tool called 'combustion,' which streamlines testing engines within their gem.
00:15:26.660 Combustion generates a minimal Rails app, enabling tests without unnecessary components.
00:15:36.700 Another relevant question is how to run engine tests on CI.
00:15:46.470 This architecture can complicate CI configuration, but on the other hand, it allows us to skip tests for unchanged engines.
00:15:54.960 Engines with unchanged code or dependencies can be safely skipped during testing, saving time.
00:16:02.640 To implement this, we developed a simple script that identifies dirty engines or changed dependencies.
00:16:12.490 The script builds an inverted index and checks for modifications compared to the previous state of the engines.
00:16:22.280 When integrating this into CI, it executes the script first; if it returns zero, we run the tests.
00:16:32.030 If it returns one, we skip the tests.
00:16:41.850 Next, we face the question of accessing entities across engines or the main app.
00:16:48.230 How do we deal with different entities across engines?
00:16:53.400 We came up with an idea called base and behavior convention that allows for flexible entity handling.
00:17:01.260 For example, an application controller in one engine can inherit from a configurable base controller defined outside.
00:17:08.860 This base controller can specify required behaviors, offering flexibility in your engine.
00:17:17.800 In practical terms, this translates to a custom application controller in the main app implementing additional behavior.
00:17:25.490 Another common scenario is extending models from other engines using concerns.
00:17:32.960 We utilize ActiveSupport's loading hooks to ensure that when a model is accessed, any registered concerns are included.
00:17:39.640 Now, let’s discuss communication between engines.
00:17:47.160 How do we manage notifications when a user registers in one engine that affects another?
00:17:54.020 In this scenario, we want newly registered users to join a default chat within the chat engine.
00:18:01.300 Here lies the crux: the registration process occurs in the connect by engine, while the chat logic belongs in another.
00:18:08.300 Our solution was to use an event-based system and a publish/subscribe mechanism.
00:18:15.740 Though many tools exist for these purposes, we chose Rails as our base framework, leveraging its features.
00:18:22.950 This integrates seamlessly with ActiveRecord, allowing synchronous and asynchronous event consumption.
00:18:31.770 The result is improved event-handling capabilities that are encapsulated within custom gems.
00:18:38.810 Moving on, let’s address the critical topic of keeping the main app intact while embracing microservices.
00:18:47.310 As we migrated towards a component-based system using engines, it raises the question of what should remain in the main application.
00:18:56.870 It’s impractical to transform the main app into a ‘zero code’ structure even when components are independent.
00:19:04.950 You still need configurations for the database, certain generative processes, and maybe other library integrations.
00:19:12.380 In ideal scenarios, you might function from a zero-code perspective, especially for applications built from scratch.
00:19:18.320 Still, common applications will tend to retain essential logic directly within the main application.
00:19:27.620 What definitely resides in the primary app includes feature/system tests. You should always write integration tests for the system as a whole.
00:19:36.240 Instrumentation, error handling, and basic configuration for the Rails framework must remain.
00:19:43.640 Lastly, let’s discuss gems and local gems, which we widely use to tackle different application problems.
00:19:52.990 By organizing code unrelated to business logic into reusable components, we enable efficient functionality across applications.
00:20:02.350 This becomes especially valuable with our event store gem, which initially served us internally before evolving into an open-sourced solution.
00:20:12.060 The benefits of local gems include not only better structure but also isolated tests that can significantly reduce CI build times.
00:20:20.920 That’s a quick overview of engines and gem self-rejection. If you noticed, we’ve only covered half of the map, but I hope I provided valuable insights.
00:20:29.780 In conclusion, using engines can prevent monoliths from becoming monsters. They come with plenty of benefits.
00:20:36.920 One significant benefit is the option to extract engines into microservices in the future.
00:20:44.930 It's essential to acknowledge that third-party gems often lack sufficient support for these practices.
00:20:50.680 Here’s a link to the resources, scripts, and other useful materials from this talk. Feel free to use them.
00:20:58.900 Thank you, and remember, don't let your monoliths turn into monsters!
Edit
Explore all talks recorded at RailsConf 2020 CE
+22