Ruby Video | All You Need is Rails (Engines)

Julián Pinzón Eslava

All You Need is Rails (Engines) - Compartmentalising Your Monolith

Julián Pinzón Eslava

1 talk

#software-architecture

#microservices

All You Need is Rails (Engines) - Compartmentalising Your Monolith

by Julián Pinzón Eslava

In the talk titled "All You Need is Rails (Engines) - Compartmentalising Your Monolith" presented by Julián Pinzón Eslava at RubyConf AU 2023, the speaker explores the challenges faced by Ruby on Rails developers when dealing with large, complex applications known as monoliths. While conventional wisdom suggests moving towards microservices when applications become too unwieldy, Eslava presents an alternative solution: using Rails Engines to manage complexity without abandoning the monolithic architecture.

Key Points Discussed:

Overall, the talk challenges common beliefs about scaling Rails applications and offers a path towards a sustainable monolithic architecture that leverages the powerful features of Rails Engines.

00:00:00 You will learn a whole new perspective on Rails Engines, a much lighter and leaner way to use them that allows teams to break up their app into meaningful modules without losing the beauty of Rails conventions. Sprinkle in the Packwerk gem by Shopify for the perfect combo to scale your domain and your team.

00:00:12 Our next speaker, Julián Pinzón Eslava, is a musician, audio engineer, self-taught software engineer, and a Ruby lover. He's heavily into pairing when he needs to learn a new codebase or a new language, like Kotlin. After that, he enjoys soloing freely.

00:00:23 For such an anti-pairing person, we are lucky to have him tonight to pause and share his knowledge on compartmentalizing the monolith. Please welcome Julián Pinzón Eslava to talk about 'All You Need is Rails Engines: Compartmentalizing Your Monolith'.

00:00:51 That was great. I have been a musician my whole life. I even went to school for it; oops, this is the wrong stuff.

00:01:00 Thank you very much. I went to school for music and audio engineering. However, I have always been interested in computers. Through the twists and turns of life, I ended up teaching myself how to code with my best friend, although I struggled quite a bit.

00:01:36 After decades of being a musician, my mind was simply not wired like that of the software engineers I was competing against at job interviews. To this day, it still isn't, but what I did know was how to write songs and express complex feelings through music.

00:02:00 After some time, I discovered that one of my strengths as a software engineer was in the design aspect. I enjoyed creating beautiful experiences in code, similar to how I spent countless hours seeking the right ways to express feelings and find the right metaphors.

00:02:24 Now, I find joy in taking the time to find the right name for an abstraction or to refactor code so that anyone can understand it. After years of studying Rails and working with it in several companies, I became comfortable building applications with it.

00:02:56 However, as those products grew, I struggled to maintain the simplicity that I had worked so hard to achieve. The Rails conventions that had once helped me in the beginning started to feel like they were working against me.

00:03:09 So I challenged myself to find a solution that would reignite my belief in creating beautiful Rails code and systems that are enjoyable to work with, regardless of their scale.

00:03:21 I happened to find the answer in Rails Engines. Welcome to 'All You Need is Rails Engines: Modularizing Your Monolith'.

00:03:36 You can find me everywhere as Pinzón Julián. I come from the beautiful city of Bogotá, Colombia, which is located 2,600 meters up in the Andes mountains.

00:03:49 This is a photo of Bogotá, but there's something even better than the view—arepas! Arepas are absolutely delicious, corn-based flatbreads enjoyed in many ways.

00:04:12 Typically, you'll have arepas for breakfast or as a snack, perhaps with a little butter and maybe a hot chocolate on the side. Some are made from sweet corn and filled with cheese, while others, called arepa huevos, are double-fried with an egg inside.

00:04:25 Lastly, arepas from the Santander region include pork rind in the dough, making them savory and absolutely delightful. If you're interested, there's a nearby place to get arepas—just hit me up.

00:04:38 All right, let's talk about the app we're going to build today. It will be a hypothetical music universe app that helps teachers organize and plan music lessons for their students, perhaps even sending billing reminders.

00:04:51 But it's my idea, so don't go establishing it yourselves or forking the repository. The beautiful feeling of a new Rails app is like freshly baked cookies coming out of your computer.

00:05:16 You start coding and building features, like user authentication. To keep things organized, you decide to create an authentication module. However, as time passes, you add more models, controllers, views, and jobs.

00:05:39 To maintain organization, you use modules to indicate that certain pieces are related, but soon your folders begin to spread across your codebase.

00:06:10 Soon enough, your app is doing well; you've gained traction and your teachers love your product. However, they're struggling to retain students because kids today gravitate toward apps.

00:06:29 Your initial idea of sending emails for homework seems outdated—who uses email these days? Everything is about apps and notifications now. They plead with you to create an app for their students, and you know in your gut that it’s a good idea.

00:06:43 You start building that app, and suddenly, you transition from a monolith you love to a monolith you might end up hating in a few months.

00:06:58 By introducing a new namespace that doesn't quite match the others, you sign an agreement to incur considerable technical debt. This new module models a new actor in your system, which complicates the structure.

00:07:26 Having encountered this situation in numerous Rails apps, I've long wondered how to make the codebase self-descriptive and reveal its boundaries. How do you design components that inherently signal their purpose?

00:07:50 Let me take a brief detour to discuss 'Norman Doors.' Has anyone here heard about them? A few hands, that’s great.

00:08:10 Every time I approach a door, I conduct a quick analysis: Is it a normal door or not? Have you ever pushed a door expecting it to open, only to realize you actually had to pull it instead? It's very frustrating!

00:08:37 In a well-designed door, the mechanism would indicate clearly whether you should pull or push. A properly designed door guides the user with its design.

00:09:03 I care deeply about the experience of a codebase and its navigability. There’s a Rails app that I love, which adheres to the conventions of the framework, making it clear and familiar even for first-time users.

00:09:22 At the heart of the age-old meme 'Rails doesn’t scale' is a misconception. It's not about scaling resources; we can distribute our code over more powerful servers and balance load effectively.

00:09:39 However, we need to consider the volume of code in our apps and how that complicates the scaling of not only our domains but also our teams. The beauty and speed of Rails conventions start working against us.

00:09:59 The typical solution seems to be extracting code into microservices, effectively moving away from the monolith, but that complexity can be overwhelming.

00:10:23 I began to contemplate how I could transform the Rails monolith I currently dislike into multiple small Rails apps that I appreciate, while keeping the solution simple.

00:10:41 I initiated my search for an answer while wearing my designer's hat. Could we structure our application to resemble two or more distinct apps within the same monolith, each having its own models, views, controllers, and tests?

00:11:04 To achieve this, I considered altering the folder structure within the app directory. It doesn’t have to follow the traditional Rails setup, so I decided to create a features folder within the app directory.

00:11:32 Within this new 'features' folder, I created subdirectories for models and controllers. This structure could offer a modular approach while respecting the sidework autoloading mechanism.

00:11:49 But I noticed that in this scenario, I would have to override the table names to avoid having the word 'models' included in them. While this is possible, it makes the structure unconventional.

00:12:05 Alternatively, one could completely deviate from Rails conventions, foregoing the traditional folders for modules and controllers. You can create your own structure as long as you adhere to the sidework loading conventions.

00:12:26 However, a challenge arises with views. You will need to instruct Rails to look for views elsewhere in your application. This approach works, but it also makes the application feel less like a typical Ruby on Rails app.

00:12:54 All the power lies in the established conventions of Rails applications. You can transition from company to company and feel at home each time.

00:13:14 Now, let's discuss Rails Engines. If you’ve read recent articles, you may have noticed that Shopify has advocated for Engines as the solution, so I decided to give it a shot.

00:13:35 I consulted the Rails guides, purchased a few books, read every article I could find, and embarked on a journey to design my dream application structure using Engines. Here’s how it went.

00:13:52 Using a completely new fresh app, I initiated the generator provided in the Rails guides and noticed it created a Git repository instantly. I didn’t want to manage two Git repositories, so I had to find out how to skip that feature.

00:14:20 Fortunately, there's a flag to skip the Git repository, so I utilized it. Upon proceeding, I found that the structure it created was quite similar to what I envisioned—a framework with assets, models, controllers, and jobs.

00:14:48 I was excited to see this structure, and when I initiated a commit, I discovered 80 files had been generated. Where did all this come from?

00:15:09 The generator assumes you will publish your Engine, so it aims to create a completely generic structure that works in any Rails application worldwide. Logically, you would also need to test it against a dummy application that is a complete Rails app.

00:15:27 Engines will also have Active Record models and require their own migrations and tables. You would need to define those migrations and models in the Engine itself.

00:15:46 This is comparable to how the generator for Devise creates migrations for your app during the installation. However, I found all this to be too complex to manage.

00:16:09 I appreciate the advancements that Rails Engines have facilitated, but they're not what I was looking for; they feel overly complicated. Instead of simplifying concepts, they seem to be expanding them.

00:16:31 In reality, these Engines appear much closer to the experience of microservices. They may not deploy to different servers, but they certainly give that impression.

00:16:53 I craved a solution that felt small and familiar, yet granted me more control over the growth of my applications. Throughout my research, I frequently returned to Shopify's articles on decomposing monoliths.

00:17:15 They stated, and I quote, 'We started with a lot of custom code. Our components evolved to resemble more and more like Rails Engines. We are doubling down on Engines going forward, as they are the modularity mechanism that comes with Rails out of the box.'

00:17:39 They further articulated that other than apps, we can execute multiple Engines within the same process. Moreover, should a component need to be extracted from the monolith, an Engine can readily transform into a standalone application.

00:18:02 I began to ponder this gap that Engines might fill. Were they managing 80-plus files per domain slice while effectively maintaining such complexity? I really didn't think so—it seemed excessive.

00:18:26 In pursuit of greater understanding, I started stalking individuals, following experts in every social media outlet and on GitHub. I started every repository—the works.

00:18:50 Finally, a key clue was revealed through a GitHub notification: Rafael from the Rails core team published a one-commit repository called 'Lego.' The commit message read, 'Prepare a componentized application.'

00:19:13 The readme indicated that this was created using the Engines generator, but it removed a few files. Intrigued, I began removing files from an Engine I created to find the leanest version.

00:19:44 I executed the generator again, closely following Rafael’s readme, and found it worked beautifully. After this success, I sought to push further to see how many additional files I could eliminate.

00:20:07 I removed the gem file, the bin folder, the gem spec, the database, the test folder, the app folder, and the config folder. Essentially, I stripped everything away until I hit a roadblock.

00:20:35 A traditional Engine needs to load as any gem would through the gem file. If I delete the gem spec, how will Rails know how to load your code? I found myself stuck.

00:20:59 To create the smallest, most accessible Engine—one that feels like my code, something I truly understand—I needed a way to load it properly.

00:21:20 In my deep dive into the Rails codebase, I stumbled upon an insightful comment about the Engine class, which isn't found in the Rails guide but rather in the documentation.

00:21:41 The Rails Engine class allows you to wrap a subset of functionality within a larger packaged application. Every Rails application is, in essence, a Rails Engine, which simplifies sharing features.

00:22:08 Specify an Engine inside your lib folder, ensuring its file is loaded at the beginning of your config/application.rb file, and—drum roll, please—it will automatically load your models and controllers.

00:22:31 This Engine class operates as a code-loading assistant. It's much simpler—no 80 files, just one file, and you have an Engine with a way to load your code.

00:22:50 Moreover, you will still be mindful of all the conventions you’ve learned over the years while using Rails.

00:23:10 Now, how does this appear in practice? Before we start creating within our brand new Rails app, we set up a folder at the root where all Engines will reside—let’s call it engines, components, or packages.

00:23:31 Creating an Engine entails a few simple conventions. Firstly, create a folder for your Engine. You can name it whatever you want; for example, I’ll call it billing.

00:23:50 Inside, you'll want to create lib and app folders. Then, within the lib folder, add a file named engine, where you'll establish a class called Engine inheriting from Rails::Engine.

00:24:09 To inform your application about this Engine, you'll need to require it similarly to how one would set up a typical gem.

00:24:26 That’s it! Now, you can add anything within the app folder of this new Engine, and it will be loaded in the same way as any code within your application.

00:24:41 When you run the Rails console, you’ll see the billing module defined along with the billing model you initially created in the app/models folder.

00:25:01 I don't know about you, but I'm starting to feel the warmth of those fresh cookies again! I was elated when I discovered this.

00:25:20 This could have significantly benefited all of my previous codebases. We finally found a way to transition from a traditional Rails app to something novel.

00:25:45 Now, what can you do with this new structure? Let’s create a brand new app—again, the music universe app.

00:26:02 Before diving deeper, let’s agree on what’s about to happen: your Rails app will continue functioning similarly to any Rails app you've worked with previously. It is still a single Rails application, albeit split up.

00:26:17 Essentially, this means one application.rb file, one database.yml file, and the same straightforward three environments present in a vanilla Rails installation.

00:26:36 With that out of the way, we’ll completely delete the app folder within our Rails app. I know, it sounds strange considering that the app folder typically contains our domain code.

00:27:01 However, since our domain code will now be spread across different Engines or packages, we no longer require a singular app folder to contain it.

00:27:17 Next, we’ll plan our app structure. We’ll want to create a teacher app, a student app, and perhaps an admin app for ourselves. We then follow the steps we discussed to create these Engines.

00:27:38 Afterward, we can simply require them within the application.rb file, ensuring each app can maintain its own models, controllers, views, and even tests and factories.

00:27:55 What happens, however, when the teacher app becomes excessively large, containing specific areas worth extracting? Well, you can actually create Engines within Engines.

00:28:10 In this case, I might use the term features to encapsulate all the new packages or Engines contained within the teacher app. Again, you can choose any terminology you prefer.

00:28:23 When I pitched this idea to a few friends, they expressed concern about the vast number of folders this strategy creates—not merely the nested Engines but the other Engines too.

00:28:38 At first, I felt discouraged by their remarks. It made me reflect; why indeed must there be so many folders?

00:28:55 However, I realized that I was trading a solid architectural mechanism for a User Experience concern—something that I could fix. In various IDEs and code editors, you can create custom views or workspaces.

00:29:15 Additionally, you can adjust settings to compact folders, thus minimizing their footprint in your workspace.

00:29:30 This optimization grants you an experience where each of your different modules feels like a very small Rails app.

00:29:47 I’m not claiming this is the best approach for starting an app. Understanding which parts of your application to extract into separate units is perhaps the most complex problem.

00:30:03 You’ll only know how to achieve this once you acquire sufficient domain knowledge. But it's essential to recognize that it's feasible.

00:30:19 And you don't require any new tools or gems to implement this strategy.

00:30:30 Let’s explore some important details you need to keep in mind when implementing this in a real application, especially concerning migrations.

00:30:49 As I mentioned earlier, we won’t follow the traditional approach of managing migrations within an Engine. Instead, we treat this application like a standard Rails application, albeit structured differently.

00:31:00 For example, we will create the model file first to identify the corresponding table name; in this case, that would be 'teachers' for a teacher model inside the teacher app Engine.

00:31:20 Afterward, we can run the migration generator to create the necessary tables. Keep in mind, we are segmenting our domain code, not the configuration.

00:31:35 You may notice that the teacher class now derives from ApplicationRecord; however, remember, we deleted the app folder, which was where that class initially lived.

00:31:51 You might presume this wouldn't function, and you're correct—it won't! I deleted the app folder merely for dramatic effect.

00:32:06 We do need a designated space to define all of these core classes, though. I propose creating an Engine for that purpose.

00:32:25 I'll establish a package named Core, similar to the others, and create all of my classes, including ApplicationRecord and ApplicationController.

00:32:40 If you examine this app structure, something may seem odd. I’m calling the Core package 'Core,' yet I'm not using the 'Core' namespace for naming my classes.

00:32:53 Given that Sidework will load all code regardless of the engine, you can define classes within any Engine at the top-level module.

00:33:07 We can maintain an Engine to organize our core classes without Core becoming a namespace.

00:33:25 After that setup, running migrations will work seamlessly.

00:33:41 One of the significant advantages for me is having my tests alongside the code they are intended to test. However, when creating a new Rails Engine, it will usually generate its own test helper.

00:33:59 If we didn't create it while setting up these Engines, we can utilize the standard helper provided in our Rails app, which is installed by default.

00:34:17 To run tests across all packages, simply execute bin rails test packages followed by a slash, or to test a specific package, you can run test packages/teacher app.

00:34:38 Just remember: packages, Engines, components—choose a name that conveys meaning to you.

00:34:57 You can also use Factory Bot to configure factories per Engine. If your Engine contains controllers, that will manage routes, and fortunately, the routing process remains identical as before.

00:35:17 You will have the regular routes in config/routes, and each Engine will maintain its own routes—these need to be configured within the routes file of the Engine.

00:35:34 Then, you simply mount the Engine within your application’s routes, resulting in a singular route folder that will mount all your various Engines, including any nested ones.

00:35:54 It may become complex, yes, but it's entirely feasible. You can even implement constraints, allowing your Engines to serve resources from a subdomain or restrict by IP.

00:36:12 You might collaborate with your local DevOps expert to explore different configurations.

00:36:28 I also stumbled upon a novel gem called Cyprus, which allows you to build a static web page within Rails. With this, you could separate your marketing site and blog within different Engines.

00:36:46 There are many other details I won’t be able to cover, such as various types of configurations you can add to your Engine, including initializers and assets, but many are pre-configured to work out of the box.

00:37:07 Some of you might wonder if there’s a significant difference between the traditional app folder with module-namespaced classes and this new setup, and you are correct; it isn’t drastically different. Technically, you can achieve the same with a single Rails folder.

00:37:25 However, the structure this approach communicates is markedly different. It indicates the intent more clearly, making the operation of classes transparent.

00:37:44 What I mean is having this separation signals a concrete home for classes, delineating their boundaries and indicating how to utilize them effectively.

00:38:02 These new homes will naturally expand within their defined limits and protect themselves from external interference with minimal friction.

00:38:16 You can start thinking about your application as a collection of manageable components. I rewatched a 2012 talk by Rich Hickey titled 'Simplicity Matters,' where he discusses developing sensibilities around entanglement.

00:38:34 In this talk, Hickey describes simplifying code as the primary source of true agility because it fosters change. It allows us to choose between solving problems simply or through complexity.

00:39:00 This analogy reminds us of that tangled drawer of USB cables at home; whenever you try to pull one out, the entire ball of cables follows, illustrating the problem of tightly coupled components.

00:39:19 In the context of this discussion, what do simplicity and complexity mean for our modular Rails applications? We need to build something that resembles the familiar structure of Rails apps and allows many classes to interact with each other.

00:39:36 However, engaging too many interdependencies could lead to messy implementations. Earlier in this talk, we created some connections using the ApplicationRecord class from within the teacher app Engine.

00:39:54 Each Active Record model within our application typically requires a valid dependency. This constant interlinking illustrates how Rails' simplistic creation process sometimes works against us.

00:40:13 The ease of creating components can lead to neglect regarding the ownership of each class, an essential element for maintaining a coherent and manageable codebase.

00:40:30 Knowing the owner of a class and who should have access to it is foundational for constructing clean systems, not intricately tangled arrangements.

00:40:48 This discussion is titled 'All You Need is Rails Engines' because all classes, modules, and constants across our Engines load freely throughout the application.

00:41:06 Thus, this makes it challenging to guard against indiscriminate use of classes. This is where the Packwerk gem by Shopify comes into play, which helps maintain healthy relationships within your domain.

00:41:24 It aids in making conscious and deliberate choices regarding new dependencies, encouraging you to design a proper public API.

00:41:45 To begin using Packwerk, you will need to add the Packwerk gem to your Gemfile, and then create the Packwerk install directory.

00:42:00 Executing 'bin packwerk init' will generate all the necessary files for Packwerk to operate. Packwerk operates by defining packages, which is why I had been using the terms Engines and Packages interchangeably.

00:42:18 In our case, a package is simply a collection of code residing within a folder, matching the Engines we developed for the music universe app.

00:42:39 We make every Engine in our app a package by establishing a 'package.yml' file within each.

00:42:56 Now that we've established how to convert our lean Engines into packages, we need to discuss a couple of important concepts related to Packwerk.

00:43:13 First is the idea of privacy. I conceive of privacy in packages similarly to how we define privacy in Ruby classes.

00:43:30 You define a public interface for your class so that other classes can invoke its methods; however, you also maintain numerous private methods exclusive to class use.

00:43:47 Privacy in packages operates in much the same way, but it pertains to the folder level rather than at the method level. Most of the code within your package will remain private.

00:44:06 This creates a robust barrier, enabling you to call classes, constants, and modules privately without fearing negative effects on external components.

00:44:20 To expose your package's functionality to others, you’ll need to make some of that code public. This means you'll open your package selectively to the world via specified classes and modules.

00:44:38 Under default conditions, everything will be private unless placed in the app/public folder. This can be customized if desired, but this is the default arrangement.

00:44:53 I like to visualize the public interface of a package similarly to the interface of any gem that encapsulates an API. Recently, I have been working with the Octokit gem, which is an official Ruby gem for the GitHub API.

00:45:12 The overall design includes an Engines module (the name here is Octokit), with any corresponding classes, in this case, a Client, sitting within it—this acts as the interface class.

00:45:29 As such, it houses methods that will invoke private classes and methods exclusive to that package.

00:45:48 For our Music Universe application, a hypothetical public interface for the teacher app's billing package might resemble this.

00:46:04 Next, let's discuss another fundamental concept: dependency. Packwerk requires you to be explicit and intentional about the relationships between packages.

00:46:22 If one package depends on another, you must explicitly declare that relationship. For instance, the teacher app package relies on the core package, as it makes use of the ApplicationRecord class.

00:46:42 Likewise, the student app package also depends on the core package for the same reason. Now, how do you confirm that you've configured everything correctly?

00:47:03 Packwerk introduces the concept of violations, which encompasses various types of violations you might encounter and will need to resolve for your application to remain healthy.

00:47:21 Take this example: the teacher app is calling another private class within the core package, creating two separate violations.

00:47:37 First, this creates a privacy violation since it attempts to access code from the private section of the core package. Secondly, it also becomes a dependency violation because we did not indicate that this call depends on the core package.

00:47:54 On the contrary, if the teacher app accesses the public interface of the core package, that is acceptable since it utilizes the public API.

00:48:08 However, if it doesn't declare its dependency on the core package, that is another oversight. You must remain intentional about that.

00:48:24 Should you declare that dependency and simultaneously access private classes, you would still commit a violation as you're not using the designated public API.

00:48:47 In Packwerk, a healthy relationship consists of clearly declared dependencies alongside the consistent use of public APIs.

00:49:00 When executing a Packwerk check without any violations, your output will reflect 'No offenses detected,' indicating that your relationships are healthy.

00:49:15 This can be incorporated as a pre-commit hook or within your CI, and there's a VS Code plugin available, enabling you to receive instant feedback during coding sessions.

00:49:32 The documentation provided by Backwerk is quite comprehensive, and I highly recommend reviewing it for further insights.

00:49:45 There is also an excellent talk titled 'Laying the Cultural and Technical Foundations for Big Rails' by Alex Evanchik that is well worth watching.

00:50:01 In conclusion, Engines function as straightforward code loading assistants that facilitate modular architecture.

00:50:17 Using Packwerk helps keep things neatly organized and creates a seamless process by which any package is merely a few steps away from becoming a microservice.

00:50:32 If the time comes to extract something that no longer warrants inclusion within your monolith, the decision will shift from focusing on code quality to addressing operational constraints.

00:50:47 I personally feel a sense of relief knowing that a clear path and techniques exist, most of which are already integral to Rails or designed to work closely with its conventions.

00:51:03 I encourage you to try this approach, explore its functionalities, and discover its potential to extract components from your applications. I envision a future where this isn't merely an obscure technique among large-scale Rails organizations like Shopify, but rather a mainstream architectural pattern.

00:51:40 I would love to see generators that automate the engine creation process or scaffolding that works not just for standard apps, but within folders of our lean Engines.

00:51:54 I hope to witness gem developers consider alternative structuring methods that cater to unique applications. Furthermore, I would be excited to see practical instances of a single Rails app being supported by various server clusters, each equipped with different resources.

00:52:17 I encourage all of you to experiment with this and share your discoveries through writing and video content to invite others to join this exploration.

00:52:31 I hope these tools reignite your passion for creating beautiful code and systems that remain enjoyable to work with, regardless of their scale. Thank you!

RubyConf AU 2023