00:00:29.720
All right, good afternoon, everybody! How's everybody feeling? Yeah, you haven't crashed from the lunchtime meal yet, that's good! Make sure you get that coffee during the breaks.
00:00:37.860
My name is Ben Smith, and I work for Pivotal Labs here in Boulder. This is where I tweet, so please feel free to tell me I'm crazy on the internet. If you don't think I'm crazy yet, wait ten minutes, I'll try my best to convince you.
00:00:45.870
I'm here today to talk to you guys about Rails architecture for a big Rails app that I built. But first, I'd like to take a little detour to talk about something else.
00:00:52.500
I'm a fairly new conference speaker; I've only been speaking for the past year. I've got to do a lot of traveling and have spoken all over. When I got accepted to Rocky Mountain Ruby, I was super excited. It's my hometown, and I felt like I had the hometown advantage, which would make this conference easier for me. However, today I started getting really nervous. At first, I thought it was because I'm wearing shoes—I usually wear flip-flops or go barefoot. I took off my shoes, but that didn't help. I realized it was because of the people in the audience. I know my friends, co-workers, old clients, current clients, and future clients are here. Some of these people are way smarter than me; they're my mentors and people I look up to. This made me realize that everyone looks up to someone. No matter who you are, there’s someone looking up to you, inspired by what you do.
00:01:17.999
What does this mean? It means that someone's looking up to you, and you need to be amazing. When that person who looks up to you comes up and starts a conversation, like at this conference, be a mentor and help them in any way you can. This leads to my mantra: be nice! We are nice, and that's something that I love. At these conferences, be nice to everyone. If someone comes up and talks to you, share everything you know. Just be outgoing and helpful with everyone. That was my slight detour; thank you for listening. Now, back to the real talk: how I architected my big Rails app for success!
00:02:34.080
This is an architecture talk, and you might say that Rails has its own architecture. I would agree; it does utilize a design pattern that we all know and love called MVC. However, once your app gets big enough, relying solely on the Rails way of doing things might get you into trouble. I'm going to tell you a story about a Rails app that we tried our best to architect for success. What does success mean for a Rails app, particularly in the context of a product for a client?
00:03:16.500
In this case, we knew that the engagement would last approximately eight months or longer. At Pivotal, that is a big chunk of time; most of our projects last three to six months. Knowing this, we anticipated a large development team that would ramp up to ten Rails developers and four iOS developers. Given this particular project's from-scratch rebuild nature, the client wanted to replace their existing codebase and toolset. To achieve feature parity with their existing tools, we would ultimately end up with a lot of code. The client also wanted to make sure we had scalability in mind—not that it needed to handle a million requests per second on day one, but it should be built in such a way that it could get there eventually. Lastly, they wanted a big bang release—replacing their old codebase all at once. This isn’t something we usually recommend, but we worked with them, hoping we could release sooner than when everything was finished.
00:04:04.050
So, what does success look like in these terms? Normally, I’m all about doing the simplest thing possible, so take this talk with a grain of salt. What I'm going to show you today is not a silver bullet; it will help you in some cases, but in others, it may only complicate things. I'm not an expert on this subject; I’m just someone who has tried a few things, and this is my experience.
00:05:03.990
With these constraints in mind, we decided to think about architecture from day one. The first decision we made was to use rails engines. However, we decided to use engines in a non-traditional way. Who here knows what a Rails engine is? Almost everybody? Great! For those who don't know, a Rails engine is essentially a miniature Rails application that can be packaged and reused. This includes models, views, controllers, CSS, JavaScript—everything that you can bundle together and reuse as a dependency.
00:05:19.589
A popular example of an engine is Devise. Devise is built as a gem and can easily be included in your Rails codebase, providing views, models, and database tables. Although we used engines, unlike Devise, all our engines were never built into gems. They weren't pushed to Ruby gems; furthermore, they didn't even have their own Git repository. All the engines we created lived within our normal Rails source tree. This meant that, for you SOA advocates, we didn't have to worry about interface versions or separate deployments. All the engines I am discussing in this talk lived inside a single Rails app, which we deployed as one. They weren't separate applications.
00:06:12.120
You might ask, 'Why use engines in the first place?' We decided that there were a few distinct components of our app that we could separate into manageable pieces. To give you a little background on the product, it was essentially a content publishing platform for videos, audio, blog posts, Facebook posts, tweets—all types of content. On top of that, it included a full social network. The content was published and consumed by an iOS app fed by a JSON API, and the content was created and moderated via a web interface. While this domain may not seem too complex at first glance, we focused on architecture, so we took our simple Rails app and began breaking it apart.
00:06:58.800
In this diagram, you can see we have our Rails app, and inside it, we created four engines. The arrows indicate the dependencies. I will refer to this main wrapping Rails app as the 'Wrapper Rails App.' To the left, we have an admin component where content gets created (videos are uploaded, and blogs are written). On the other side, we have the social network. At the bottom, there's a user-facing JSON API providing content. We created a 'Common' engine to house items like users and roles that were required by both the admin and social network engines. At the top, we included a 'Scheduler' engine that was tasked with publishing content so that the admin could create it and have it copied over to the social network or published to an external web service when it was scheduled to go live.
00:07:45.300
Moving forward, I won't draw boxes around the engines, so just assume that all the engines I'm discussing are within a single Rails app. Remember, the arrows indicate the direction of the dependencies: the Scheduler depends on the Admin engine, the Common engine, and the Social Network engine. The Admin engine only depends on the Common engine, and the Common doesn't depend on anything. Does this example make sense to everyone? Because it’s about to get quite a bit more complicated.
00:08:28.200
The next thing we did was write a gem to wrap an SMS service called Mobile Compass. We chose to make this a gem rather than an engine because it didn't need a database connection, the asset pipeline, or anything else that Rails provides. As shown, Mobile Compass is represented as a gem and not an engine like the others. Despite this, we wrote a significant amount of code using just these four gems and one gem, but as we progressed, we began identifying users of the system for which we could create specific engines.
00:09:07.680
We realized that there were two types of admins: a global admin, who could CRUD users, and a content admin whose sole purpose was to create content. Thus, we split these roles into two engines that both depended on an 'Admin Asset' engine. The Admin Assets engine contained the layouts and styles that the admin engines used. Once we realized it made sense to create separate engines for each role in the system, we created another engine called 'Social Admin.' The Social Admin engine was tasked with moderating wall posts, status updates, and other user-generated content. We felt good about this; we were successfully breaking our app down into smaller pieces.
00:09:48.360
Then, we had a breakthrough, but it wasn’t ours; it was our office director, Mike Bare. He walked by and pointed out that our Social Admin engine depended on too much. It didn’t need a dependency on things like the controllers in the Social Network engine; all it needed were the models. Remember, the Social Network engine is the user-facing JSON API, and the Social Admin engine is the web interface to moderate wall posts, status updates, and more. It took me a while to come around to this idea, but he had a valid point. The Social Admin engine depended on the entire Social Network engine, which meant it had references to things like controllers that simply weren't needed. Keeping your dependencies to a minimum is always a good idea.
00:10:32.400
What we wanted to do was depend solely on the models of the Social Network. This allowed us to create controllers around wall posts and status updates for moderation. We went back to our engines and performed a refactoring, moving the models out of the Social Network engine into a new engine called 'Social Network Content.' We renamed the Social Network engine to 'Social Network API' because it now contained only API code: just controllers and JSON presenters, without models or domain logic. Finally, we established a dependency from the Social Admin to the Social Network Content engine, which allowed the Social Admin to only require what it needed. Instead of the previous structure, we simplified it to look like this, addressing the issue of excessive dependencies.
00:11:51.600
We realized that the Scheduler engine had a similar problem; it was requiring the Content Admin engine when all it needed were the models from that engine. Thus, we extracted that into a dedicated Content Admin engine and updated the dependencies to reflect that refactoring. At this point, we began to notice a pattern: some engines could respond to HTTP requests, specifically those containing controllers and routes, while others only contained models or business objects. We started categorizing them as web engines and domain engines, with corresponding color-coding for visualization, making it easier for the team.
00:12:35.760
Identifying these patterns and naming them is incredibly helpful. Once you identify a pattern, you can think in terms of what makes sense for that pattern. For example, if someone suggests adding a users controller to the Common engine, I could say, 'No, that doesn’t make sense; it’s a domain engine.' Speaking of common engines and patterns, we realized that naming an engine simply “Common” is a poor idea. The term 'common' implies that you can place anything there, similar to the 'lib' folder in Rails where you can throw in anything.
00:13:23.460
We renamed it to accurately represent what we wanted it to be—'Users.' Upon doing so, we realized there were items in there that shouldn't be there, specifically a profanity filter. So, we extracted that into a separate engine, and through our continued work, we reached a more detailed architecture after about six months of development. However, as the app kept growing in complexity, visualizing it became increasingly challenging.
00:14:02.640
This led me to stop and assess how our architecture appeared. I won't lie; it looked complex and overwhelming. When observing this diagram, it’s understandable to think, 'This is insane; there’s no way I could work with this.' However, keep in mind that this complexity isn't due to the use of engines; it reflects the complexity of the app as a whole. We would have these same dependencies even without engines, but engines allow us to discuss them more easily. In reality, while working on this app, you only need to understand the part you're working on, focusing on one subset of the graph itself.
00:14:46.560
Generally, the pattern we noted was that the web engines at the top would depend on the domain engines below them, which in turn depended on the database. If you think about it, that’s not far removed from the traditional Rails stack, where controllers and views depend on models that depend on the database. While this setup might look intimidating, it doesn’t differ significantly from the standard Rails stack. If you don’t believe me, now might be a good time to tweet this handle and tell me I’m crazy!
00:15:30.540
Now that you’ve seen how engines can be implemented, you may be wondering how to get started in this intriguing world of engines. One of my pet peeves with other architecture talks is the lack of actionable steps provided, so I’m going to share some insights on how to create a Rails engine. You begin by running a command from within the root of your Rails app, and don’t worry, I’ll post these slides online, so you don’t have to take pictures or notes.
00:16:06.509
In the Gemfile, add a line that creates the dependency for your newly created engine. Of course, you can add dependencies from your wrapping Rails app to an engine or from one engine to another. Lastly, if your engine has routes and controllers, you need to mount it—but this only applies if you’re working with a web engine. If you want to create a gem, it's almost as easy. Here’s how you do it, and how you require it.
00:16:44.580
Once you're comfortable creating these engines and gems, instead of running the command each time you need a new engine, run it just once to create an engine template. From that point, all you have to do to create a new engine is copy and paste the engine template and indeed do global find-replace on a few strings and rename a couple of files, ideally through a script you write. This process provides a standard base template to work off. If you want to include basic dependencies like RSpec or HAML, you can easily do that within your template. The same concept applies to plain gems—you'll want to create a template to reproduce these gems efficiently.
00:17:23.640
The goal is to lower the barrier for creating engines and gems to just a couple of minutes tops. We found it’s much easier to decide to merge code together than to split it apart later. If it’s easy to write code that’s already split apart, developers are more likely to do it. Alright, now that we’ve covered that, the key takeaway is that these engines should be completely self-contained.
00:18:03.600
What that meant for us was that all database tables were namespaced. All migrations lived inside the engines, and the tests were run only within that specific engine, ensuring that it remains self-contained. This architecture requires clear, logical boundaries—almost in a way that resembles a full Service-Oriented Architecture (SOA). Instead of the tables in one database, each table was prepended with the engine’s name it belonged to. This transition helped create clarity and organization.
00:18:56.310
Another goal for our engines was to keep the migrations inside of them. Interestingly, Rails doesn’t fully support this out of the box—while it has some capabilities in this direction, it's not designed for the use case we required. Often, you install a gem (like Devise, for example) as a dependency, and you run a rake task that copies all migrations out of that gem before using them in your main app. However, this was cumbersome for us, as our engines were often changing, and running that rake task every time we had a new migration was quite tedious. Thus, we chose to monkey-patch Rails to allow migrations to stay tied to their respective engines.
00:19:35.619
Additionally, we ensured each migration touched only one database table. This may seem like an arbitrary constraint, but as we worked with this architecture, it proved beneficial. Should we wish to move an entire database table from one engine to another (a somewhat radical idea for a service-oriented architecture), we could easily do so. If each migration only affected one table, all we had to do would be to copy and paste those migrations into the new engine, rebuild our test databases, and everything would continue to operate smoothly.
00:20:29.370
Lastly, we imposed a constraint of testing our engines in isolation. This means that tests associated with an engine’s functionality remained within that engine, ensuring when run, they require only that engine being tested. This isolation proved that an engine was indeed self-contained—without dependencies on any code not explicitly required. For instance, tests for the Admin engine would reside within the engine itself, allowing us to validate that there’s no invocation of methods or classes from the Social Network engine unless an explicit dependency was defined.
00:21:33.540
In keeping with this approach, we made some observations regarding how engines should interact with one another. Avoiding circular dependencies was key for us since they complicate refactoring efforts down the line. If you look at our dependency diagram, you’ll note there are no circular dependencies—it forms a directed acyclic graph. In instances where it seems unavoidable, we generally found it indicates that we were trying to split components out too broadly.
00:22:12.120
For example, if you have two engines where Engine A needs to depend on Engine B, but Engine B also requires Engine A, it's often more logical to combine those into a single engine. Avoiding circular dependencies promotes clearer architecture. Understandably, Rails can often facilitate them; for instance, a user may have many posts and a post belongs to a user, which seems natural and is indeed the Rails way—but it complicates refactoring once your app grows.
00:22:53.160
For prevention, some teams may opt to minimize the use of ActiveRecord associations entirely, while others may implement a domain API layer or service layer, serving as an interface to the domain logic. This layer comprises simple Ruby classes that wrap around ActiveRecord calls, taking in IDs or parameters, and returning plain Ruby objects. To illustrate, a simple example might be a UserManager class with a method 'find_user_by_id' that leverages the ActiveRecord find method. Placing the ActiveRecord class under a 'private' section of your manager reduces its visibility and prevents its direct manipulation outside of the intended context.
00:23:52.890
Instead of returning the ActiveRecord object directly, we would pass the user attributes to a user instance, thereby reducing the likelihood of someone directly invoking ActiveRecord’s methods on the user class. Although this is a core design pattern, and it may seem like I've coined the term 'Ben’s Awesome Pattern,' it really isn't new—this has been an established best practice in object-oriented design. The intention here is to ensure that when you modify or extend your application, changing the user class is often unnecessary, allowing you to maintain clean separation and avoid creating murky dependencies that can lead to tightly coupled components.
00:24:47.429
In terms of circular dependencies (or the absence of them), minimizing these allowed us to transition from a scenario where one Rails app references everything else, to a more modular arrangement where we could break everything out into engines with clear, single-direction dependencies. The reduced circular nature of the architecture creates opportunities for more efficient testing processes.
00:25:31.800
For instance, if you're making changes to the Admin engine, you’re not required to run tests across all the engines. You only need to run tests relevant to the Admin engine or the Scheduler since any modifications in the Admin engine might affect the Scheduler’s functionality. However, you would not need to run tests for the Common engine or Social Network if there were no code changes to those respectively dependent components. This provides the advantage of constructing a much more efficient and smart test suite.
00:26:16.560
Through establishing one-way dependencies, we noted that usually, only a fraction of your tests need to be executed following a change in code. As previously stated, the user engine seldom changes, preserving semblance in the overall structure, while when it does change, its test cases only run when necessary. Consequently, when you include new features, such as comments, 'the user engine' itself does not require modifications, thus promoting a cleaner and less complex architecture.
00:27:00.930
How did the integration of engines, domain APIs, and all the conventions we've discussed work out for us? Initially, engines posed some difficulties; we experienced low velocity during the first month and a half, causing the client to become concerned. More developers couldn’t be brought on board as we were learning how to effectively implement these patterns and best practices for using engines. Continuous refactoring was a requirement throughout the project, to the point at which we found ourselves allocating about a third of our time solely on refactoring.
00:27:54.180
Brian talked this morning about effortful versus automatic work, and all of this was, indeed, effortful—requiring intensive thought and planning, in contrast to standard Rails work, which is often more automatic. While we faced several technical hurdles—such as monkey-patching of migrations—over the long run, the use of engines proved beneficial. By the end of the eight months, we noticed no slowdown in development time, nor did we experience delays in pivotal feature implementations.
00:28:36.420
Engines aged remarkably well. Often, after completing work on an engine, we could ignore it for months without any issues. They allowed for efficient parallel development since we could assign teams to focus solely on their engine, removing concerns of conflicting changes. Furthermore, the potential for scaling is immense; it’s simply a matter of extracting an engine, placing it into its own Rails app, and scaling independently. Finally, the engines allowed us to run rapid tests, contributing to the overall success of this architectural design.
00:29:28.560
Overall, I give this architecture a thumbs up, albeit it took me two months to come around to its advantages fully. If you find this information intriguing or think it might help your current apps, give some of it a try! Create a new engine for the next major feature you develop, or consider extracting some models into their own engines.
00:30:13.300
Avoid circular dependencies whenever possible, or explore the creation of a service layer. Most importantly, think outside the traditional Rails paradigm, and adopt practices that perfectly align with your app's needs. Thank you!