How I Architected My Big Rails App For Success

by Benjamin Smith

In his talk at RubyConf AU 2014, Benjamin Smith discusses the architectural strategies employed in building a large Rails application aimed at long-term maintainability and scalability. The central theme is how to effectively structure a codebase to handle an expanding team and ensure performance as the application grows.

Key points of the talk include:

- Understanding Success: Success for the project meant anticipating future scalability and handling a large codebase, given the client's needs for a complete rebuild of their existing application over an extended timeline.

- Utilizing Rails Engines: Smith introduces the concept of Rails engines, which bundle an entire Rails app's components, enhancing modularity within the application. He explains how they opted for a unique approach by keeping engines within a single Rails app while avoiding versioning issues usually encountered in service-oriented architectures.

- Architecture Design: The architecture consisted of multiple engines tailored to different functionalities, such as an admin engine for content creation and a social network engine for content presentation. This separation facilitated clearer dependencies and enhanced code organization.

- Patterns and Best Practices: As development progressed, identifying functional patterns allowed the team to refine their approach. Smith emphasizes naming conventions for distinguishing between web engines (which handle routing) and domain engines (which contain pure logic).

- Avoiding Circular Dependencies: To maintain flexibility during refactoring, the team implemented practices to prevent circular dependencies, creating a directed acyclic graph for engine interactions that simplify understanding and maintainability.

- Domain API Layer: Smith introduced a domain API to act as a buffer between controllers and models, encouraging a loose coupling of components, which aids in independent development and testing.

- Testing: By structuring tests within individual engines, the team ensured high confidence in the stability of each engine independently, reducing the overhead involved in comprehensive integration testing.

- Long-term Outcomes: While the initial phase posed challenges, the architecture eventually led to enhanced parallel development, straightforward scaling, and improved testing capabilities.

The conclusion drawn from Smith’s experiences is that although adopting such strategies can initially slow down development, the long-term advantages, including better maintainability and scalability, make them worthwhile. He encourages peers to experiment with engines and alternative structure approaches in their Rails applications.

00:00:07.359 Good afternoon, everyone! Is everyone awake? Getting your coffee in? Haven't crashed yet? Um, like Josh was saying, I'm Ben Smith, and I work for Pivotal Labs out of Boulder, Colorado.

00:00:15.200 Before I start this talk, I want to share a few observations I've made on this trip, especially regarding the differences I've seen between the US and Australia.

00:00:23.000 So, potato gems—does everyone know what this is? I don't know if we have these in the States. Well, we do, but they're called tater tots—those little fried bits of potatoes you eat in the morning.

00:00:34.559 It's a little weird that they're called gems, though, right? Because in order to consume a gem, you have to gem install it first.

00:00:41.800 And that's a little weird. We all know that you have to be careful when you gem install stuff because you could be running malicious code. Everyone knows that, right?

00:00:49.160 I think, in general, tater tots in America are a little bit more secure than potato gems, but Australia definitely is ahead of the US and other areas of security.

00:00:56.800 For example, your ATMs here actually have guards. I saw that at Manly Wharf yesterday, and I was worried he was going to steal my PIN.

00:01:06.160 But it turns out he's just protecting me from my ser... something. I don't know. But I'm not here to talk about security, Ruby gems, or even potato gems.

00:01:15.200 So, who here saw my talk last year? Nice! Good!

00:01:20.118 Last year, I talked about all that good stuff—security and gem stuff. I'm super happy to be back this year, and I'd like to thank everyone for having me back.

00:01:28.080 I feel great to be here again. I especially want to thank the organizers for making it possible for me to make the trip over. Thank you so much!

00:01:36.040 This year, I'm going to talk about architecture and Rails. You might say that Rails has its own architecture and that we don't have to worry about it too much.

00:01:42.200 Rails does use a design pattern that we all know and love—MVC. But once your app gets big enough, relying on just the Rails way of doing things might get you in trouble.

00:01:50.880 So I'm going to tell you a story about a Rails app that we tried our best to architect for success.

00:02:00.479 And what does success mean? What does it mean for a Rails app to be successful for a product or for a client?

00:02:08.000 Well, in this case, we knew our project would be a long one—it would last eight months or longer, which for Pivotal Labs standards is a big chunk of time.

00:02:17.400 The majority of our projects last about three to six months. We knew we'd have a large development team that would start off with just two Rails developers and then ramp up to ten, along with four iOS developers.

00:02:25.920 We anticipated that in eight months we would end up with a ton of code, given the number of developers we had.

00:02:35.600 This particular project was a from-scratch rebuild. The client had an existing codebase and toolset, and their goal was to replace all of that with a new one.

00:02:41.080 They also wanted to add some new features on top of it. We knew to get that existing feature set parity, we would end up with a considerable amount of code.

00:02:50.960 The client also emphasized the importance of scalability—not that it needed to handle thousands of requests per second on day one, but that we had it in mind.

00:02:58.480 At some point in the future, it could handle load like that. Lastly, the client wanted to replace their entire old codebase all at once—a big bang release.

00:03:05.640 This isn't something we normally do, but we worked with them hoping we could release a bit sooner than when it's all done.

00:03:13.560 So what does success look like in these terms? Normally, I'm all about doing the simplest thing possible, so I'd suggest you take this talk with a grain of salt.

00:03:21.919 What I'm going to show you is not a silver bullet; it will help you in some cases and only complicate things in others.

00:03:30.279 I'm also not an expert on this subject; I'm just some guy who spent some time trying some things, and I'll talk about them today.

00:03:38.480 So success with these constraints meant we wanted to think about architecture from day one, and our first decision was to use Rails engines.

00:03:44.840 We decided to use Rails engines, but not in the normal way they're usually used.

00:03:51.560 Who knows what a Rails engine is to start? Okay, a good number of people. A quick intro to engines.

00:03:58.639 Engines allow you to take an entire Rails app—so models, controllers, CSS, JavaScript—everything, bundle it up, and you can require it as a dependency.

00:04:05.840 You can potentially turn it into a gem that you can reuse in other apps. For example, Devise is a great example of an engine.

00:04:12.679 Who knows what Devise is? Has everyone heard of that great authentication engine?

00:04:18.679 The engines we created for this project were a little different; they were never built into gems and were never pushed to Ruby Gems.

00:04:27.159 Not only that, they never even had their own Git repository. All the engines we created went under our normal Rails sorcery.

00:04:34.360 This meant that for all you crazy SOA advocates, we didn't have to worry about versioning interfaces or separate deployments.

00:04:40.760 All the engines I'm going to show you today lived inside of a single Rails app, and they were deployed and versioned together.

00:04:47.240 Of course, you might ask why we used engines in the first place. We decided early on that there were a few distinct components we could separate out into their own engines.

00:04:55.680 But let me give you a little background first, so you can understand.

00:05:02.000 The product itself was a content publishing platform for things like video, audio, blog posts, tweets, Facebook posts, SMS—everything you can think of.

00:05:08.440 On top of that, it included a full social network with all the bells and whistles.

00:05:14.479 The content was consumed via an iOS app fed by a JSON API, and the content was created and moderated via a web interface.

00:05:20.020 So that doesn't sound too complex, but we're focused on architecture.

00:05:26.279 We took our simple Rails app and started to break it apart, and in this diagram, you can see we have our Rails app.

00:05:32.320 Inside it, we have four engines, and the arrows point in the direction of the dependencies.

00:05:40.240 I'll refer to this main Rails app that includes the engines as the wrapper Rails app from here on out.

00:05:46.439 On the left there, we have an admin engine—that's where the content was created, such as videos uploaded and blog posts written.

00:05:53.760 On the right side, we have the social network. This is the user-facing content in the form of a JSON API consumed by our iOS app.

00:06:01.160 We also created a common engine, where we put things like users and roles required by both the admin and social network engines.

00:06:08.240 At the top, we have another engine called 'Theer,' responsible for publishing this content.

00:06:15.400 So the admin could create content, and it would get copied over to the social network or posted out to a platform like Twitter whenever it was scheduled to go live.

00:06:22.240 Going forward, I'm not going to draw this box around all the engines; just assume that all the engines I show you are within a single Rails app.

00:06:31.640 Remember, the arrows point in the direction of the dependencies.

00:06:38.920 Theer depends upon the admin engine, the common engine, and the social network, while the social network just depends upon the common engine.

00:06:46.600 And common doesn't depend upon anything. Does this make sense to everyone?

00:06:54.600 It's about to get more complicated.

00:07:02.400 So the next thing we did was write a gem—not an engine, but a gem—to wrap an SMS service called Mobile Compass.

00:07:10.960 We chose to make this a gem instead of an engine because it didn't need a database connection or the asset pipeline.

00:07:18.560 I made Mobile Compass here red so you can see it's not an engine like the rest, just a gem.

00:07:27.040 In this architecture, we wrote quite a bit of code, but as we got further into building this application, we identified specific users.

00:07:34.400 We realized we could create their own engines for them. Specifically, we found there were more than one type of admin.

00:07:42.480 There was a global admin who would do things like create users and a content admin whose sole purpose was to create content, like writing blog posts.

00:07:51.040 So we broke these two out into separate engines, and they both depended on an admin assets engine.

00:07:58.480 Admin assets contained things like layouts and styles that both admin engines used.

00:08:05.920 Once we realized it made sense to split out engines based on roles in our system, we created another admin engine for social admins.

00:08:12.640 Social admins were moderators of wall posts, status updates, or any other user-generated content.

00:08:19.120 We felt pretty good about this; we were identifying engines and keeping our app broken up into smaller pieces.

00:08:26.239 Then we had a breakthrough. It actually wasn't us; it was our office director who walked by.

00:08:33.560 He said, 'Your social admin engine depends upon too much.' It doesn't need a dependency on things like controllers and the social network engine.

00:08:40.720 All it needs are the models. Now remember, the social network engine is this user-facing JSON API.

00:08:48.760 The social admin is an admin web interface to moderate things like wall posts and status updates.

00:08:56.160 The models for wall posts and status updates live inside of the social network. It took me a while to come around to this idea, but we did have a point.

00:09:04.960 The social admin engine depended upon all of the social network, meaning it had references to things like controllers or views, which just aren't needed.

00:09:12.560 And keeping your dependencies to a minimum is always a good idea. What we really wanted was to just depend on the models of the social network.

00:09:20.160 This would allow us to create controllers and views to moderate things like wall posts and status updates.

00:09:27.680 So we went back to our engines and did a refactoring.

00:09:32.480 We pulled the models out of the social network and put them into an engine called social network content.

00:09:40.120 We renamed social network to social network API because now it truly was just an API.

00:09:46.920 It contained controllers and presenters, but it didn't contain any models or domain objects.

00:09:54.640 Finally, we added a dependency from the social admin to social network content, which going back to the original problem allows the social admin to only require what it needs.

00:10:01.440 Instead of having this, we now have something that looks like this, addressing our original problem of requiring too much.

00:10:09.840 We thought, 'Why stop there?' We also had the same problem in our content admin engine.

00:10:17.280 It was requiring all of the content admin engine, but all it really needed was the models inside of it.

00:10:24.080 So we split out the content admin engine into content admin elements.

00:10:31.040 We updated our dependencies, and at this point, we started to notice a pattern.

00:10:38.080 Here we had some engines that could respond to HTTP requests—engines with routes and controllers.

00:10:45.400 And then we had other engines that only contained models or business objects.

00:10:52.680 So we started calling them web engines and domain engines.

00:10:59.680 Here, I've color-coded the web engines in blue and the domain engines in brown.

00:11:06.760 As a team, we found that identifying these patterns and putting names to them was really helpful.

00:11:12.640 Once you identify a pattern and name it, you can think in terms of what makes sense for that given pattern.

00:11:19.680 For example, if someone told me to add a users controller to our common engine, I could say no; that doesn't make sense.

00:11:26.040 This is a domain engine, so it shouldn't have controllers.

00:11:32.760 Speaking of an engine called common and patterns, we realized that having something called common is a bad idea.

00:11:39.040 Common means you can put whatever you want in there, like the lib folder in Rails.

00:11:46.080 So we renamed it to what we wanted it to be called—Users.

00:11:53.520 Once we renamed it, we realized there was stuff in there that shouldn't be in there—specifically, a profanity filter.

00:12:00.880 So we pulled that out into an engine. From here, we did some more work and more work.

00:12:06.880 This is where we ended up after about six months of development.

00:12:12.720 We kept on working on the app, but it just keeps getting more complex and crazier.

00:12:19.440 So I'm just going to stop here. How does this look? The first word that comes to mind when drawing this diagram is 'hell.'

00:12:27.200 Drawing a diagram like this makes the app seem crazy, right? It feels like it would be impossible to work in.

00:12:33.440 But here's the hard part to understand: the complexity is not because of the engines themselves.

00:12:39.960 The complexity is part of the domain of this application. We would have the same dependencies and the same crazy mess if we didn't have engines.

00:12:47.480 But without engines, we wouldn't be able to talk about them or see them visually.

00:12:57.560 In reality, you really don't need to know that entire graph by heart; you only need to focus and understand a subset of it.

00:13:05.720 There's also a general pattern to these engines, and that general pattern is that web engines depend on domain engines, which depend on the database.

00:13:14.280 If you think about it, that's not far off from your standard Rails stack that has controllers and views depending on models, which depend on the database.

00:13:20.520 So as crazy and scary as this looks, it's not far off from your standard Rails stack.

00:13:27.840 If you don't believe me, here's my Twitter handle in the bottom left corner. Go ahead and tweet at me, echo me on the internet.

00:13:34.520 So here’s the craziness of engines. Now you're probably asking yourself, 'How do I get there? This looks great!'

00:13:43.239 I'm going to show you how to get started with engines really quickly. Don't worry, all these slides will be posted online, so you don't have to write this down.

00:13:51.280 This is how you create a Rails engine: you run this command from your Rails app. This is how you require it in your Gem file.

00:14:00.520 You can require one engine from another engine, or you can require an engine in your wrapper app.

00:14:07.120 Then, if it has routes and controllers, this is how you mount it.

00:14:13.000 If you want to create a gem instead of an engine, it's as easy as this—same thing to require it.

00:14:19.800 The next step after you start creating these engines and gems is to run this once and create an engine template.

00:14:27.440 Once you create an engine template, all you need to do is copy and paste that engine template anytime you want to create a new engine.

00:14:33.920 Do a global find and replace on a couple of strings, rename a few files—ideally in some sort of script that you write.

00:14:40.680 This gives you a standard engine template to build off of, so if you want some standard basic dependencies or setup in your template, you can do that.

00:14:48.680 For example, if you want to standardize on using RSpec or HAML, you could do that in your template.

00:14:55.520 So that any engine you create from here on out will have these libraries included.

00:15:02.920 The same thing applies to plain old gems—you want to create a template to build off of.

00:15:10.560 The point here is to really lower the bar to create a new engine or a new gem.

00:15:17.680 You want to get to the point where it only takes a couple of minutes to create a new engine, and it cannot be screwed up.

00:15:25.000 We found that it was easier to take two pieces of code and merge them together later rather than to have one big chunk.

00:15:33.160 If you start with code separately and then have to merge it together later, you're going to be in a better spot in the long haul.

00:15:41.080 So, got all that? Makes sense? All the slides will be posted online. Let's get back to the interesting stuff—doing engines the right way.

00:15:56.720 From the beginning, we decided that engines should be completely self-contained.

00:16:02.840 For us, this meant database table namespacing, putting the migrations inside the engines themselves, and having all the tests contained within each respective engine.

00:16:09.000 Let’s talk about database table namespacing.

00:16:15.440 With this architecture, we wanted clear and real boundaries—almost as real as if you were doing full SOA.

00:16:23.079 So if you took our simple Week One architecture and converted it to SOA, it might look something like this.

00:16:30.520 This is what we wanted to mimic: messages over HTTP, separate databases, different Rails apps.

00:16:36.320 But we wanted things simple and easy, so we wanted one database, messages in Ruby, and engines instead of apps.

00:16:44.440 We took our engines with their single database and chose to namespace all the tables in that database.

00:16:52.000 So each table is prepended with the name of the engine it belongs to.

00:17:00.479 Moving from this to this would mean putting each engine in its own wrapper Rails app, breaking out the database tables by namespace.

00:17:08.479 And we added a small wrapper so you could send HTTP messages rather than Ruby messages.

00:17:16.880 The other cool thing about being able to pull out an engine into its own app like this is that you could actually scale that app on its own.

00:17:25.440 For example, the social network might need to handle a lot of requests compared to the admin, which may not need to handle very many.

00:17:32.960 Putting migrations inside each engine was another decision we made. The interesting thing is Rails doesn't fully support this out of the box.

00:17:40.000 It sort of supports it, but not for the use cases that we wanted and were using.

00:17:47.440 In cases where you run something like this—Rails install migrations—it copies your migrations out of the engine into your wrapper app where they can be run.

00:17:55.520 This works fine if you never add new migrations to your engines, but if you're adding migrations and changing things, this becomes a pain.

00:18:02.640 I don’t want to spend too much time on this, but here's a monkey patch that allows you to leave your migrations inside your engines.

00:18:09.520 The last thing I want to mention about migrations and engines is it's best to have each migration only touch one database table at a time.

00:18:17.520 The reason for this is that if your migrations only touch one database table, it becomes easy to move tables between engines.

00:18:24.560 It basically boils down to cutting and pasting all the migrations from one engine to another, rebuilding your test database, and you're good to go.

00:18:31.920 There's more info here. This is a crazy idea, though, because if you think about having a full service,

00:18:37.920 and you want to move a database table from one service to another, that could be very difficult.

00:18:43.920 In this case, it just becomes trivial.

00:18:50.800 The final self-imposed constraint we put on engines was testing them in isolation.

00:19:00.800 This means the tests for the engine's functionality actually live inside of the engine.

00:19:06.840 When the tests run, they only require the one engine they're testing, proving that engine is self-contained.

00:19:12.040 For example, if you put your tests for the admin engine inside the engine itself, you can prove there isn't any code that references things in the social network.

00:19:18.000 If you put all your tests in your wrapper app at the top level, you can't prove that.

00:19:26.760 The wrapper app loads all of these engines together.

00:19:32.880 Going back to doing engines the right way, these were the things we felt were necessary for keeping everything inside of each engine clean.

00:19:40.040 But outside each individual engine, we also concluded how engines should interact with each other.

00:19:47.600 Not having circular dependencies was key for us. Circular dependencies make it much harder to refactor later.

00:19:56.000 If you look at our dependency diagram here, there's no circular dependencies.

00:20:02.800 This is a directed acyclic graph, and if there were a circular dependency, it's just because I drew it wrong.

00:20:09.000 As a general practice, we decided whenever it seemed impossible to avoid a circular dependency, it meant we were trying to split things apart too much.

00:20:15.640 So if you have two engines, A and B, and it seems like there's a circular dependency, what you probably want is an engine that is both A and B combined.

00:20:21.520 Whenever you can avoid doing this, you should.

00:20:28.480 In Rails, it's super easy to create circular dependencies. Something like this is common: `User has many posts, post belongs to User`.

00:20:35.680 It's easy to do but makes your application harder to refactor when it gets big.

00:20:42.520 This type of code results in a circular dependency between users and posts, each referencing the other.

00:20:48.400 So how do you avoid falling into the circular dependency trap? Well, you could stop using ActiveRecord associations.

00:20:55.520 And how do you stop using ActiveRecord associations? Well, you could stop using ActiveRecord.

00:21:02.720 We didn't quite do that, but we did something along those lines.

00:21:09.640 The approach we took was to create a domain API layer or a service layer—a layer between our controllers and our models.

00:21:16.240 An interface to our domain logic.

00:21:23.680 All our domain API was comprised of simple Ruby classes that wrapped ActiveRecord calls.

00:21:30.560 The arguments to the methods were just simple IDs or params, and the output was always plain old Ruby objects.

00:21:37.480 Let's take a look at an example.

00:21:43.480 Here's a simple example of a domain API class called UserManager.

00:21:49.320 We have a method here called `find_user_by_id` that takes an ID, finds a user, and calls `UserRecord.find`.

00:21:57.520 The UserRecord, if you look below, is an ActiveRecord class.

00:22:04.640 You'll notice that the classes are defined below the private line, meaning they should only be used within the UserManager.

00:22:11.520 The ActiveRecord user is passed off to `User.new`, which copies the attributes off the ActiveRecord class and onto itself.

00:22:19.680 We could actually just return the ActiveRecord object if we wanted to, but we wanted to deter people from using ActiveRecord as much as possible.

00:22:25.040 By returning just a plain old Ruby object like this User object, the caller of this method can't do things like call `update_attributes` or `destroy`.

00:22:32.080 We want to avoid exposing any ActiveRecord calls that we don't want to expose.

00:22:39.120 This is a simple example. You'd probably want to meta-program this a bit.

00:22:46.960 Meta-program the copying over of the attributes using things like ActiveModel or ActiveNaming.

00:22:53.760 You can get form and URL helpers from Rails, but in general, this is an example of the pattern we used.

00:22:59.920 This is a brand new design pattern that I call Ben's Awesome Pattern, so please tell all your friends about it.

00:23:05.920 How does having this domain API affect this situation?

00:23:12.240 Well, in this case, there are two possible things you might want to do.

00:23:19.240 Given a user, we might want to know all the posts they've written, or given a post, we might want to know who authored that post.

00:23:26.760 So if we were to write a domain API to handle both these cases, what would it look like?

00:23:33.600 For the first case, we want to know the posts by a user. It would be as easy as creating a `find_all_by_user_id` on a post manager.

00:23:40.320 We could do something like this: `new up a post manager` and call `PostManager.find_all_by_user_id`, passing it the user ID.

00:23:47.960 This gives us back the posts, handling the case of knowing the user and wanting the posts.

00:23:54.560 So we don't need this line; the user no longer has a dependency on posts.

00:24:02.720 When we want to know the post for the user, we're going to the post manager.

00:24:11.200 Now, if we start using our user manager, we should be able to do something like this.

00:24:17.680 To find the author of a post, we create a new user manager given a post that has a user ID attribute.

00:24:24.560 We can now find the author of that post.

00:24:31.200 The crazy thing here is that using these domain API objects, there are no real code dependencies between users and posts anymore.

00:24:38.920 There is a dependency because the post has a user ID, but no circular dependency exists.

00:24:46.800 Another great thing is that the user doesn't have to change when we add posts to our application.

00:24:54.600 The same applies when you add other things; for instance, when you add comments and videos, the user still doesn't need to change.

00:25:02.880 You'd be creating small domain API objects for each of those things.

00:25:10.560 So you'd have a comment manager and a videos manager, and all those things would depend on the user, not the other way around.

00:25:18.760 Now, who's seen user models in Rails apps that are hundreds or even thousands of lines long? Just gigantic mess, right?

00:25:26.520 This pattern completely avoids having those huge classes or those God objects.

00:25:34.080 Instead of having a few huge classes in your application, you end up with a bunch of small ones that are loosely coupled and easy to refactor.

00:25:41.640 So how does the lack of circular dependencies affect our engines?

00:25:49.520 Reducing the number of necessary circular dependencies lets us go from a single Rails app where everything references everything else.

00:25:56.320 To a Rails app where things are all broken apart into engines.

00:26:03.160 If you have a circular dependency, then you should group those two things together.

00:26:09.760 If you have a lot of circular dependencies, you're going to have a much harder time breaking things out.

00:26:17.200 The other great thing about having engines and an acyclic dependency graph is that you can create a smart and fast test suite.

00:26:24.640 Imagine you're working within an architecture like this, and you're making changes to the admin engine.

00:26:32.640 When you run your tests, you actually don't need to run them for all the engines.

00:26:39.840 All you need to do is run the tests within the admin engine and the tests within Theer.

00:26:47.120 The Schuler engine depends upon the admin engine, so changes made in the admin engine could break things in Theer, but you don’t need to run tests for the social network or common engines.

00:26:54.600 We didn't make any changes to those engines or any engines they depend upon.

00:27:00.680 So you can write a smart test suite using something like Git to get what changed, which lists what files have changed since your last push.

00:27:08.480 You can determine what engines have changed and what engines need their tests run based on your dependency graph.

00:27:16.480 What we found is that normally, you only need to run a fraction of your tests.

00:27:23.560 We have our one-way dependencies to thank for that because, like I mentioned, the user engine, which is heavily depended on by the rest of the system, rarely changes.

00:27:30.200 For example, when we add posts to the system, the user doesn't actually need to change.

00:27:36.760 So you don't have to run the majority of the tests in your app.

00:27:44.960 How did using engines, domain APIs, and all these conventions work out for us?

00:27:51.440 Well, engines are a pain in the beginning. We had super low velocity for the first month and a half.

00:27:57.840 The client was actually getting worried, and we really couldn't bring on more developers.

00:28:04.560 We were still developing these patterns and best practices for engines in our application. Constant refactoring was a must.

00:28:11.480 At one point, we were spending about a third of our time just refactoring.

00:28:17.920 There were a few technical hurdles we had to overcome—the monkey patching of Rails being one of them.

00:28:25.520 But in the long term, engines definitely won out. At the end of those months, we saw no slowdown in development time.

00:28:34.480 There was no slowdown in the time it took to do fairly major refactorings.

00:28:40.440 Engines aged really well—oftentimes we'd finish working on an engine and ignore it for months at a time.

00:28:46.480 It makes for easy parallel development, so you have these little boxes that certain teams can work within.

00:28:55.320 You can talk in terms of which team will be working within which engine.

00:29:01.720 The potential for scaling is great—if you need to scale a certain area of your app, all you have to do is pull that engine out into its own app.

00:29:07.440 Lastly, the ability to write better tests was a big advantage. So overall, I give this architecture a thumbs up.

00:29:14.640 But it probably took me two months to really come around to it. I really hated it for the first month.

00:29:21.520 If this stuff seems interesting to you, or if you think it might help one of your current apps, give it a try.

00:29:27.120 Try creating a new engine for the next major feature you add, or try pulling out some of your models into their own engine.

00:29:34.720 Try avoiding circular dependencies and creating a service layer.

00:29:41.920 Most importantly, try to think outside of the Rails way of doing things.

00:29:48.960 As Cam said this morning, build the right thing the right way.

00:29:56.040 Start thinking in those terms.

00:30:00.000 Thanks! If you want to learn more about engines, my coworkers and I have been blogging about engines.

00:30:06.560 Here's a link. There's also a link to my SpeakerDeck. I'll post these specific slides in a few minutes.

00:30:14.000 That's it. Thank you!

00:30:20.800 Awesome, thank you very much, Ben! Do we have any questions?

00:30:26.440 We do have a whiteboard and a dry-erase marker, but we went low tech on that.

00:30:32.960 It was actually a good exercise in our process—anytime someone added a new engine, they would erase the whole thing and redraw it.

00:30:40.560 This made sure that everyone on the team knew how everything was interacting.

00:30:47.440 When you redrew it, you would look through the gem files and ensure there were no circular dependencies.

00:30:54.800 That’s the approach we took; we kind of took the process approach on that.

00:30:58.000 We've got another question over here.

00:31:04.560 Sure! I really like the idea of splitting out the tests into different engines.

00:31:11.400 Was there a test for the overall application to ensure everything tied together?

00:31:18.040 Yes, we did have some tests, and those tests lived at the highest level.

00:31:24.960 They lived in the wrapper app and tested to make sure that data was moving correctly between each of these engines.

00:31:31.800 There were a few high-level smoke tests, so to speak.

00:31:39.840 Thank you! Yep, here.

00:31:45.840 Thanks for the good talk! You mentioned at the beginning trying to split the app into engines early.

00:31:54.080 What are your general rules or advice regarding how to split the app into engines at the very beginning, because it feels like premature optimization sometimes?

00:32:02.080 That's a hard question—trying to decide where the seams are to break your app apart.

00:32:09.480 We started by looking at a high-level overview of the big areas of functionality or the users of the system.

00:32:16.200 That's how we began breaking things out.

00:32:23.880 I've also explored breaking things out at the lowest possible level.

00:32:29.680 For example, I created a couple of engines where the engine only contained a single model and a single database table.

00:32:36.800 It really just depends on your application and your team—the comfort level with it.

00:32:43.880 So it’s not really a good answer, but it’s the best I can give you—it depends.

00:32:50.000 I'm happy to talk more with you about it after.

00:32:54.560 I've got one over here.

00:32:57.600 Was each engine in its own codebase included as a gem, or how were you structured?

00:33:03.600 They were all under the same codebase—so we had a single Rails app, a single sorcery.

00:33:10.160 We had a subdirectory for engines next to the app directory or the lib directory.

00:33:16.960 That's where we had all of our engines underneath.

00:33:24.320 I was just wondering how you got your tests to trigger for a single engine on a commit?

00:33:31.160 Rails gives you a 'dummy app,' which is a little, full Rails app inside of your engine that requires that engine.

00:33:38.120 You boot up this little dummy app that requires the engine inside of it, and then you can run your tests just within that context.

00:33:45.720 That's basically how it works.

00:33:52.440 Okay, just, if you stop exposing ActiveRecord objects in your Rails application, don’t you find it hard to work with forms?

00:33:57.600 Even optimizations, like including relationships, is challenging?

00:34:04.160 It is harder to not use ActiveRecord; it does make things really easy.

00:34:10.760 It also makes your code really tangled, so it's a trade-off.

00:34:16.960 People who have written code before there was Rails know this is stuff we had to do.

00:34:24.560 On the other hand, there is a balance.

00:34:30.960 Things like form helpers are essential to getting everything to work well in your views.

00:34:38.160 You can get some of that using parts of Rails, so using Active Naming and Active Model, you can include them in the ruby objects.

00:34:48.160 It doesn’t expose all the ActiveRecord stuff, but it still gives you a lot of those helpers.

00:34:55.120 So it is a little bit of a balance.

00:35:02.720 Awesome, we've got time for one more question.

00:35:07.320 A comment and a question.

00:35:13.600 The wonderful thing about new frameworks is getting to discover the same problems again and invent the same solutions.

00:35:21.440 Large software systems have had to deal with these problems obviously for a long time.

00:35:29.440 What I wanted to ask was, were you using foreign keys? If so, how are you dealing with foreign key relationships between different engines?

00:35:37.120 Foreign keys can be interesting because you have tables living in different engines.

00:35:42.040 They don't necessarily know about each other, but there are two solutions.

00:35:49.640 If there's a need for performance, one option would be to merge them together.

00:35:57.760 We did that in one case where we need two tables to be performant, so we pulled those together.

00:36:05.440 The other option is to maintain two separate things and create something above it that depends on them.

00:36:12.560 That thing above it creates the join between the two.

00:36:18.960 It's a little bit of gray area that we toyed with.

00:36:26.600 Awesome, thank you very much, Ben. Everyone, a big round of applause for Ben!

RubyConf AU 2014