Creating Mountable Engines

Talks

Patrick Peak

1 talk

#rails-engine

#content-management-system-cms

Creating Mountable Engines

by Patrick Peak

The video titled "Creating Mountable Engines," presented by Patrick Peak at Rails Conf 2013, delves into the concept of engines in Ruby on Rails, particularly following the significant changes introduced in Rails 4.0 that rendered plugins obsolete. The discussion begins with an audience interactive segment regarding their familiarity with engines, which sets the stage for a detailed exploration of why engines matter and how developers can create and leverage them effectively.

Key Points Discussed:

- Understanding Engines: Engines are analogous to mini Rails applications that can be packaged as gems and added to other Rails applications, allowing for easier code reuse and modularization.

- Problems with Duplication: The speaker emphasizes that duplication across similar applications is common, leading to challenges in maintenance and upgrades. Engines provide a way to encapsulate common features, reducing duplicative coding.

- Creating an Engine: Peak walks through the basics of building an engine, using a sample name "RubyPress" for a hypothetical content management system. He covers the command to generate an engine, the structure of the generated code, and key components like controllers, routing, and migrations.

- Configuration and Middleware: Each engine can define its own middleware and initializers, allowing for customization and enhancements that integrate seamlessly into the Rails ecosystem.

- Asset Pipeline: With Rails' Asset Pipeline, engines can manage their assets like JavaScript and CSS, making it easier to package view-related code without manual copying of files.

- Collaboration of Engines: The talk highlights the challenges of making different engines work together, primarily due to conflicting dependencies, varying security models, and mismatched user interfaces. For engines to coexist nicely, the examples encourage creating common conventions and robust dependency management practices.

Significant Examples:

- The discussion references well-known engines such as Spree for e-commerce and Devise for authentication to illustrate practical implementations.

- The talk also cites the common practice in the BrowserCMS project demonstrating how teams can utilize engines effectively in a real-world context.

Conclusions:

The speaker concludes that when properly designed, engines can enhance code reusability, streamline the development process, and improve maintainability in Rails applications. Peak emphasizes that as the community matures in its use of engines, it is essential to develop best practices for dependencies and security to foster a healthier ecosystem of Rails engines.

00:00:16.480 All right, good afternoon everyone. It's barely afternoon, I guess.

00:00:21.680 So we're going to get started now. My name is Patrick Peak, and today I'm going to be talking to you about engines.

00:00:28.080 I'm always curious to see where folks are coming at this topic from, so a quick show of hands: who here has written their own engine? Go ahead and raise your hand.

00:00:39.920 All right. Who here has used an engine written by somebody else on a project? All right, a few more hands. And who here has never heard of engines and is here to figure out what they are?

00:00:46.719 All right, that's right. You guys are honest. That's great! Hopefully, there will be a little bit in this talk for everyone in all those groups.

00:00:59.840 A couple of other points: most of the code samples that I'm using today are written for Rails 4. However, not much has changed between Rails 3.2 and Rails 4, so you can use these samples pretty much as they are.

00:01:10.880 Hopefully, I'll have time for questions at the end, and I'll be hanging out here after lunch to answer them as well.

00:01:17.040 A little about me: I'm the Chief Technology Officer for Browser Media. We're an interactive web agency in Washington, DC, where we build websites for clients.

00:01:24.159 I'm also the project lead for BrowserCMS, which is an open-source content management system built on Rails that we frequently use to build those sites. BrowserCMS, as it happens, is an engine. Engines are exactly the sort of thing that make building tools like content management systems easier.

00:01:37.040 I want to start first with a problem. As our friend from Dilbert illustrates, sometimes repeating yourself is a little too easy when you're writing a Rails application.

00:01:49.280 You’re probably creating duplication even if your current application is completely DRY. If you look at any two apps you've written, there’s probably a good amount of duplication between those two apps.

00:02:00.000 As we should know in programming, duplication is the root of all evil. In Rails projects, we have historically had a number of ways to avoid repeating ourselves. The first option is to create a library if you need to reuse some code, especially if it's not Rails-specific.

00:02:26.560 You can make a library of it and distribute it as a gem, which is most appropriate if the code you're working with does not have any user interface components, like an XML parser or a testing library.

00:02:45.680 But what if you want something that's Rails-specific? For a long time, the go-to solution was plugins. Plugins could be added directly into projects into the vendor plugins directory, and later they were designed to be packaged as gems.

00:03:02.319 However, they were limited in what they could do. As of Rails 4, though, plugins, as you know them, are dead. And thank God for that! It's about time—if you have code in your projects under Rails 3, it will pester you insistently about deleting that.

00:03:21.760 So what other options do you have? Generators are another pretty common solution that folks will use. Rails has a few, and typically plugins would add some more of their own. These feel super productive, until the point where there's a bug in your generated code.

00:03:39.280 The more generated code you have in your project, the more costly that upgrade process becomes. For example, who here has ever used Restful Authentication on a project?

00:03:51.920 Keep your hand up if you ever went back and upgraded that code to a newer version of Restful Authentication. So, like one or two brave souls in the room. That illustrates the problem with generators: once you have all this code in your application, it's yours. You own it; it's not living in a gem elsewhere that you can just update.

00:04:09.760 So, generators are not going to be good enough. So what to do? As you may have guessed from the title of this talk, engines are our solution. The goal for today is to take a little bit of time and go through how we can better leverage these and take advantage of them.

00:04:20.880 Here’s the agenda I want to cover today: I'm going to talk about what engines are and why I care about them. We’re going to walk through a sample of building an engine, look at some interesting things you can do with engines, and finally discuss how we can get engines to work together.

00:04:44.720 Why do I care about engines? I've been working with content management systems for a long time. About 2003, we started building sites for clients, and there were no viable open-source CMS options available at the time. Like any eager young programmer, I figured I could knock out a CMS in a weekend. Totally easy, right? Ten years later, I'm still working on the same project.

00:05:27.200 The first version of BrowserCMS was written in Java, which had a big set of limitations on component reusability. We rewrote it in 2008 on Rails 2, around the same time that vendor plugins were falling out of favor.

00:05:52.160 We wrote it as a gem, but we still had to monkey patch Rails extensively. Every time you upgrade Rails, the entire thing would break.

00:06:09.759 Then in Rails 3, engines were introduced into the core for the first time. This was incredibly beneficial. The upgrade process was painful, as anyone who has done a Rails 2 to Rails 3 upgrade can attest, but it had a nice payoff.

00:06:30.160 When Rails 3 came out, they did a complete overhaul of engines, adding features that let us isolate our engines from application code and take advantage of features like the asset pipeline.

00:06:57.440 In the next release we’ll upgrade to 4.0, and I’m confident this will be seamless because most of the stuff now is dependent on these engine APIs, which are stable public APIs.

00:07:11.520 So what are engines? The short answer is, it’s a bit like a mini Rails application that you can add to other applications. Since it's packaged as a gem, you can easily reuse engines between projects.

00:07:25.440 The longer answer is: suppose you have a giant Rails application. I'm sure nobody other than me has one of those right now! Your business domain is what your client is paying you to build, but you’ve got a bunch of other stuff in there too—maybe you're selling stuff, managing users, or doing some blogging.

00:07:59.840 All of those might be useful for a second project, and you don’t want to be copying that blogging code out of your first app into your second app just to avoid rewriting it. With engines, we can split the behavior of this massive app into separate gems.

00:08:18.640 Instead of this one massive app, we can have a smaller focused app that concentrates on the business domain, and then just have it depend on and configure these external engines.

00:08:39.680 Now we have another app, and we want blogging; we can just have it also use those engines. Here are some popular engines that folks may be familiar with, showcasing what you can do with these tools.

00:09:10.160 The first is Spree, an e-commerce shopping cart solution built as an engine. It has all the standard stuff you would expect from a store, like product and order management. I don’t personally have much experience with it, but the Spree team is primarily based out of DC, and I know who to pester with questions.

00:09:51.360 Another popular engine is Devise, which is perhaps the most popular authentication solution for Rails right now. It provides a flexible way to handle user management and authentication. It’s very pluggable, allowing everything from email configuration to remember-me functionality.

00:10:30.640 A common use for engines is general-purpose admin dashboards, like Rails Admin. These allow for general exploration around the models in your application, generating a stock UI and dashboard that enables management of models. This can be a low-cost way to kickstart an admin area, similar to how Django provides an auto-generated management interface.

00:11:23.439 Now, let’s talk about building an engine. We'll work through an example of how to create your own engine and demonstrate some interesting functionalities within it. Suppose we decide what the world really needs is another content management system. It probably doesn't, but let's use that for our example.

00:12:12.400 To make it easier for other developers to add this to their projects, we're going to build it as an engine. I'm also super original with my project name and we'll call it Rubypress. Here is the command to generate a new mountable engine. And yes, I know that the command is called 'plugin,' and you may have heard somewhere that plugins are dead, but just trust me that’s what the command is actually called.

00:12:43.840 A quick digression: There are actually two sorts of engines: full and mountable. I'm not really going to cover full engines for today's discussion, as they don’t namespace things—making them problematic for gem distribution—but I'm happy to discuss that afterward.

00:13:11.440 When we run this command, we get a pile of code that looks quite similar to what you would get when creating a new Rails application. A couple of key points: you’re going to get a gemspec because we’re going to distribute this as a gem, an app directory like any Rails application, and a dummy Rails application placed in the test directory, which is automatically set up to use our engine.

00:13:55.680 The key takeaway here is that engines can’t live on their own; they need a Rails application to host and test them. The plugin command will generate that into your engine for you. So, for any content management system, we need a way to manage pages.

00:14:26.640 We'll add a resource to our engine, using the Rails generator command to give it a few fields. I’m using Rails 3.2 shorthand to implicitly get 'name,' 'path,' and 'content' as strings.

00:14:58.000 If you look at the controller file that was generated, in a normal Rails application that’s what you’d see. Since we’re working with a mountable engine, you’ll notice two changes: the controller is in a subdirectory called Rubypress and is namespaced under Rubypress.

00:15:25.760 Why is that? Because we created a mountable engine, everything gets isolated under a namespace. The 'engine.rb' file is the central configuration point for engines, and since we created an engine called Rubypress, all our classes fall under that namespace.

00:15:50.720 The engine.rb file will also automatically set up routes and generators to include that namespace in any generated files. Now, let’s talk about routing.

00:16:20.000 Since engines have their own controllers, you may wonder how to avoid conflicts. Your app’s routes will avoid conflicting with the engine’s routes. Rails solves this conflict by essentially having the engine be its own bucket of controllers.

00:16:51.440 Routes from one engine don’t mix with each other or the main application. The routes file in our engine project is similar to your project’s routes file, with the exception that the routes are being added to the engine class, rather than the application object.

00:17:27.040 How would we access our newly defined controller? Here’s the UI for our application. Assuming we add a bit of standard CRUD behavior to our pages controller, we would access the index action using the appropriate URL. The key thing to note is that we’ve nested this with a '/rubypress' in the URL.

00:17:53.680 Where does that come from? It’s defined in the application routes file, where we mount the engine using the mount method and specify the path for the engine.

00:18:19.760 You can also see we have another root route defined here, and the takeaway is that each engine and application can have their own separate route that’s accessible.

00:18:54.080 Once the routes are defined, let’s discuss linking to them. Both the engine and the main application have their own helper that hold collections to the routes for that engine. The main app routes will allow you to access the main application’s routes.

00:19:27.280 If you want to access the engine’s routes, you use this 'rubypress' helper. Accessing the root URL resolves to '/rubypress' — and then, for standard CRUD access, we can pass a page object to the 'pages' path in the Rubypress engine.

00:20:03.760 There’s a bit of a hassle with prefixing all our routes this way; fortunately, Rails can implicitly decide which set of routes you want based on the file you're in.

00:20:16.760 In the layouts file from our engine, we can create a menu that accesses both main application and engine routes without requiring any prefixes for engine routes.

00:20:44.800 If we want to access the main application, we do need to specify the helper method. If we flip this around to look within the main application itself, our application routes don't need a prefix, but we do need them for accessing Rubypress paths.

00:21:10.760 Now, where does that 'rubypress' helper come from? True to Rails’ convention over configuration approach, Rails automatically generates this based on the engine's name. If you want to change this name, you can do so in two places: in your project’s routes file or in the engine itself.

00:21:49.520 Let’s talk about migrations next. This is one of the significant additions introduced in Rails 3.1. Before then, migration management was fairly non-standard. This is the migration file from the resource we generated earlier, and you can see the namespacing showing up in two places: in the name of the file and in the name of the table.

00:22:58.400 The great thing is that the engine will namespace all of your tables, which reduces the chance for conflicts. Since this migration exists in the gem, the next step is to get it out of the gem into your project.

00:23:45.760 Engines will provide a rake task, and when you invoke it, it copies the migration files from the engine into your project. It won't override migrations that already exist with the same name.

00:24:09.440 This is beneficial when needing to upgrade and release a new version of your engine. Users can simply rerun the task, receiving only the new migrations without disrupting their old setup.

00:24:28.640 After running the command, this is the name of the migration file you'll get, with a timestamp reflecting when you added it to your application. This preserves the order of your migrations.

00:24:48.160 Additionally, it suffixes the name of the engine to the migration file, making it easy to see which migrations are from which engine. If you have several engines and want all migrations at once, there’s a rake task that will copy everything across.

00:25:08.320 Next, let’s talk about configuration. In a traditional Rails application, you might find configuration in either the environment file or in an initializer. You can do the same in an engine, allowing users to just require the gem and only change configurations if they don’t like the defaults.

00:25:47.760 Most of the standard Rails application configuration behavior is available from within an engine. The engine.rb file is very similar to the application.rb file. I encourage you to look at the Rails configuration guide and experiment with what you can do.

00:26:21.440 One thing you may want to do is add initializers. Engines can define their initializers that hook into the Rails boot-up sequence. Rails defines a series of hooks during startup where we can specify our initializers.

00:27:04.160 For instance, we might want to add additional DSL to the routing mapper. Here’s how we might define a new method in our engine project, allowing us to easily mount Rubypress at a reasonable default and adding another custom route.

00:27:39.760 You will need to namespace the controller name because this method gets invoked within the context of your application routes file. This example is somewhat trivial but demonstrates how to streamline adding things to a project.

00:28:21.280 You can use the same technique to augment any of the core Rails pieces, like ActiveRecord or ActiveController. Most of the core components in Rails are now simply Rails engines, which defines the configuration of how things fit together.

00:28:58.080 Engines can also define their middleware, so we could configure Rack Cache since we're a CMS that needs to cache pages. This middleware will automatically process any routes that get passed to our engine because engines also act as their own Rack endpoint.

00:29:35.040 With engines, you can pre-configure the generators a project will use. For example, if we want our spec files to be used everywhere instead of test unit whenever we run generators, we can simply include an engine that changes the project defaults.

00:30:13.920 Providing configuration options for engines greatly increases usability, as users will often want to make changes after adding your engine. When offering configuration options, it’s vital to set reasonable defaults.

00:30:51.680 For instance, we can create a configuration namespace for the Rubypress engine, set up reasonable defaults, and allow users to override them in their application’s environment files.

00:31:26.640 Now that we’ve created our engine, how do we get people to use it? The best way to facilitate this is by making it easy for new users to get started. At the very least, write some documentation that informs users where to start.

00:32:00.000 Update the README file to contain useful information, rather than just the default content it generates. GitHub will display the README prominently, so it’s a crucial place for users.

00:32:36.640 Document the exact steps needed to install your project. By listing the steps, you can see if there might be too many, which is a cue for simplification. Ideally, we want the installation process to be as seamless as possible.

00:33:00.320 By leveraging generators, engines can bundle installation scripts that automate more of the setup process. This gives us a powerful tool to streamline user onboarding.

00:33:25.440 Now, let’s talk about the asset pipeline. One of the major changes introduced in Rails 3.1 was the asset pipeline, which many developers are familiar with today. The asset pipeline allows us to organize and compress JavaScript, HTML, CSS, and images.

00:34:01.040 Importantly for our purposes, the asset pipeline allows us to package view-related code in the engine in a logical manner. Previously, managing view-related assets was tedious.

00:34:23.280 For example, if you had a gem with complex JavaScript code and HTML, such as CKEditor, you would previously need to write a Rake task to copy those files into the public directory of your project.

00:34:56.000 If changes needed to be made, it would be your responsibility to manage those files. With the asset pipeline, engines can package and provide all view-related code within the engine, avoiding the need for duplication.

00:35:29.920 For instance, the CKEditor Rails gem operates solely as an engine with HTML, JavaScript, and CSS. No Ruby code is needed, allowing for more straightforward management of packages.

00:36:09.280 This pattern of utilizing pure front-end engines makes it easier for developers to utilize various reusable components without worrying about duplicating code.

00:36:27.200 As someone who prefers to delete unnecessary code, the asset pipeline has been a blessing, enabling me to clean up a significant amount of redundant files from the BrowserCMS project.

00:37:00.000 Now let’s discuss how engines can and can’t work together. There are several challenges, but because engines are a relatively new concept, we have yet to fully develop optimal patterns for this integration.

00:37:29.440 The most straightforward use cases arise when you have a set of engines designed to work together. For example, using BrowserCMS alongside additional engines for news or blogging makes perfect sense.

00:38:01.920 However, it’s not as simple as just dropping in an unrelated engine, such as one from Radiant, since they haven’t been designed with compatibility in mind.

00:38:32.640 One advantage of establishing a common stack is the ability to create conventions that related engines can follow. This yields a cohesive experience for both developers and users.

00:39:03.120 For instance, in BrowserCMS, all engines support a unified installer script, allowing us to quickly add engines to our projects with a single command.

00:39:36.320 However, several challenges arise when multiple engines are used together due to potential dependency conflicts.

00:39:59.760 Consider engines like Spree and Rails Admin that may have dependencies on different versions of libraries like Devise. Resolving these dependency conflicts may require forking gems or abandoning one of the engines.

00:40:22.520 When writing an engine, you can create a better experience by minimizing dependencies, following rational versioning, and being liberal with what your engine accepts as a dependency.

00:40:46.160 Next, there’s no common security model in Rails. Any non-trivial engine will likely need some form of authentication. For example, BrowserCMS uses Restful Authentication, while Spree has its own authentication approach.

00:41:08.800 The lack of a standard method complicates developing engines meant to work together, as there is no unifying authentication solution in our community. With a fragmented landscape, finding a common denominator is essential.

00:41:37.680 Lastly, mismatched user interfaces can cause other issues when adopting multiple engines. It’s rare for two UIs from different engines to be designed to work together seamlessly.

00:42:05.680 Two examples are Active Admin and Rails Admin, which are intended for similar purposes but offer different user interfaces that may not integrate well.

00:42:36.760 You can approach this by designing engines with UI-free functionality, leaving it to developers to build their UI according to your APIs, or by embracing a single UI paradigm.

00:43:06.640 To recap what I hope you learned from today’s talk, engines are the future, and they are powerful tools for Rails development.

00:43:36.480 When properly executed, engines make code reuse easy and can reduce errors stemming from duplication. Thank you for your time, and that’s the end of the talk.

RailsConf 2013