How Sprockets Works

EuRuKo 2016

00:00:04.520 And our next speaker is Rafael França. He comes from Brazil and he's one of the most vibrant Rails contributors, the most frequent Rails release manager, and also known as the Rails patch monster. He works at Shopify, and he'll tell you about Sprockets.

00:00:28.519 Hello everyone! I'm here to talk to you about how Sprockets works, but first, I want to know how many of you know how Sprockets really works. Please raise your hand if you know how it works. Oh, we have two, three... I think three people know how that works, and that does not include me because I don't know anything about it.

00:00:53.840 So yeah, that's not true. I'm not the original maintainer of the project. This Sprockets project was created by Josh Peek and Stenson from Basecamp. Originally, since last year, Josh left the project and no one in the Rails community knew how Sprockets works. So this talk is exactly my attempt to understand how Sprockets works and to show you in the community how it works originally.

00:01:29.680 As it was said, I'm Rafael França, and it's hard to say my last name. You can find me on GitHub as Rafael França and on Twitter as Rafael França. I'm a member of the Rails core team, and I work at Shopify. One thing that I really like about the Rails core team is that we are allowed to work on anything that we want to work on.

00:01:52.840 Some people like to do interesting and exciting things, such as implementing new features and making performance improvements. But for people to be able to do that, someone had to do the hard, boring job. I am the person that does that boring job.

00:02:08.360 So, I can say that I'm the Rails maintainer. As it was mentioned, I was the release manager for the last three years, and I'm always dealing with issue trackers, reporting, and reviewing pull requests—not just in the Rails repository itself, but in the Rails project, which has a lot of different components that most of you are already familiar with. There are components like Action View, Action Controller, and Active Record, but we also have things present in all layers of your web development framework.

00:02:52.879 At the process level, you have Spring; you have the MVC stack framework; you have things running in the browser that are part of the Rails project, like JavaScript and links; and you have the asset pipeline, which is what I'm going to talk to you about today. This is the agenda of my talk: First, I need to tell you why we need an asset pipeline and what that means. What are the responsibilities of the assets pipeline inside Rails? How does it work in a Rails application? And how can you extend it?

00:03:34.040 So first, why do we need an asset pipeline? Before Rails 3.1, Rails had no conventions on how to organize your assets. Apart from that, you knew how to organize a Rails application; you have folders like app/models, app/controllers, but all the assets at that time were put in the public folder. So there were no conventions back then on how to organize assets, and usually, you ended up with a lot of files where you didn’t even know if they were being used in your application.

00:04:04.360 Also, at that time, you had to make some trade-offs between code organization and performance, like whether you should use small contained files, which are better for modularity and maintenance of the code, but create more requests from the browser to the web server, which could negatively affect performance on the client side. Should I take this route or should I make fewer asset requests by maintaining one huge file that's hard to manage? This created issues with code legibility, documentation, and performance.

00:04:44.960 So to solve all these problems, we introduced the Rails asset pipeline, and I'm going to explain how it works right now in Rails. Now we have conventions for assets; all the assets live in the app/assets folder. You have folders for stylesheets, folders for JavaScript, and folders also for images, making it easier to know where each component of your client side is. You can also have assets inside the vendor directory, and the assets are compiled on the fly in development and need to be precompiled in production.

00:05:57.840 Apart from that, Sprockets also set good standards. You are able to cache your assets on the client side forever, and Sprockets will be able to do cache busting by including digest in the name of the assets. Now that you know how the asset pipeline works in Rails, I'm going to talk about the gems that are responsible for this behavior.

00:06:58.400 We have several gems that make this possible: Sprockets, Sprockets Rails, and Rails asset gems—the Sprockets gem is the principal gem of the setup. What it does is compile and serve assets. It is a simple process pipeline; you have a pipeline made up of different processors. These are the key components: processors, directives, environments, manifest, and pipelines.

00:08:01.840 I’m going to talk about one of each of those. The processor is the most important component of Sprockets. It is any callable object that accepts input and returns a hash of metadata. Here’s one simple example of what a processor inside Sprockets looks like: it's just a lambda that receives an input as a hash and returns another hash with the data and some metadata.

00:08:41.759 This processor is performing the job of removing all the semicolons at the end of your JavaScript files, as you don’t need semicolons in JavaScript. The input hash consists of information like the data of the assets itself, an environment instance of this processing environment, as well as metadata. You have to return another hash that simplifies to only the required information, which is the data itself, so you can provide necessary metadata like required assets, links to external assets, or source maps.

00:09:50.240 Sprockets comes with some built-in processors, like the most common are the JavaScript processor and the CoffeeScript processor. In version four of Sprockets, there’s also a Babel processor to compile ES6 files to JavaScript. There’s a special kind of processor called a bundle processor. It is a processor that concatenates files to individual files. This is how you register a processor in Sprockets, so I'm saying that to bundle all the assets together, I’m going to use the processor called bundle. The same thing goes for CSS.

00:10:49.720 What the bundle processor does is take a single file of assets and process all the URLs in the contents of these assets. Another kind of processor that you have is called transformers. They are simpler than general processors because they know how to convert a file from one format to another. For instance, we have the CoffeeScript processor that only transforms CoffeeScript into JavaScript.

00:12:00.000 The implementation of any processor follows the same structure as a callable object, so the implementation of the CoffeeScript processor looks like this: you have a call method that receives an input and also returns another hash. What this processor does is compiling the data read from the file system to JavaScript and returning that data to the next processor in the chain.

00:12:50.480 We also have compressors, which are a special type of bundle processor. The difference between compressors and general processors is that compressors have a special way to declare them and also how to use them. There are only two compressors: the JavaScript compressor and the CSS compressor. You have to specify which compressor you want to use when you are trying to compress JavaScript.

00:14:33.040 Directives are the most common things that you see in a Rails application. They are the special commands that declare the bundles. An example of a directive in use is where I have an application.js file that depends on jQuery and other user files while also requiring all the files in the same directory. This is how you define my application.js bundle.

00:15:30.440 To use that in your application, you have to register this bundle in the precompile configuration. This is the content of your config/environment/assets file, saying that for generating my application, I have those two bundles: application.js and application.css. You usually don't need to do that because this is the default for any Rails application.

00:17:04.720 So by Rails, we define two things on the precompile list, which is this strange lambda that tells you that any file that's not CSS or JS will be precompiled, like images, MP3s, and videos that you have in your asset folder. Also, files named 'application' are going to be precompiled. There's a problem with this approach because it's not easy to understand what's going to be precompiled and what's not, leading to many issues.

00:17:50.560 In Sprockets 3, we introduced a new kind of bundler called a manifest file that comes by default in any Rails 5 application. This file simplifies the precompile lambda and configuration to say in a more declarative way which assets are going to be precompiled. For instance, I need all the images inside the images folder, all JS files inside the JavaScript folder, and all CSS files inside the stylesheets folder. I can also ask for all the assets that are inside my application.

00:19:43.880 There are several directives by Sprockets. The most common one is the `require` directive, but you can also use `require_self`, `link`, `depend_on`, and other directives. Another important object in Sprockets is the environment object. The environment is the main object inside Sprockets because it is the one responsible for methods to retrieve assets, manipulate the load path, and register processors.

00:20:52.000 This is how we usually use the environment. I'm telling this code that I need to compile the application.js file, and this is how the JavaScript text works. I can also specify new transformers, like wanting to transform my SVG file to PNG using that transformer.

00:21:16.600 The environment always compiles the assets when you ask for them, so every single asset that you ask of the environment is going to be precompiled on the fly. That's a problem in production because you don't need to always precompile the same file. So to fix that, we introduced another object called the manifest, which is similar in name to the configuration but serves a different purpose.

00:22:40.000 The manifest is an object that logs all of the contents of the assets that are precompiled inside the directory and also handles caching, so you don't need to recompile every single time you access an asset. It works as a simple file that points out to fingerprinted versions. When you do this in your views, you are calling a JavaScript include tag with the application.

00:23:25.600 The manifest is responsible for retrieving the name of the file used as the source of the script. This file is essentially a JSON file that points the path of the asset to the fingerprinted version of that asset. Additionally, from the fingerprinted version, you can easily determine the logical path, type, and digest of that file, which is useful for cache invalidation. Every single time you modify an asset, Sprockets already knows how to handle that, so you don't need to delete the cache.

00:24:36.800 There are more components inside Sprockets that I won’t talk about here because they are sometimes too simple or irrelevant to this talk. These include MIME types, dependency resolvers, the S suffix bundle, and metadata headers, which are mostly utilized by extensions of Sprockets that you can build.

00:25:54.879 One such extension is the Sprockets Rails gem. As the name suggests, it integrates Sprockets into Rails applications because Sprockets are part of Rails and can also be used in any kind of application, even those built in Elixir.

00:27:14.400 What Sprockets Rails does is define helpers in your Rails models. For instance, these two helpers defined by Sprockets Rails will also configure the Sprockets environment using the Rails configurations. Another function of Sprockets Rails is to check the precompile list because previously, we faced many challenges when our application had problems with assets.

00:28:29.760 If you pointed to the wrong asset, like when you have a typo in the name of the asset, you would only see this problem in production when an asset didn’t get served because the name was incorrect. However, now you can see a clear error page stating you're trying to serve an asset that's not in the precompile list.

00:29:50.000 Another change in the Sprockets setup is the Sprockets Rails gem, which integrates Sprockets processors with Rails applications. It defines generators so that when you generate a Rails application, the application.css file will be generated as well. This gem also creates importers that know how to handle glob patterns and ERB because the Sprockets processor by default doesn’t handle that—it's only made possible by the Sprockets Rails gem.

00:30:37.680 Speaking of ERB, we have another gem that allows you to run JavaScript code inside your Ruby process. You might ask why you'd want to do that. One useful aspect is that it allows you to use the JavaScript environment available on your machine. For instance, if you're on a Windows machine, it will use the Microsoft Windows Script Host; if you have Node.js on your machine, it will use that.

00:31:50.000 This is essential because if we want to compile CoffeeScript, we could have written our own CoffeeScript compiler in Ruby, but that would be inefficient as it would duplicate existing work. Instead, we use the same compiler that JavaScript libraries utilize, which is a CoffeeScript compiler. This structure allows us to compile the same CoffeeScript inside the Ruby process.

00:32:47.760 In summary, Sprockets is used in many Rails applications out there. It is included in every single new application generated. However, as we find ourselves here today, many users don’t realize Sprockets exists or understand how it works. Therefore, it’s important for everyone to understand their tools and document their understanding. This is what I am doing here today.