How Sprockets Works

00:00:04.520 and our next speaker is Rafael franka he comes from Brazil and he's one of the

00:00:10.040 most vibrant rails contributors the most frequent rails release manager and also

00:00:16.920 known as the rails patch monster so he works at Shopify and he'll tell you

00:00:22.760 about

00:00:28.519 sprockets hello every everyone I'm here to talk to you about how Pro works but

00:00:35.040 first I want to know how many of you knows how this SP really works so please

00:00:41.280 raise your hand if you know how that works oh we have two three I think three

00:00:47.120 people knows how that works that does not include me because I don't know anything about it

00:00:53.840 so yeah that's not true uh but I not the original maintainer of the project

00:01:00.559 so this pro project was created by Josh P in St Stenson from base camp

00:01:07.640 originally and since the last year Josh left the project and no one in the r

00:01:15.000 called knew how his Pro works so this talk is exactly my try to understand how

00:01:22.840 his Pro works and to show you in the community how it works originally so

00:01:29.680 like like it was said I'm Rafael fra is hard to say my last name

00:01:37.840 so you can find me in GitHub as Rafael franka and Rafael franka also in Twitter

00:01:45.200 I'm member of the Ros Scot team and I work at shop so one thing that I really like

00:01:52.840 about the Ros Scot team is that we are allowed to work you anything that we

00:02:00.479 want to work so some people like to do interesting and exciting things like

00:02:08.360 implementing new Fe features and also make performance improvements but to people be able to do

00:02:17.200 that someone had to do the hard Bing Shob so I the person that do that Bing

00:02:24.080 job so I can say that I'm the ra Mainer like it was said I

00:02:30.400 was the relas manager of the last three years and I'm always dealing with issue

00:02:37.959 trackers and Reporting and review Pro requests and not just in the Ros story

00:02:45.440 itself but the r project has a lot of different components that you can most

00:02:52.879 of you already are familiar with then like action view action controller activ

00:02:58.120 records but we also have things R is present in all the

00:03:04.000 layers of your web development framework like you

00:03:09.440 have things at the process level like spring we you have the MVC stack

00:03:15.480 framework you have things running the browser that are part of the Rails

00:03:20.640 project like JJs in links and you have the asset pipeline that is what I'm

00:03:27.239 going to talk to you today so this is is the agenda of this talk

00:03:34.040 first I need to to tell you why do we need a asset Piper line and what that

00:03:41.760 means what are the J responsible for the asset Piper line inside the rails how it

00:03:47.480 works in a ra application and how to extend it so first why do we need asset

00:03:55.519 pipeline before the ra 3.1 we raos had no conventions how to organize

00:04:04.360 your assets apart to that you knew how to organize a real application you have

00:04:10.400 folds like app models app controllers but all the Assets in that time were put

00:04:16.880 in the public folder so there were were no convention in that

00:04:22.720 time how to organize assets and usually you end up with a lot of files that you

00:04:28.800 don't know even if they were be used or not in your

00:04:33.840 application and also in that time you had to do some tradeoffs between code

00:04:38.919 organization performance like should I use small s

00:04:44.160 contained files that's better for modularity and maintenance of the code

00:04:51.360 but it's also create more request to the from the browser to the web server so

00:04:58.520 that could C better wor performance in your client side so should I take this way or should

00:05:07.639 I make few asset requests like maintaining one huse file that is hard

00:05:14.960 to maintain and also do he use components all the three off that where

00:05:21.600 common that is was should I write legible code with dtive name and code

00:05:28.479 documentation or should I send a few bytes to the users so the connection

00:05:35.280 very faster in you can get your page faster and also in that time we have you

00:05:43.440 had new technologies being used like CP SAS and recently

00:05:50.360 es6 so to solve all those problems we introduces the rail assess

00:05:57.840 Pipeline and how it's working right now in raos so

00:06:04.039 now we have conventions for the assets all the assets live in the app assets folder so you have folders for St shets

00:06:10.960 you have folders for JavaScript you have folders also for images so is easier to

00:06:16.039 know where each component of your client side is present and you can also have

00:06:24.360 assets inside the Le in Vendor and the assets are compiled on

00:06:30.160 the fly in development and need to be pre compiled in production apart from that is pro also

00:06:37.720 set good standards like you are able to cash out your

00:06:43.840 assets in the client sign for forever and his pockets will be able to do cash

00:06:52.160 busting setting uh digest in the name of the assets

00:06:58.400 itself now that you already know how the assas pipeline Works in rails I going to

00:07:05.360 talk about the Gen that are responsible for this Behavior so we have several chains that make this

00:07:12.160 possible sprockets sprockets rails size rails isgs in C

00:07:18.120 rails the spr Gen is the principal gen of the setup what it does is compile and

00:07:25.639 service assets and it's as it is a simple process pipeline like

00:07:34.080 you have a pipeline made of different processors these Pro key components are

00:07:41.759 processors to S components directives environment manifest and pipelines are

00:07:47.840 going to talk about one of each of those so the processor is the most

00:07:55.680 important component of sprockets it is any cable object that

00:08:01.840 accepts input and returns a hash of metadata so this is one of one simple

00:08:09.840 example of what is a processor inside sprockets is just a Lambda that receives

00:08:17.080 input is a hash and ret return another hash with the data and some

00:08:24.159 metadata so what this processor is doing is removing all the sem colons in the

00:08:30.319 end of your JavaScript files because you don't need semicolons in

00:08:36.080 JavaScript so the input hash is consisted of those

00:08:43.640 information like you have the data of the assets itself you have the environment that is a instance of this

00:08:50.720 proess environment you have the cache thei The Source path load path name

00:08:57.000 content type and metadata and you have to return another hash that

00:09:03.959 is simpler than this one the only required information the data itself so

00:09:10.079 you have to return the data that is going to be used by the next processor

00:09:15.600 in the Shain and you can also provide some metadata like all the required

00:09:21.200 assets that you need to build that asset links that you have to external assets

00:09:27.640 other dependence the Source maps of the assets and also the M type this Pro come with some bu

00:09:37.519 processors like the most common are the S processor and the C processor and the

00:09:45.040 verion four of sprockets also have a bable processor to compile AC script six

00:09:51.560 files to JavaScript one special kind of processor

00:09:56.680 is called bundle processor it is a processor that runs concatenated files R

00:10:05.000 the individual files this is how you

00:10:10.320 register a processor in sprockets so I'm saying that to bundle all the assets

00:10:16.720 together I'm going to use the processor called bundle the same thing for

00:10:23.200 CSS what the Bund bundle process does is to take a single file of assets it

00:10:29.839 prends all the URL that's uh to the contents of these

00:10:36.160 assets I'm going to explain that later another kind of processor that you

00:10:41.720 have are the Transformers they are simpler than the general processors

00:10:46.839 because they know how to convert one file from one format to another format

00:10:53.639 like we have the CPT processor that only transform cof scrip in Javas

00:11:00.480 and the implementation of any processor is the same is any colable object so the implementation of the C processor is

00:11:07.680 something like this you have a call method that receive a input and also

00:11:13.839 return another hash and what is important in this implementation is that

00:11:19.399 what this processor does is actually compiling the data that was read from

00:11:26.240 the file system to JavaScript and returning that data later to the next

00:11:31.519 processor in the Shain we have also have the compressors

00:11:36.680 and the compressors are a special type of bundle processor the difference

00:11:42.800 between compressors in any kind of processor is that it has a special way to to declare

00:11:52.600 them and also to use like this is thei compressor that take inal code and

00:11:59.560 compress in minif Fes to create few bytes when you are saving and to use uh

00:12:07.399 compressor you have a special API that you need to specify for each type of you we only

00:12:15.360 have two compressors the GS compressor and CSS compressors so you have to

00:12:20.480 specify which compressor you want to use when you are trying to compress

00:12:25.880 JavaScript for example also has

00:12:31.000 directives those are the most common things that you see in ra application

00:12:37.440 like directives are the those special commands that decare the Bund in

00:12:43.839 theis this is the example of the directive is being used I have a application GS file

00:12:52.600 that depends on you carry you carry Y and also in the user files and also ire

00:12:59.480 all the files in the same tree so this is the definition of my bundle my application yes

00:13:07.000 bundle and to use that in the application you have to register this

00:13:12.519 bundler in the pre compile configuration this is the content of

00:13:19.240 your config envirment asset file and this is

00:13:25.760 saying that to generate my application I have those two bundles the application

00:13:31.399 GS the application SS you usually don't need to do that because this is the defa

00:13:37.240 of any Ros application so by the raos Define three two thingss on the pr

00:13:44.600 compile list that is this strange Lambda

00:13:50.160 that is saying that any file that's not CSS or GS is going to be back compiled

00:13:56.000 like image and mp2 and videos that you have in your asset

00:14:02.959 folder and also in file that's name application is going to be precompiled

00:14:09.639 there is a problem with this approach because it's not easy to understand what's going to PR compile Don not and a

00:14:16.759 lot of people had problem with this setup so in spet 3 we introduced a new

00:14:24.079 kind of bundler that is called manif file that is coming by the in any r five

00:14:33.040 application so this file is a simplification of that PR compile Lambda

00:14:39.440 and configuration that is basically saying in a more declarative way which

00:14:45.800 assets are going to be PR compiled so I'm saying that I need all the images

00:14:52.279 inside the image fold that I need all the GS file inside JavaScript all the

00:14:57.360 CSS file inside sty sheets and I also can say that I want all the assets that

00:15:03.839 are inside my machine there are several directives by

00:15:09.839 theowing in sprockets the most common one is the required D directive but you

00:15:16.240 can also use require self link depend on and depend on

00:15:24.160 asset another object that is important in spets is the environment and the

00:15:30.440 environment is the main object inside this spets because it is exactly what has methods

00:15:39.040 to reive in s assets to manipulate the load path into registering

00:15:47.199 processors so this is how we usually use our environment I'm telling this code

00:15:54.399 that I need to compile the application GS file the this is how the javascrip

00:16:02.319 Tex works and I can also specify new Transformers like I want to transform my

00:16:09.079 SVG file to P using that Transformer so the environment always

00:16:15.600 compiles the assets when you ask to so every single asset that you ask to the

00:16:21.399 environment is going to be pre compil the fly that's a problem with production because you don't need to always

00:16:27.639 precompile the same file so to fix that we introduced another object that's

00:16:34.160 called manifest is the same name of the configuration but different thing and

00:16:41.399 the Manifest is a object that logs all the contents of the assets that are

00:16:47.560 precompiled inside the directory and also have a caching so you

00:16:53.279 don't need to promile every single time you access asset

00:16:59.000 it's a really simple file that points aspf to finger pint versions like when

00:17:05.160 you do this in your views you are calling JavaScript include tag with

00:17:10.319 application the Manifest is responsible to get that name of the file that is

00:17:19.679 used as the source of the script so that file is basically a j file that points

00:17:27.400 the the path of the asset to the fringed

00:17:33.200 version of that asset and there is also the another way that from the finger pit

00:17:39.840 version you know what is The Logical path the a type and Das of that

00:17:45.880 file this is us it to to cash inv validation so every single time that you

00:17:51.600 change asset PR again SP already know how to do that so you don't need to

00:17:57.799 delete the cash at all there are more components inside the

00:18:03.159 spets that I'm not going to talk about because they are also simple or not

00:18:08.559 relevant to this stock like M types dependenc resolvers the S suffix bundle

00:18:14.799 metadata huers they are mostly used by extensions of sprockets that you can

00:18:21.600 make in like gen and another gen that I going to talk

00:18:29.400 about this Pro ra gen like the name can point to you it's a way to integrate

00:18:36.559 sprockets to raos application because sprockets are part of being used in raos can be used in any kind of application

00:18:44.480 even if you just agree in Elixir

00:18:49.640 application and what this SPX R does is Define helpers in your rubby models like

00:18:56.880 those two helpers are def fin by spus raos and it also configures the spus

00:19:03.880 environment using the raos configurations another thing that SP

00:19:08.919 raos now does is check the pre compile list because before we had the check it

00:19:15.559 was really hard to understand when your application have problems with your

00:19:20.840 assets so if you were pointed to wrong asset like you have a typo in the name

00:19:27.480 of the asset you would only see this problem in production when I asset was not going to

00:19:35.080 be served because the name is wrong but now you can see this beautiful EO page

00:19:41.360 that say you're trying to serve asset that's not in the pre compile list

00:19:48.039 please fix that by adding this file to the pre compile list or actually fix

00:19:53.799 your typo the another change that you have in this Pro setup is the size rails and it

00:20:01.600 does the same things that this Pro ra does that integrating the size processors with the raos

00:20:08.400 application it generates it defines the generators like when you generate discal

00:20:14.720 fold the application css. SSS file is going to be generated by

00:20:21.400 D it's also create import that knows how to handle Globs and erbs because the

00:20:28.640 size processor by Thea doesn't know anything about Erb or even glob path so

00:20:36.360 this is only possible if you are using the size R because the first line in the

00:20:43.480 glob the globe imported is is not building the size itself and the second

00:20:50.200 one the IB processor needs to be also declared in the Gen and it l it's

00:20:58.080 configur this processor the exgs Gen is a gen that was made to allow

00:21:06.880 you to run jav code inside your Ruby process you may be asking why should I

00:21:14.320 want to do that yeah but one of the things that

00:21:19.360 this is useful is that it's uses the JavaScript environment that is available in your

00:21:26.000 machine so if you have uh Windows machine is going to use the

00:21:31.559 Microsoft Windows script host if you have no GS in

00:21:37.080 your machine is going to use the no GS and there are a lot several options of

00:21:43.640 environments for japt and we need that because for instance we want this spets

00:21:51.559 that is a h programing to compile C script we could write our own C

00:21:59.000 compiling Ruby but that would be at least inefficient because you would be

00:22:05.840 duplicate work so we actually use the same compiler that in JavaScript library

00:22:12.000 use that is the C compile and this gen is what make

00:22:17.679 possible to us to run the same C compile inside the rubby

00:22:24.000 process so the is used by the CP to comp C script and many others use that we

00:22:31.400 have and speaking C script we have also the cscript ray chain that only configur

00:22:37.720 the generators so if you don't need generators you don't need the gener

00:22:43.039 all it's also defines a template Handler so you can serve C script files inside

00:22:52.000 your controller and so how the assets are generated in

00:22:59.960 development like I explained all the ches but how they work together in my R

00:23:06.720 application so say that you have a JavaScript include tag in your yourb

00:23:12.080 file your view and like I showed before what this is going to do in development

00:23:19.279 and production is generate a script tag with a source that points to uh where we

00:23:27.679 that is inside his proct itself so your Brer is going to do a get request in

00:23:35.120 that URL and that request is going to reach

00:23:41.880 the rail server and the sprocket ra we use the sprocket pipeline called the bug

00:23:50.400 to serve that asset that it was asked the the bug pip line was reged

00:23:58.000 like this is just array of processors and the only processor that you have in

00:24:04.240 that pipeline is the source map command processor this processor will generate a

00:24:10.440 asset bundler and in the end add a source map comand it is a comment like

00:24:16.679 this that is only pointing to the source map of that

00:24:21.720 file so to build the content of this F before adding the comment

00:24:28.480 is spets we use the default Pipeline and the default pipeline is

00:24:35.039 declared like this it's we take the type the file type of the file that is going

00:24:42.840 to be served and it's going to ask to this proess environment what is the default

00:24:50.080 processors for that type so this is implementation of the

00:24:55.200 default processor for is really is simple it's only takes the bundle

00:25:02.000 processor registered for that M type and if there is any processors registered

00:25:08.399 it's going to use that in example here

00:25:13.960 the bundle pipeline that's going to be used we just PR

00:25:20.799 compile all the required files in match then all together to compile each file

00:25:28.760 the self pipeline is going to be used the self pipeline is just the pipeline

00:25:33.799 that's get only one file and it's reg like

00:25:39.559 this it's pretty similar to the default and the implementation is also not so

00:25:45.760 complic complicated like it's get all the push processors of that type get all

00:25:52.480 that s to that type and later it get the prep processor process for that type in

00:25:59.840 the end it it adds the file reader processor that special kind of processors that reads the file from the

00:26:06.640 file system so to explain better I'm going to show uh image how that works so

00:26:14.880 we have the file read as the first processor later we have the C processor

00:26:20.600 and then the r directive processor so the file reader will read the file from

00:26:26.799 the file system and pass that data to the CCRI processor that's going to process the cscript code

00:26:35.039 and generate JavaScript code from that and later it's going to read out the

00:26:40.440 directives we have inside that asset and include include it in the metadata of

00:26:48.000 the asset so the processors always run in reverse order that they are

00:26:55.080 registered so first is the file reader later the cscript reader later the

00:27:01.240 directive processor after this is done for each individual file the BR processor will

00:27:08.559 measure of them and the result is going to be sent back to the browser so it's a

00:27:15.720 little bit complicated a lot of things have been done done in that single asset

00:27:24.799 request and you can imagine that this can be really slow in production so the

00:27:31.640 key difference between development and production is that in production all of

00:27:36.919 this happen in the prile test that we run in the deploy and only I st assets

00:27:44.159 is returned to your browser you don't need to do all this work every single

00:27:51.000 time now I'm going to sh show you how to extend these Pockets like one of the

00:27:56.679 things that we did in shopy was creating new directives in shopy we are using npm and

00:28:04.159 node modules to to be included in all Javas setup and

00:28:10.200 we created a new directive processor that's Define a npm directive so to

00:28:16.880 define a new directive you need to define a method that is called process

00:28:24.480 underscore something uncore directive and this I is the name of the directive

00:28:30.320 that is going to be used I'm not going to talk about the implementation of this

00:28:36.039 so after that's defined I can change my Thea directive processor with my new

00:28:44.640 directive processor that's in this case is my npm directive processor and I can use this new direct

00:28:52.240 directive in my JavaScript files now like when I saying that I need the low

00:28:58.240 Dash npm package I only use this new directive another way to extend these

00:29:04.799 Rockets is we also use that in shopy is sometimes we need to generate PNG image

00:29:13.760 from SVG because PG SVG does not work in

00:29:19.080 all the browsers so instead of doing that manually every single time that

00:29:25.000 someone's add a new SVG we have have a transform that does that automatically

00:29:30.200 for you at compile time so we register a transform from SVG to

00:29:36.519 PG and in my manif file I can say given

00:29:42.000 that I have a SVG file Al compile P version for

00:29:47.919 it and the implementation of this is really simple we are using the AR gen

00:29:55.360 just to read the data set the format to SPG

00:30:01.519 and later create the the result of that PG

00:30:10.600 image so this proess is used in many raos application out there it's inside

00:30:18.919 every single new application that's is generated but like we are right know here many users

00:30:26.840 don't know it's exist many users don't know how it works so it's important to everyone to

00:30:35.640 understand your tools to document your understanding this is what I doing here

00:30:53.480 you thank you rapael awesome uh so you know to deal with the questions you find

00:30:59.320 Rafael around the

00:31:16.760 stage

How Sprockets Works

EuRuKo 2016