Inside Rails: The Lifecycle of a Response

HTTP

Krystan HuffMenne

Inside Rails: The Lifecycle of a Response

by Krystan HuffMenne

In her talk "Inside Rails: The Lifecycle of a Response" at RailsConf 2020, Krystan HuffMenne explores the intricate processes involved in returning a response from a Ruby on Rails application back to the browser. Building on a previous talk about the request lifecycle, she delves deeper into the response lifecycle by illustrating how data flows from the Rails controller to the client. This talk is structured to educate developers, whether they attended the last year’s session or are new to the topic.

Key points discussed include:

- HTTP Protocol: HuffMenne explains that HTTP (Hypertext Transfer Protocol) establishes the communication between the browser and server, setting the groundwork for their interaction.

- Rack Protocol Integration: After the HTTP request is parsed, it undergoes a transformation via the Rack protocol which converts it into a hash that Rails can understand, integrating middleware along the way.

- Response Array: Rails generates a response array made up of a status code, headers, and the response body after executing the controller action. The status code is crucial, as it indicates whether the request was successful and provides information about any issues.

- Status Codes and Their Classes: HuffMenne categorizes status codes into five classes: informational, success, redirection, client error, and server error, clarifying their significance in web interactions and SEO.
- Headers and Their Roles: Additional information is conveyed through response headers, which guide the browser on caching and content type. HuffMenne outlines headers like content type, content length, and cache control that impact how responses are handled by browsers.

- Rendering Views: The talk culminates in how Rails translates controller actions into user-visible HTML pages, employing Action View helpers to compile response content with dynamic data.

HuffMenne uses engaging examples throughout her presentation, such as how different status codes lead to distinct browser behaviors and caching strategies for optimizing application performance.

Concluding her talk, HuffMenne emphasizes the beauty and complexity of the Rails response lifecycle, praising the elegance of how Rails operates from the server-side to deliver user-requested information seamlessly. She encourages developers to leverage these concepts to build faster and more efficient applications as they better understand the framework's inner workings.

Overall, this session serves as a comprehensive guide to the Rails response process, emphasizing best practices in handling responses and the valuable role of HTTP and Rack protocols, along with status codes and headers.

00:00:09.000 Hello RailsConf! I'm Krystan HuffMenne, joining you from Portland, Oregon, home of Powell's Books, Voodoo Donuts, and Skylight, the smart performance profiler for Ruby on Rails applications. Last year, the Skylight team gave a talk titled 'Inside Rails: The Lifecycle of a Request.' In that talk, we covered everything that happens between typing a URL into your browser to the request reaching your Rails controller action. However, that talk ended with a cliffhanger: once we are in the controller action, how does Rails send a response back to the browser? Together, these two talks will paint a complete picture of the browser request-response cycle, the foundation on which the entire field of web development is built. Don't worry if you haven't seen the first talk; we'll start with a little recap of the important concepts. Buckle up because we are headed on a safari into the lifecycle of the response inside Rails.

00:01:11.350 First, let's get into our Safari Jeep and head on over to skylight.io/safari. When we visit this page, we should see 'Hello, World!' But oh no! It appears that our Safari server has been overtaken by lions. Instead of 'Hello, World!', we see 'Roar, Savanna!' How did this happen? Let's find out. First, we need to answer the question: when our browser connects to a server, how does the server know what the browser is asking for? The browser and the server must agree on a language to communicate, and this set of rules is called HTTP. It stands for Hypertext Transfer Protocol, which is the language that both browsers and web servers can understand. Protocol is just a fancy word for a set of rules.

00:01:29.530 Let's try it out. We can use a program called Telnet to open a connection to the Skylight Safari server to get skylight.io/safari. Here is the simplest request we could make: it specifies that it is a GET request for the path '/safari' using HTTP protocol version 1.1 and it's for the host skylight.io. The HTTP compliant response from the server looks something like this: it specifies that the request was successful, gives a bunch of header information like content type, content length, and date, and finally delivers 'Roar, Savanna!' as the response body. But what happened in between sending this request and receiving the response? The request is sent from the browser through the interwebs to our web server, and then another protocol kicks in: the Rack protocol. This is a set of rules for how to translate an HTTP-compliant request into a format that a Rack-compliant Ruby app like Rails can understand.

00:02:25.030 The web server, such as Puma or Unicorn, interprets the HTTP request and parses it into an environment hash. The web server then calls your Rails app with this hash. Rails receives the environment hash, passes it through a series of middleware, and ultimately into your controller action. In the Safari controller, there is an action called 'hello' that tells Rails to render a plaintext response that says 'Roar, Savanna!' Rails runs your controller code, passes it back through all of that middleware, and then returns an array of three things: the status code, a hash of headers, and the response body. We'll call this the response array.

00:03:12.160 The Rack-compliant web server receives this array, converts it into an HTTP-compliant plaintext response, and sends it cheerfully back to your browser. Easy-peasy, right? But how did Rails know what to put in this array? And how does the browser know what to do with this response? That's where this year's talk comes in. The first item in the response array is the status code. Simply put, the status code is a three-digit number indicating whether the request was successful or not and why. Status codes are divided into five classes: 100s are informational (these are pretty rare, so we won't go into more detail), 200s indicate success, 300s signify redirection, 400s indicate client errors (errors originating from the client that made the request), and 500s denote server errors (errors originating on the server). Standardized status codes help clients make sense of the response even if they can't read English or whatever human language the response body was written in.

00:04:29.090 This allows the browser, for example, to display the appropriate UI elements to the user or in development tools. Status codes also tell search engine crawlers what to do; for instance, pages responding with 500 errors will be revisited later, while pages with 200 OK status codes will be indexed. For these reasons, we aim to be precise when choosing a status code. The simplest responses are those that require no response body. Even better, we can instruct the browser not to expect a response body at all by selecting the correct status code. For example, let's say our Safari controller has an action to 'eat hippo.' This action allows the current user to consume the hippo as long as they are a lion. If successful, it responds with a simple 204, which means the server has successfully fulfilled the request and there is no additional content to send in the response body.

00:06:00.000 In Safari speak, that's the lion successfully eating the hippo, and we can expect the hippo to have no body. In Rails, the HEAD method is shorthand for responding only with this status, headers, and an empty body. The HEAD method takes a symbol corresponding to a status—in this case, 'no content' for 204. Another common set of status codes is the 300 series for redirects. For instance, if we send a GET request to skylight.io/find_hippo, the 'find hippo' action redirects us to the Oasis URL because it's the dry season and the hippos have moved to find water. The Rails 'redirect_to' method responds with a 302 Found status by default and includes a location header with the URL to which the browser should redirect. This status informs the browser that the hippo temporarily resides in the Oasis URL.

00:07:22.300 Sometimes, the hippo resides elsewhere, so it's crucial to check this location first. But let's say the hippo has permanently moved to the oasis, perhaps due to climate changes. In this case, we could pass 'moved_permanently', which corresponds with a 301 status code. This informs the browser that the hippo has moved permanently to the Oasis. Now, whenever you're looking for the hippo, look in the Oasis. The next time you try to visit the hippo at '/find_hippo', your browser can automatically visit '/oasis' instead without making an extra request to '/find_hippo'. Alternatively, we could modify our routes file to remove the controller action altogether; the redirect helper in the router responds with a 301 as well.

00:08:58.000 There is one important thing to note about the redirect_to method: the controller continues to execute the code in the action even after you've called redirect_to. For example, look at this version of the 'find hippo' action. We redirect to the Oasis URL if it's the dry season, but if it's the rainy season, we'll just visit the endpoint to see it in action. Oops! Because we didn't return when we called redirect_to, we actually hit both redirect and render, leaving Rails uncertain which response to respond with: a 301 or a 200. Rails throws a double render error, which can be fixed by moving the redirect_to into a before_action. This way, we ensure that render doesn't also get called, as it would skip our entire controller action.

00:10:10.000 Okay, let's move on. Status codes are a very concise way to convey important information to the browser, but often we need to include additional instructions to the browser about how to handle our response. Enter headers: headers provide additional information about the response. This information may instruct the browser on how and for how long to cache the response, or it might provide metadata for use in a JavaScript client app. Headers are included in a hash as the second item in the response array that our Rails app returns to the web server. We've already discussed the location header, which informs the browser where to redirect. Here are some other common headers you might encounter in a Rails app.

00:11:42.000 The content type response header tells the browser the content type of the returned content—be it an image, an HTML document, or plain formatted text. The browser checks this to determine how to display the response in the UI. The content length header informs the browser of the size, in bytes, of the response. For example, you might send a HEAD request to an endpoint that can respond with 'HEAD OK' and a content length, allowing you to see how many bytes the response will contain, aiding in, say, generating a download percentage without waiting for the entire body to download. Setting this header is automated by the Rack content length middleware. The Set-Cookie header consists of a semicolon-separated key-value string representing the cookies shared between the server and the browser. For instance, Rails creates a cookie to track a user's requests during a session. Cookies in Rails are managed by a class humorously called the Cookie Jar.

00:13:25.000 These headers, along with many others, are managed automatically by Rails. You can also manually set a header using response.headers. Headers can give the browser directions regarding caching. HTTP caching occurs when your browser or a proxy server stores an entire HTTP response. The next time you make a request to that endpoint, the stored response can be returned more quickly. Caching behavior varies depending on the returned status code, which is yet another reason why status codes are important. The primary header used to control caching behavior is aptly named the Cache-Control header.

00:14:44.000 Let's look at some examples. Consider our 'find hippo' action, which finds the hippo and renders it. The code is straightforward, but we're experiencing performance issues. It turns out that hippos are large and cumbersome to render, so perhaps we should render the hippo once and then cache it indefinitely. HTTP cache forever allows us to achieve this by setting the cache control headers' max-age directive to three billion one hundred fifty-five million six hundred ninety-five thousand two hundred seconds, equivalent to one century—essentially forever in computer terms. It also sets the 'private' directive, indicating that the browser and any proxies along the way should prefer to cache this private hippo only in the user’s browser rather than in a shared cache.

00:16:30.000 If we want to allow caching by shared caches, we can simply specify 'public: true' in the HTTP cache forever to inform browser proxies that we are okay with caching the hippo response as it moves along to the browser. For further examples of indefinite caching, let's include a picture of our hippo in the template. When we visit the page and check the image source, we notice that Rails doesn't serve the image at '/assets/hippo.png'; instead, it serves it at '/assets/hippo-gobbledygook.png'. What is that about? When the server serves our image, it configures the Cache-Control header to effectively achieve HTTP cache forever. Browsers and browser proxies, like CDNs, which I mentioned earlier, will cache that hippo pic perpetually. But what if we change the picture? How will our users access the most up-to-date hippo pics on the internet?

00:18:48.000 The solution is fingerprinting. The 'gobbledygook' is essentially the image's fingerprint, which is generated each time the Rails asset pipeline compiles the image based on its content. If the image changes, the fingerprint linked in the HTML changes, prompting the browser to retrieve the updated version rather than displaying the cached hippo pic. Now, back to response caching: was it smart to cache the entire hippo forever? The hippos only live about 40 years and undoubtedly change during that time! We should consider caching the hippo for an hour instead. 'Expires in' sets the Cache-Control header’s max-age directive for the given time increment, prompting the browser to reload the hippo if we visit the page after an hour. But how do we ensure that the hippo won’t change within that hour? Well, this is difficult to ascertain.

00:20:48.000 Fortunately, this is the default behavior without any specific caching code; the Cache-Control header appears like this: Rails adds the 'must-revalidate' directive to the Cache-Control header, meaning the browser should revalidate the cached response before displaying it to the user. Rails also sets the max-age directive to zero seconds, indicating that the cached response should instantly be considered stale. Together, these directives instruct the browser to always validate the cached response before showing it. So, how does this revalidation process work? The first time we visit the '/find_hippo' endpoint, Rails executes our code to generate the response body, including performing all the tasks to find and render the hippo.

00:22:26.000 Before Rails passes the body to your server, middleware called Rack ETag digests the response body into a unique identifier. This functions similarly to the asset fingerprints discussed earlier. Rack ETag then sets the ETag response header with this entity tag, and the browser caches this response, including the headers. When we revisit this page, our browser notes the cached response is stale, with a max-age of zero, and that we've requested a revalidation. Thus, when our browser sends the GET request, it includes the entity tag associated with the cached response back to the server via the 'If-None-Match' request header. The server executes our code to reproduce the response body, repeating the work involved in locating and rendering the hippo.

00:24:38.000 Once again, it passes the body to Rack ETag, which digests the response body into the unique entity tag and sets the ETag response header. The next middleware in the chain, Rack Conditional GET, assesses whether the new ETag header matches the entity tag sent by the 'If-None-Match' request header. If they match, Rack Conditional GET replaces the status with a 304 Not Modified and discards the body. The browser avoids waiting to download the redundant body, and the 304 status instructs the browser to use the cached response instead. If the new ETag does not match, the server transmits the full response alongside the original status code. Consequently, the browser will render a fresh hippo. It seems like the server invests considerable resources rendering an entire hippo merely to generate and compare ETags.

00:26:06.000 Recognizing that the sole reason the response body changes is due to the hippo herself changing, there must be a more efficient method. Enter the stale method: now our action specifies rendering the hippo only if she is stale. Although we still receive the same caching headers as with the default action, the ETag is different. Even if our response body is identical, what changed? The stale method instructs Rails not to burden itself with rendering the complete response body to construct the ETag; instead, it merely checks if the hippo herself has changed and builds the ETag on that basis.

00:27:30.000 Under the hood, Rails generates a string derived from a combination of the model name, ID, and updated time, then processes that through the ETag digest algorithm. This alleviates the server from the heavy lifting of rendering the entire body solely for the purpose of generating the ETag. Finally, what if the hippo is so private that she never wishes to be cached? Interestingly, Rails does not yet have a built-in method for this, which requires manually setting the Cache-Control header directly. The 'no-store' directive indicates that the response must not be retained in any cache, either in the browser or in any proxies along the route.

00:28:40.000 This should not be confused with the poorly named 'no-cache' directive, which permits the response to be retained in any cache, but mandates that the stored response must be revalidated before utilization. Thus, after utilizing the status code and headers to communicate with the browser about how to handle our response, we should address one of the most crucial components of many responses: the body. The body is the final item in the response array; it is a string representing the actual information the user requested.

00:29:50.000 When we request '/find_hippo', how does Rails convert the code we wrote in our controller and view into an HTML page about a specific hippo? Let's discover that. When we visit '/find_hippo' in the browser, our Rails app delivers an HTML response. We can verify this by checking the content type response header. But how did Rails determine to respond with HTML? Rails first looks at any explicitly requested file extensions, such as '/find_hippo.html'. If none are provided, as is the case with the request we made, it assesses the 'Accept' request header.

00:30:58.000 Our Safari browser defaults this 'Accept' request header to 'text/html,' indicating it prefers to receive the HTML content type formatted as a MIME type in the response. It also denotes that if no HTML version is available, the browser is content accepting an XML version as an alternative. The render method invoked in our controller searches for the template matching the requested content type, so in this case, it seeks the 'Safari/hippo.html.erb'. Moreover, it sets the content type header to correspond with the rendered body. Now, we also want a JSON hippo, so let's request '/find_hippo.json'. Oops! We don't have a template for a JSON hippo yet. We could create one, or we could add a respond_to block to manage differing formats.

00:32:45.000 Now, if we request '/find_hippo.json', we receive the 'json_hippo.json'. Interestingly, browsers are not required to adhere to the content type header and might attempt to sniff out the type based on the file's content. For this reason, Rails sets the 'X-Content-Type-Options' header to prevent this behavior. Now that we understand the type of response to generate, there are three ways our Rails controller can produce a response. We've already covered two of the methods in detail: the 'redirect_to' and 'head' controller methods generate responses with status codes, headers, and empty bodies. Only the 'render' method generates a full response that includes a body.

00:34:59.000 For our '/find_hippo' example, let’s say the template appears as follows: upon visiting '/find_hippo' and calling 'render :hippo', the render method locates the appropriate template, fills in the blanks with our instance variables, and generates the body to send back to the browser. To do this, Rails creates a view context class specific to each controller. Here’s a very simplified illustration of what that view context class for the Safari controller resembles. When the view context initializes, Rails iterates over all instance variables set in our controller, such as @hippo, and copies them into the view context object for use in the template.

00:36:05.000 These instance variables are termed assigns. The view context class incorporates all available helpers from Action View, such as 'link_to', along with all helpers defined in our app, like 'current_user'. Each template compiles into an instance method of the view context class. Essentially, each template's method is a sophisticated string concatenation. In this instance, we start the output string with 'Hey, get self.current_user', which is available because we included our app helpers in the view context class. We escape 'current_user.name' as it might have user input, then add it to the output string, followed by ', me.' Next, we invoke 'self.link_to' to generate a link to the page for our hippo, remembering that 'link_to' is available due to our inclusion of Action View helpers as a module.

00:37:43.000 Then, we escape '@hippo.name' for the link text and append the link to the output string, finishing with an exclamation point to conclude the output string and returning the final output string. So let’s put it all together: 'Hey RailsConf, meet Phyllis!' Wow, we finally found the elusive hippo, Phyllis, and she's 200 OK! Along the way, we've encountered rare actions, unimaginable scales, impossible locations, and intimate moments captured from the deepest levels of Rails internals. We've traversed the expansive text plane, appreciating the spectacular Action View as we made our way back to the browser. Thank you for joining me while we explored the amazing lifecycle of a Rails response.

RailsConf 2020 CE