00:00:21.260
I work with Gusto. Gusto is a people platform that provides payroll, benefits, and HR solutions in a single platform for thousands of businesses in the United States. I've been here for six months. I received an email just today.
00:00:32.640
I brought up the infrastructure back-end team, and some of us are actually here today. Thanks for coming out, team. My team built the background job system and handles other work, such as ensuring the health of the CI pipeline and upgrading Ruby and Rails. I really enjoy this work, so if you're interested, please come and talk to me.
00:00:46.020
With that being said, I would like to assert that we should all strive to be on the latest Ruby versions, and here's why: Ruby keeps improving, thanks to the Ruby core team, and we benefit from important updates like bug fixes, security patches, performance improvements, and new Ruby features in both major and minor releases. I believe it is to our advantage to be on the latest Ruby version.
00:01:12.180
This is a list of the current stable releases of Ruby: we have Ruby 3.2.2, 3.1.4, 3.0.6, and 2.7.8. However, if you've been following along, you would know that Ruby 2.7 has reached its end of life as of March 31st of this year.
00:01:30.119
Thus, version 2.7.8 will be the last release for Ruby 2.7. Next year, Ruby 3.0 will also reach its end of life and will no longer be supported. This means that every single year, a team or individual will have to upgrade Ruby for your Rails application to ensure that you're off the deprecated version or at least on one of the supported Ruby versions.
00:01:54.299
Let me share with you how we updated Gusto from Ruby 2.7 to 3.1, including some of the blockers that we encountered. Gusto's main Rails monolith ran on Ruby 2.6 last year, and we knew Ruby 2.7 would reach its end of life in March this year. Therefore, we worked on a project to upgrade Ruby.
00:02:13.800
As Nan points out here, it took us three to four months to complete this upgrade, including my contribution, as I joined a bit later. We successfully upgraded two of our largest Rails applications at Gusto. Let me share with you one blocker we faced during this project.
00:02:31.140
We experienced database timeouts on various tests in our pull request to introduce Ruby 3.1. The database timeouts were caused by an issue within the Active Record connection pool. We investigated this issue and eventually found a minimum reproducible example of the problem. When performing database queries inside an enumerator, such as through a transaction, the program would hang, leading to test timeouts in CI.
00:03:01.980
To address this issue, we needed to understand why this behavior was only apparent in Ruby 3.1 and not in Ruby 2.7, which we were currently running. The first step was to identify the Ruby version that exhibited this behavior. We also needed to run Ruby 3.0 and various patch versions like 3.0.1 and 3.0.2. Ultimately, we determined that the deadlock did not occur in Ruby versions prior to and including 3.1, while deadlocks did occur in versions above 3.1.
00:03:39.239
The next step was to track down the commit that caused the issue. We knew that the change was introduced between 3.1 and 2.0.0, so we reviewed the list of commits. As we understood some parts of the enumerator, we recognized that it was implemented using fibers in Ruby. We found a particularly suspicious-looking commit in the list of backports into the 3.1 version.
00:04:10.680
We traced back this issue to a change on Ruby made approximately two years ago. Ultimately, we understood the reason for the deadlock. As Nan points out in his tweet, Ruby monitors now use Fiber as of 3.0, and since Active Record queries are framed inside an enumerator within a database transaction, it can lead to deadlocks.
00:04:42.540
Figuring out the issue was a significant breakthrough for us, and we collaborated with Jean Busier from Shopify, realizing that an issue had already been opened for this problem on Rails. We shared our experience regarding the bug, which opened discussions with others. Jean and I paired on a solution, leading to an approved pull request.
00:05:40.919
Our team at Gusto was able to backport the changes into our Rails application, thanks to Nan, who diligently documented this entire issue. On December 1st, we successfully backported it into our Rails application. It required extensive investigation into Rails and Ruby source codes, alongside collaborative efforts within Gusto and with others outside Gusto. Ultimately, we were able to upgrade from 2.7 to 3.1, taking around one month from the moment we identified the root cause.
00:06:11.100
Now, consider what would happen if we were running the latest version of Ruby and encountered the deadlock issue as soon as it was introduced. This situation could have been avoided, and that’s what I'm trying to convey.
00:06:31.199
We will also talk about how we would run Ruby head and why we should all do it. So, what is Ruby head, exactly? Ruby head is the development version of Ruby, and "head" refers to the latest commit on the main branch on GitHub, which is what we're referring to here when we talk about 'Ruby head.' It's also known as Ruby trunk or Ruby Edge and contains changes that are not yet available in the stable version of Ruby.
00:07:12.720
This is why we opt not to run it in production but rather execute it in our application tests and CI pipeline. This means we can run two parallel CI pipelines for our Rails application: one for the production Ruby and another for Ruby head. This approach allows us to upgrade Ruby progressively, minimizing potential impacts on our application and letting us see how tests behave.
00:07:54.600
Once a new Ruby version is released, updating the Ruby version file is all that's necessary. Now let's discuss upgrading Ruby gradually versus all at once. Gradual upgrades involve addressing build issues as they arise over time, whereas an all-at-once upgrade is a concentrated effort involving potentially multiple engineers working on a project.
00:08:51.840
While the net sum of work may appear similar, I argue that upgrading Ruby gradually is more efficient. Let's explore the benefits of gradual Ruby upgrades. The first benefit is that we spend less time debugging. Whenever a failure occurs in the Ruby head pipeline, we have a smaller surface area to investigate managing issues rather than going through the commits of two Ruby versions.
00:09:24.780
Another advantage is that it provides us with a list of non-urgent tasks. For instance, we can identify gems that will be incompatible in the latest version or take on new language features, such as keyword argument changes introduced in Ruby 3.0. It allows us to compile a list of tasks over the year, improving our code.
00:10:15.540
We also reduce the risk of encountering major upgrade issues. For instance, we didn’t foresee the issues that I previously described, but if you have visibility into the changes made to Ruby in real time, you lessen the risks of those changes becoming significant blockers in your project during the upgrade process.
00:10:56.880
As for those of you who are not at Shopify or who may feel resource-strapped, it is a better business investment. Once you set up the CI pipeline for Ruby head, you can maintain it throughout the year while being confident in your Ruby upgrades for production once the new version is released. Gradual upgrading would also mean that you won’t have to continually seek leadership buy-in for specific projects every year or when a new Ruby version comes out.
00:11:40.680
Finally, this approach places less burden on a team or an individual. Gradual Ruby upgrades can be perceived as maintenance work, allowing the effort to be distributed over time among engineers. This way, individuals can take ownership of specific issues and learn from these experiences, reducing pressure on any single team.
00:12:12.660
Now, let's discuss some disadvantages of gradual Ruby upgrades. The first one is the upfront work and resources required to set up the CI pipeline to run Ruby head. Additionally, the increase in maintenance effort must be considered, as someone will have to maintain and understand the pipeline.
00:12:28.440
Lastly, the additional pipeline in the CI system will incur extra costs. Personally, I might not remember specific technical details after attending conferences, but I do remember concepts and ideas.
00:13:05.160
If there's one key takeaway from this talk, it’s the importance of feeling comfortable running Ruby head with your Rails application tests. It doesn't have to be Ruby head; it can be the next Ruby version you're using.
00:13:48.420
Let me discuss how we can build Ruby head both locally and using a Docker image. We will create a CI schedule for the Ruby head image and run a toy Rails application test on it. A quick disclaimer: there are many ways to build Ruby head, and this is just one method. I chatted with colleagues from various companies, including Gusto, and we each follow slightly different methods.
00:14:28.560
So, let's consider how Ruby head can fit into your Rails application and CI system. To build Ruby head locally, the first step is to compile it from source, which involves generating the configuration script, compiling Ruby, and installing it.
00:15:04.800
It’s important to differentiate between compiling and installing Ruby. Compiling Ruby means converting the Ruby source code into machine-readable code, requiring configuration of build settings and generating necessary binaries. On the other hand, installing Ruby involves placing the compiled code into the appropriate directories on your local machine for use by applications.
00:15:41.580
If you check the Ruby website under the installation section for building from source, you’ll find three commands: configure, make, and sudo make install. The configure command generates a configuration script for the Ruby build, where you specify build options to enable or disable certain features.
00:16:22.200
The make command compiles the source code into binary executables and libraries, while sudo make install takes the compiled Ruby binaries and installs them into the system, requiring sudo privileges because the default path is not user-writable.
00:16:57.600
The Ruby website advises that using third-party tools or package managers might be a better option, so let’s follow that recommendation. Setting up Ruby locally can be accomplished through various installation managers like RVM, rbenv, or RubyInstall. RVM stands for Ruby Version Manager, and it differs from rbenv.
00:17:36.960
We are going to use Ruby build to install Ruby head. The command you see will build the Ruby head version 3.3.0 Dev and install it to the desktop. We will use Ruby build directly to compile and install Ruby head in our Docker image for several reasons, primarily for simplicity and configurability. With Ruby build, we can compile Ruby with specific configurations without needing to manage multiple versions.
00:18:46.380
I want to share how Ruby build utilizes a build definition file, which acts like a DSL for Ruby build. Each method call maps to functions in the Ruby build script, allowing for customized compilation and installation.
00:19:14.700
For example, the 'install-packet-share' method ensures the correct version of openssl is installed. In some cases, we can install Ruby from the master branch on GitHub rather than downloading.
00:19:58.560
We should also bear in mind that Ruby installers do a lot of work for us, which is why they’re more convenient than manual installations. They handle both system and Ruby dependencies, and they even download and apply patches if needed. It's also worth mentioning that there are other methods to set your Ruby environment, including using a package manager like Homebrew, yet those typically use pre-compiled binaries.
00:20:49.440
Now let’s get to building Ruby head on Docker. Docker is a tool that packages and runs applications in a consistent manner. We will create a Docker image with Ruby head, allowing us to run our applications using this image. There are existing Docker images for Ruby maintained by the Docker community, supporting various Ruby versions based on OS preferences.
00:21:44.820
However, the official Docker images do not support Ubuntu-based images, which is what we use at Gusto. There was an open PR on the Docker library to introduce Ubuntu support, but the maintainer stated a preference for more lightweight operating systems. That's fine because we can build our own.
00:22:20.160
Here’s a Dockerfile for Ruby head, and I will step through it so we can understand what's happening in the image. The first step is to specify the parent image, which in our case will be an Ubuntu image.
00:23:01.320
Next, we copy the build definition file into the Docker image, which we will use to compile Ruby head. We then set up the environment for gem and bundler, as the image is fresh and lacks defaults.
00:23:31.740
Then we install the required system dependencies before compiling and installing Ruby. In this section, we begin by downloading Ruby build from GitHub and extracting the tarball to manually install. This is where we utilize the build definition file to build Ruby head.
00:24:17.880
Afterward, we remove unnecessary files to keep the container clean. Next, we specify the creation of a directory for the gems if it doesn’t already exist. Now let’s look at how the Ruby head build definition file appears.
00:25:01.860
Similar to the previous one, we aim to use a nightly snapshot from Ruby Lane. Nightly snapshots represent the current state of the development branch. This snapshot is what we'll use to install Ruby head, which we can build and run using Docker.
00:25:46.860
Let’s walk through what this looks like once we have the image built. We utilize Docker to build this image and run it in IRB, where the Ruby version is confirmed to be 3.3.0 Dev.
00:26:38.820
Moving forward with time constraints, I won't dive deeply into the topic of Jemalloc, although I'm happy to discuss it if there's time later. How many of you are interested in Jemalloc?
00:27:14.460
Let’s continue. Large-scale Rails applications often consume a significant amount of memory, especially when they're multi-threaded. For instance, if your Rails application employs Puma, which is multi-threaded, it's worth noting that Ruby is a dynamic language.
00:27:49.620
This means Ruby can dynamically create and manage objects during runtime. However, this flexibility can lead to high memory consumption. This is where Jemalloc comes into play.
00:28:21.480
Jemalloc is a memory allocation library that provides an alternative memory allocator. It is specialized in reducing fragmentation within memory, an issue that occurs when memory is allocated and deallocated in ways that leave behind small gaps.
00:29:04.460
Over time, these gaps can accumulate and lead to memory inefficiencies or worse—memory allocation failures. Jemalloc utilizes per-thread caching, meaning that it stores cached memory blocks, which increases efficiency during allocation.
00:29:29.100
However, it should be noted that Jemalloc is not a drop-in replacement for your existing memory allocator. It's not a one-size-fits-all solution, which is partly why Ruby does not ship with it by default. To use Jemalloc with Ruby, you can employ one of two methods.
00:30:09.360
The first method involves using a compilation flag. This means, when using the configure command to build Ruby, you specify it with '--with-jemalloc' to tell Ruby to utilize Jemalloc as the default memory allocator.
00:30:51.840
The second method involves preloading the Jemalloc library before executing Ruby. You do this by utilizing an environment variable called LD_PRELOAD, which allows you to overwrite functions or symbols in shared libraries.
00:31:26.640
This means that if you specify the library to be preloaded, Ruby will employ Jemalloc for memory allocation instead of the default from the operating system. Preloading Jemalloc generally yields more benefits in memory management than compiling Ruby with it.
00:32:08.420
Lastly, it’s crucial to acknowledge that while using LD_PRELOAD can significantly improve memory efficiency, it may result in compatibility issues with certain applications. Please exercise caution when using it.
00:32:43.140
That said, you can also compile Ruby with Jemalloc to test any potential memory or performance benefits you might gain.