00:00:10.240
Welcome to the session. My name is George Ma, and I am a backend developer on Shopify's Rails infrastructure team. I love improving the developer experience. Currently, I am building dependency tools for Ruby and Rails to make upgrades delightful, and I'm excited to share them with you!
00:00:24.320
Let's give him a warm round of applause!
00:00:36.399
Hello, Rails! Whether it's a software update on your iPhone or an app update for your favorite application, we've all experienced the ease of modern software updates. With a simple click of a button, our devices go to work installing the latest features, security patches, and performance improvements. By the time we check back, the update is completed, providing us with a fresh experience—all without us having to lift a finger.
00:01:01.120
Now, let's shift gears and think about a framework we all love: Rails. If you work with Rails, there's a good chance you've either upgraded a Rails application, reviewed a coworker's upgrade, or are simply curious about how upgrades can be done at scale. If you've done an upgrade in the past, you'll know that it isn't quite as streamlined as it could be.
00:01:20.119
Why is this the case? More importantly, how can we make Rails upgrades as smooth and effortless as our routine software updates? In this talk, we will explore all aspects of Ruby and Rails versioning. We'll start with some context regarding our motivations behind improving the upgrade process. Next, I'll share how introducing an upgrade schedule alleviated some of the unexpected challenges that come with version updates, and showcase how we scaled Rails upgrades at Shopify.
00:01:43.360
Lastly, I'll share how we leveraged and contributed to open-source projects to streamline Ruby upgrades. The end goal of this talk is to equip you, the audience, with new insights and practical strategies that you can apply in your own companies or projects today.
00:02:10.239
Hi, I'm George, a developer on the Rails infrastructure team at Shopify. A good chunk of my time here has been spent improving the developer experience of dependency upgrades.
00:02:15.280
A little bit about myself: I grew up just northeast of here in the Greater Toronto Area, which you might find familiar if you plan on attending Rails World in September, as Toronto is the host city for that conference.
00:02:33.319
If you do ever find yourself in Toronto, here are two quick tips to help you blend in with the locals. First, when pronouncing 'Toronto,' say it without the second 'T'—just 'Toronto.' You'll instantly earn brownie points from the locals.
00:02:39.519
Secondly, if you meet someone local for the first time and they say they are from Toronto, don’t just assume they are from the downtown area. Toronto as a city is quite large, and it’s common for people living in the suburbs surrounding the city to just say they are from Toronto for ease of explanation.
00:03:04.360
Now, let's shift our focus to the main topic of today's talk: Rails upgrades. I'll begin by providing some context on Rails versioning and the upgrade process so that everyone is caught up to speed.
00:03:09.879
We will highlight specific aspects that could be improved and offer insights into why simplifying upgrades has proven so invaluable to us. When considering any gem such as Rails, it's important to note three versions: major, minor, and patch versions.
00:03:32.799
Major and minor versions introduce new features that may alter the public API. These upgrades could lead to deprecations, potentially causing errors in your application if not addressed. The third number represents a patch version, and these upgrades contain no API changes and are generally for bug fixes. There's also a fourth version to make note of, which is the security patch number. These version releases are typically associated with a security incident (CVE). In this example, you can see how this patch contains a fix for a security vulnerability in Actionpack.
00:04:02.480
Now that we're all on the same page, I'll provide a walkthrough of a standard Rails upgrade process. A highly recommended prerequisite is to have solid test coverage for the important parts of your application. What constitutes good coverage? Ideally, we want to strive for perfection, but if you're starting with little to no coverage, I recommend a target of 75% or above.
00:04:27.000
You can use a gem called SimpleCov to analyze your test coverage, which is super easy to set up. It only requires two steps: first, add SimpleCov to your gem file and execute a bundle install. Then, add the load files to the top of your test helper if you're using MiniTest or to your spec helper if you're using RSpec.
00:04:53.679
Once this setup is complete, running your tests will generate an HTML file that provides a report of your test coverage. With high test coverage after an upgrade, you can feel confident that as long as your CI passes, your application should work correctly. It's still wise, however, to test any business logic in production post-deployment to ensure everything functions as expected.
00:05:29.240
Now onto the Rails upgrade process. Starting off, if you’re on version Rails 6.1.5, first update the version of Rails in your Gemfile to the latest patch version.
00:05:46.160
Next, fix any deprecations and ensure your CI passes before updating to the next minor version. The process for these three steps mirrors that of updating any gem. However, although Rails is installed like any other gem, it is upgraded uniquely.
00:06:06.039
After upgrading to a new major or minor version, you will need to run the bin/rails app:update task. This is an interactive script provided by Rails that guides you through overwriting or modifying existing Rails configuration files that may have changed between versions.
00:06:35.919
This task generates a new framework defaults file in your config folder for the version you are upgrading to. It’s recommended that you uncomment each new line one by one and implement each change gradually.
00:06:54.160
Once all configurations have been updated, you can bump the load defaults to the version you've upgraded to—7.0 in this case—and remove the new framework defaults file. If you started on a version older than one minor from the latest, you would repeat these steps until you arrive at your desired Rails version.
00:07:18.960
With that, you’re done! This is all the context you need to know to complete a Rails upgrade thoroughly. These steps are front and center in the Rails guide and likely the first search result if you were to search for Rails upgrades.
00:07:45.840
Given that the upgrade process is clearly documented, all Rails apps at Shopify should be upgraded correctly, right? If only it were that easy!
00:08:03.400
For some context, at Shopify, we have hundreds of Rails apps outside our core monolith. The majority are internal apps and libraries that are critical for the merchants or the teams they belong to. This includes Rails apps such as those powering our checkout carts, automation tools like Shopify Flow, and merchant-focused web applications like the Logo Creator and Linkpop.
00:08:30.680
So, what is the problem with Rails upgrades as they currently are? For our Rails 7.0 upgrades—before our investments in simplifying the upgrade process—we found that the experience of Rails upgrades left a lot to be desired. This graph shows the thoughts of a Shopify developer after upgrading and, as you can see, the feedback was not very positive.
00:08:57.240
Described as an overall painful process, a review of the Rails 7.0 upgrades showed that despite calling out the upgrade guide, 25% of all upgrades had forgotten a critical step—like bumping the load defaults—or made common mistakes, such as skipping over a minor version, creating more work for themselves in the long run.
00:09:33.040
At its core, Rails is like any other Ruby gem; however, it is upgraded uniquely. It's manual, prone to missteps, and often handled by developers during their sprint work.
00:09:58.600
There are a few additional factors that make Rails upgrades difficult at scale. This can be best explained with an analogy. Raise your hand if you have ever procrastinated a Mac or Windows PC update. Nice! That’s a lot of people! You probably have never thought about it, but there are two main reasons behind this.
00:10:28.720
First is unexpectedness; unless you're explicitly waiting for a specific version to release, usually an update notification comes out of the blue and disrupts your flow. Second is a lack of perceived benefit; nobody really wants to save and close what they're doing just to perform an upgrade when it's unclear what benefits there are to gain.
00:10:56.360
Why do I bring this up? The psychology behind why we are so adverse to operating system updates has many parallels to what a developer feels when they get hit with a Ruby or Rails upgrade during their project sprint.
00:11:22.960
We have these procrastination reasons with operating system updates where we don’t need to lift a finger; we just click a button. In contrast, upgrading Ruby and Rails has many steps in between. You have to manually fix deprecations, and there are additional upgrade steps to perform to complete the upgrade.
00:11:50.440
If you've performed a Rails upgrade before, you know that the process isn't too difficult. But with 500 Rails repositories at Shopify and tens of thousands of Rails repos worldwide, expecting all of them to follow the upgrade steps perfectly is a challenging ask!
00:12:08.480
The most common mistake we saw was mismatched load default versions. As a whole, we saw that organization-wide, there were 20 different versions of Rails and 15 different versions of Ruby in production. This is unsustainable for upgrade agility, security, and developer happiness.
00:12:36.720
With how things currently were, we could not guarantee that the whole company could upgrade to a specific version of Rails within a short timeline to take advantage of new Rails features.
00:12:56.720
The upgrade experience left a lot to be desired. At Shopify, we recognized the need for a more automated solution to simplify both Ruby and Rails upgrades. Our goal was to create process improvements and enhance tooling to make things more predictable and automated.
00:13:23.960
This would simplify the upgrade process, enabling timely upgrades, reducing the learning curve for developers, and ultimately allowing app owners to easily stay up to date with Ruby and Rails features.
00:13:52.600
Some of the improvements included optimizations such as object shapes in Ruby 3.3 and upcoming developments like Dev container generation and official Language Server Protocol (LSP) in Rails 8.
00:14:12.000
Now that we've identified some of the problems, let’s dive into how we tackled them, starting with those psychological blockers. Our first solution did not require any coding; it required some retrospection and a revamp of our processes.
00:14:43.760
This approach doesn't require any proprietary code; it can be easily implemented in your workflow or at your company. Our solution was to introduce an upgrade calendar at the start of 2022.
00:15:03.720
This calendar gave teams advanced notice of intended upgrade dates, allowing them to weave dependency upgrades into their annual roadmaps. For example, teams had until April 1st to upgrade their app to Ruby 3.1 and until August 1st for Rails 7.0.
00:15:30.320
An upgrade calendar intuitively provides a consistent cadence for app owners, helping to alleviate some of the unexpected challenges we mentioned earlier.
00:15:51.280
So, what happens to apps that are not upgraded by the deadline? Our framework for applications not upgraded is as follows: for Tier 1 and Tier 2 apps—critical to the business—if they are not upgraded, the issue is escalated to the app team’s VP to prioritize the work.
00:16:14.000
For Tier 3 and Tier 4 apps—either internal tools, experimental, or test applications—an expiry process is initiated if they are not upgraded. After a period of time, the applications will be deleted.
00:16:40.280
Setting up a regular upgrade schedule in this manner also offers another significant advantage. By creating a consistent cadence of upgrades, it acts as a forcing function for app owners to consistently evaluate whether their Rails app truly needs to exist.
00:17:00.360
If an app is no longer needed, this allows for an opportunity for that app to be archived. Over the last two Rails upgrade cycles—Rails 7.0 and Rails 7.1—the upgrade calendar directly led to the sunsetting of hundreds of apps.
00:17:22.120
This resulted in less code to maintain overall, which is a net positive for us. This built-in reflection period for archival may be one of the strongest reasons to implement an upgrade schedule today in your company or workflow.
00:17:40.520
So far, we've shown how an upgrade calendar addressed the issue of unexpectedness while also providing the secondary benefit of giving app owners space to consider sunsetting unused apps.
00:18:10.320
To address the second psychological factor—the perceived lack of value—we made it a point to communicate to app owners the many reasons for maintaining an up-to-date Rails version.
00:18:30.520
This starts with the three S's: support, stability, and security. First, there is simply no support for bugs in older unsupported versions of Rails, so by staying up to date, you are less likely to debug something that was fixed years ago.
00:18:54.240
Secondly, outdated Rails versions can lead to compatibility issues with new features or unexpected behavior due to deprecated logic. By keeping your Rails version current, you ensure a stable, consistent, and dependable platform for your application.
00:19:20.640
Next is security; keeping up to date ensures protection from security vulnerabilities and provides access to improved security features and tools. Any app older than Rails 6.0 is unsupported for CVEs and should be updated as soon as possible.
00:19:44.920
Here are some examples of potential vulnerabilities: first, a cross-site scripting vulnerability in Actionpack, which can lead to unauthorized access or data theft; second, a denial-of-service vulnerability in Active Record, which can cause a service to become unavailable for legitimate users; and lastly, a locally encrypted vulnerability in Active Support that can lead to data breaches.
00:20:13.000
While software vulnerabilities are sadly inevitable, by staying up to date and off unsupported versions of Rails, you give your apps the best chance of staying secure.
00:20:35.280
Lastly, keeping apps up to date is a net positive in terms of developer happiness for both new onboarding developers and established team members alike.
00:20:54.120
For a developer onboarding for the first time to a new team or project, imagine how nice it would be to only need a single minor version of Ruby and ensure consistency across any organization repository.
00:21:17.480
Similarly, for established team members, staying up to date prevents frustration. Leaving an app on an old Rails version and postponing upgrades can lead to a buildup of issues, such as having to deal with many deprecations all at once when an upgrade becomes necessary.
00:21:37.080
This brings us to the investments we made to scale Rails upgrades at Shopify. The introduction of an upgrade calendar alleviated the element of surprise and clearly highlighted why it is beneficial for all apps to be on the latest stable Rails version.
00:22:01.600
Looking back at the statistics of the Rails 7.0 upgrades, it was clear that there was plenty of room to create tooling to simplify upgrades even further.
00:22:23.680
While there are existing tools for dependency bumps, like Dependabot, which is great for security and version upgrades for most gems and dependencies, Rails is not upgraded like most gems. Dependabot will create PRs to upgrade Rails but does so in a somewhat naive way, treating it like any other dependency upgrade.
00:22:57.280
Because of this, the Dependabot PRs for major or minor updates can confuse app owners more than they help. We recommend that app owners disable Dependabot from making automated bumps for Rails.
00:23:24.680
So, what approach did we take instead? If you take a step back and examine the upgrade steps, you'll see they involve repeated actions. Knowing this, we decided to create tooling that can automate these repeated steps.
00:23:42.120
The tooling we developed to simplify and automate the upgrade process is a gem called Rails Upgrade and a web app called Solid Track. The Rails Upgrade gem we developed is a command-line script designed to semi-automate the Rails upgrade process.
00:24:07.720
When run on a Rails app, the script determines the state of your application and executes the appropriate upgrade steps accordingly. The script is designed to be idempotent, meaning it can be run on a Rails app in any state and ensures that no steps are missed, following the proper upgrade flow.
00:24:37.640
The script applies a series of steps for a Rails upgrade that are consistent with the upgrade process we discussed earlier. Commits are made with atomic changes at the end of each step, culminating in the creation of a pull request (PR) for user testing and review.
00:24:55.360
Additionally, we can add steps to enhance or improve the upgrade process according to our needs. For instance, we incorporate RBI updaters, auto-corrections for deprecations, and leverage the deprecation helpers built into Rails.
00:25:16.680
In our case, we load the plugin Rails Upgrade Shopify into our CLI code and run custom steps, such as regenerating RBI files for gems used in your application using a gem called Tapioca. This plugin runs after a patch or minor update, saving developers from updating their RBI files manually.
00:25:46.720
We also have a step called 'disallow deprecations.' It raises deprecations on the test and development environment prior to updating to the next minor version, helping to draw additional attention to any deprecations that may exist in an app.
00:26:05.280
Lastly, each step will also log guidance and a task list for the actions performed, which is included in the body of the PR.
00:26:33.360
Now let’s shift our focus to Solid Track. Solid Track is a web app for everything related to Rails upgrades at Shopify. It includes a dashboard for all repositories, a form to onboard your application, and statistics for the Rails versions of all currently onboarded apps.
00:26:54.240
GitHub action workflows enable the pull requests we saw earlier to be created. These workflows call the Rails Upgrade gem, making it easier for app owners.
00:27:12.160
How does it work behind the scenes? The process begins by onboarding your application using a simple form in Rails. Rails Upgrade runs are created weekly through a Sidekiq job or can also be triggered manually.
00:27:40.640
In our case, we have a weekly schedule that runs every Monday. Once a run is made, a GitHub action workflow is created, which fetches and runs the Rails Upgrade gem.
00:28:02.160
Any changes created by the Rails Upgrade gem are compared to the main branch and then applied to a separate branch. If changes are made, a PR is created on that branch, similar to how Dependabot creates PRs.
00:28:28.960
With this setup, we were able to realize great success. In January of 2023, we announced the ability to upgrade more than 90 applications to Rails 7.0 in the first week. By February, we had upgraded 80% of services to Rails 7.1.
00:28:52.240
The final percentage of upgraded apps sat at 96% for Rails 7.1! Previously, we had 20 different versions of Rails in production, which has now consolidated to just two minor versions.
00:29:12.240
So far, we've talked a lot about how we improved the Rails side of things, from the introduction of an upgrade calendar to better prepare teams for upgrades to better tooling in the form of the Rails Upgrade gem and Solid Track.
00:29:31.679
Now, let’s discuss how we scaled Ruby upgrades at Shopify. Since the inception of our upgrade calendar, we have kept all Ruby apps on the latest versions of Ruby for the past two years.
00:29:50.760
During this period, we encountered challenges with Ruby versions being defined in multiple places across our systems. Specifically, Ruby version definitions were duplicated across gem configuration files, such as the RuboCop YAML file and internal development tooling.
00:30:24.080
When definitions exist in multiple locations, it becomes easy to forget to update one of them during an upgrade, potentially resulting in runtime errors.
00:30:49.040
The most common oversight was that the target Ruby version defined in your RuboCop YAML would become out of sync. After completing our investigation, we found that a PR by Bundler compelled us to standardize all Ruby definitions into a single ecosystem standard: the Ruby version file.
00:31:30.560
Standardizing Ruby definitions ensures consistency and simplifies future Ruby upgrades, as owners only need to change a single file.
00:31:57.240
However, this journey was not without its challenges. One issue we faced was Dependabot performing incorrect dependency resolutions when the Ruby version was not defined explicitly in the Gemfile.
00:32:21.760
Previously, Dependabot only supported reading the Ruby version from the Gemfile, leading to complications if a Ruby version was missing.
00:32:48.560
To address this, we merged a patch that allows Dependabot to use the version defined in the Ruby version file when none is specified in the Gemfile.
00:33:10.080
We also worked on ensuring compatibility with RuboCop for cases where the target Ruby version line was set to be removed. RuboCop has logic to prioritize the Ruby version if it exists in the Ruby version file. However, we needed to adapt it for Ruby gems, which often define a minimum supported Ruby version.
00:33:31.840
To resolve this, we provided a patch that prioritizes the minimum supported Ruby version across all sources, allowing for more consistent use of the Ruby version file.
00:34:00.079
Now that we’ve solved these problems and standardized on the Ruby version file, we’re still in the process of ensuring all Ruby definitions are consistent across our apps.
00:34:27.080
Despite these challenges, the reception has been great, and we've experienced little friction in migrating both Ruby apps and gems to this standard. Going forward, developers only need to change one line to upgrade their Ruby version, significantly improving upgrade ease and consistency.
00:34:44.520
In conclusion, our improvements in upgrade tooling were well received by the Shopify developers. We conducted an annual developer survey that asked how painful it was to upgrade to a new version of Ruby or Rails. The results showed overwhelmingly positive feedback, with 94% of participants reporting little difficulty in upgrading their applications.
00:35:05.520
We will continue to strive to improve that remaining 6%. As key takeaways, first, psychological blockers are real. Introducing structure, whether through a predictable upgrade calendar or an advanced reminder of new version releases, is beneficial.
00:35:27.010
Second, Rails upgrades are unique and should not be done hastily. While they are straightforward, you can always refer back to the official Rails guides. For Ruby upgrades, however, the investments made within Dependabot and RuboCop are readily available for you to take advantage of.
00:35:43.510
If you have not done so already, we highly recommend considering standardizing on the Ruby version file across your Ruby apps and gems.
00:35:59.000
Thank you very much for attending this talk. Many people have contributed to or supported the work we've done improving Ruby and Rails versioning at Shopify. These are just a few who come to mind—thank you!
00:36:19.000
Enjoy the rest of RailsConf, network with as many people as possible, and don't forget to update your Mac!
