Ruby Video
Talks
Keynote: The Past, Present, and Future of Rails at GitHub
Eileen M. Uchitelle
@eileencodes
#rails-upgrades
#ruby-on-rails
#open-source
#technical-debt
#continuous-integration
#development-process
#github
Summarized using AI

Keynote: The Past, Present, and Future of Rails at GitHub

by Eileen M. Uchitelle

In her keynote address at Ruby on Ice 2019, Eileen Uchitelle, a Senior Systems Engineer at GitHub and member of the Rails Core team, discusses the intricate journey of upgrading GitHub's application from Rails 3.2 to 5.2. The talk emphasizes not only the challenges they faced but also the lessons learned that can guide other developers in similar situations.

Key Points Discussed:

  • Introduction to GitHub's Journey with Rails:

    • GitHub has utilized Rails since its inception in 2008.
    • Initial rapid upgrades slowed as GitHub dealt with maintaining a fork of Rails.

  • Historical Context of Rails Development:

    • Rails was created from an actual application, Basecamp, to ensure practicality and effectiveness.
    • GitHub originally forked Rails due to performance issues that surfaced in Rails 3, complicating future upgrades.

  • Technical Debt Accumulation:

    • Over time, GitHub's customized fork diverged significantly from the upstream Rails repository, making upgrades increasingly difficult.
    • The lack of urgency to upgrade led to a build-up of technical debt that hindered GitHub's engineering effectiveness.

  • Upgrade Process Description:

    • After years of being impacted by their fork, GitHub's engineering team identified the need for a structured upgrade plan.
    • A dedicated team was assembled to facilitate the transition through an incremental upgrade approach, utilizing dual Rails versions during the process.

  • Successfully Upgrading to Rails 5.2:

    • After a myriad of challenges, including a substantial number of errors, the team completed the upgrade without downtime.
    • The process was marked by significant teamwork, tracking issues through GitHub project management tools, and refining the codebase.

  • Conclusions and Future Aspirations:

    • Uchitelle highlights the ongoing importance of regular upgrades, the value of staying current with Rails, and engaging with the Rails community.
    • GitHub's commitment to contributing back to the Rails framework was reinforced through this experience, leading to improved collaboration and innovation in the future.

Main Takeaways:

  • Importance of Upgrading: Neglecting upgrades can lead to substantial technical debt, impairing performance and security.
  • Trends in Application Development: Adaptation is crucial for maintaining efficient and robust applications, as outdated frameworks hinder both productivity and talent acquisition.
  • Collaborative Efforts: Building a dedicated team for upgrades and utilizing community support can significantly enhance the upgrade process, promoting better practices and future-ready applications.
00:00:11.830 Our keynote today starts with a bang! For those who don't know her, Eileen Uchitelle is a Senior Systems Engineer at GitHub and a member of the Rails Core team. Please, give her a warm welcome as she comes on stage. Eileen is passionate about open source and will discuss the past, present, and future of Rails at GitHub.
00:01:25.580 Some of you may think, 'Oh, well, you don’t work for GitHub anymore; you work for Microsoft.' This is technically true, but GitHub is still the same GitHub that you have always known and loved. We just have a little extra support from our new partner, Microsoft. I also got an email address that I ignore, which doesn’t make sense. It's like the first two letters of my name and the last five letters of my last name. I can’t even remember it; I always have to look it up! Luckily, we still get to keep our GitHub email addresses. I am also on the Rails Core team, for anyone who isn’t familiar with it. We are responsible for defining what the future of Rails will be.
00:02:07.880 Each year, I reflect on what I've learned, the work I've been doing, and what I'd like to share with you at conferences. When I started writing the abstract for this talk, I initially wanted to delve into the intimate details of upgrading Rails at GitHub. I thought I wanted to share our exact process, what I worked on, what worked, and what didn't. I had spent more than a year and a half working on the Rails 3.2 to 5.2 upgrade and I could certainly talk about it for many hours. However, as I explored the themes surrounding the upgrade, I realized there was a deeper story I wanted to share.
00:02:40.000 How did we fall so far behind Rails master? What compelled us to upgrade despite being lagging? Ultimately, the story I want to tell is one of the past, present, and future of Rails at GitHub. We have been using Rails at GitHub since day one, and at times, Rails and GitHub have had our differences. Many years ago, we forked Rails and almost wrote our own version. We challenged the framework, deviating from the original. At one point, we even wondered if Rails was the right choice for us. However, at the end of the day, Rails has been successful because GitHub is successful, and vice versa. This upgrade wasn’t just a good blog post for Hacker News to criticize; it made it possible for us to continue using and investing in Rails for the long haul.
00:03:52.420 By upgrading, we can change and influence the framework to suit our needs while benefiting the broader Ruby community and the open-source ecosystem. This narrative is part historical, looking back at how GitHub ended up maintaining a custom fork of Rails 2. It’s also part technical, exploring what compelled us to upgrade, our processes, and the difficulties we faced. We will examine the costs of not upgrading and how technical debt accumulates to the point where applications and frameworks start to conflict with one another. Lastly, this story is forward-looking; we will dive into GitHub's efforts to clean up technical debt, our commitment to open source, and our responsibility to support Rails for the long haul.
00:05:03.919 Let's take a trip back to 2004 when DHH announced a new Ruby framework called Ruby on Rails. Immediately, Rails caught the attention of the Ruby community. At RubyConf that year, DHH talked about the history of Rails, its inception, and why it was superior to existing Ruby frameworks. He emphasized that frameworks usually fail because they are built without an application in mind; frameworks should be extracted, not built from scratch.
00:05:23.060 Rails gained popularity and success because it was derived from an actual application—Basecamp. In the early years, Rails complexity grew slowly. Rails 1.0 was released in December 2005, and two years later in December 2007, Rails 1.2 was released. That same year, Tom Preston-Warner showcased a Ruby tool called Grit that allowed you to view Git repositories in an object-oriented way. This tool eventually became the backbone for Git repositories at GitHub. After seeing Grit, Chris Perring was hooked, and shortly after, GitHub was born, utilizing Rails 1.2.3. Back then, we didn’t even have a gem file; the only frameworks we had were Action Mailer, Action Pack, and Active Record. After a short beta period, GitHub was publicly launched in 2008 and moved Rails from its own SVN server to GitHub in April 2008.
00:07:52.150 In the early days, every time Rails released a new version, GitHub quickly upgraded to gain new features and bug fixes. However, at some point between 2008 and 2009, GitHub forked Rails. The exact date is hard to pinpoint because we vendor our gems, and I couldn't find any commits detailing when we forked. Previously, I believed that GitHub forked Rails due to performance issues with Rails 3, which made it challenging for many applications to upgrade. It turns out we had already forked Rails before the performance issues of Rails 3 became apparent. This was a chaotic time for Rails startups, and no one knew what the future of either Rails or GitHub would hold.
00:11:55.090 We were not yet discussing the importance of staying current with the upstream and Rails was less stable than it is today. It would be too extreme to say that Rails didn’t care about performance or stability back then, but many app developers certainly felt that way. It wasn’t a concern the way it is now. Perhaps this was due to our inexperience, as Rails was 'good enough' for its prominent user base, Basecamp. Whatever the reasons, GitHub did not contribute sufficiently upstream. I’m not critical of the past; instead, it’s important to recognize those events to improve for the future. The crucial issue, however, is that GitHub didn’t just fork Rails and make minor bug fixes or performance enhancements. Our fork wasn't merely a copy; it was a customized version of Rails tailored for our needs.
00:13:48.600 As GitHub doubled down on our fork, adding more and more functionality, Rails continued its rapid progress. At the time, no one predicted the cost of forking Rails would have on our application or engineering team. In 2010, Rails 3.0 was released, but many applications couldn't upgrade due to significant performance issues that increased response times drastically. Some applications reported that requests were taking twice as long, and Active Record in Rails 3 was five times slower than Rails 2. Despite knowing about these performance concerns, a few GitHub engineers began working on an upgrade to Rails 3 and Ruby 1.9. What we didn't realize is that GitHub had not only forked Rails; we had also forked Ruby 2, complicating upgrades.
00:16:01.900 The Rails 3 upgrade was not simply an upstream upgrade, but our modified version of Rails 3 with a mix of our unique changes. Trust me, trying to upgrade while back-porting changes made during your initial fork can be downright discouraging, to the point where you'd want to quit programming altogether. Rails 3.2 was released in 2012, and thanks to Aaron Patterson and other contributors, many performance issues were resolved. However, after these improvements, progress on the Rails upgrade at GitHub stalled. The engineering team questioned whether the upgrade efforts were worth the time since they didn’t feel the pain of being on a fork yet. They posed questions like ‘Why upgrade when this version isn't causing us pain? Why upgrade to Rails 3 when our own fork has more features and is better?'
00:20:15.000 These questions suffocated our engineering team and made it increasingly difficult to delineate where the framework ended and the application began. GitHub engineers were forced to fight against the fork, and their security updates felt like a nightmare. Each time Rails announced a new vulnerability, GitHub had to manually fix it. Hiring became a challenge; no one wanted to work on Rails 2.3 applications that didn't resemble modern Rails. Developers lacked the knowledge to work on older versions, leading to slow and inefficient development processes. We eventually realized that we needed to upgrade to avoid being suffocated by our own fork. In 2014, a group of four full-time engineers and a few volunteers authored an upgrade plan and got to work. It took the team six months of coordinated effort to deploy Rails 3.2 to production, and soon after, Rails 3.2 was successfully implemented.
00:24:22.648 It's crucial to note that Rails 3.0 and 3.2 were still forks of Rails with custom GitHub patches. The Rails 3 series began receiving limited security patches. Although the upgrade to Rails 3.2 was a success, the codebase remained far behind master. The work needed to upgrade from Rails 3.2 to 5.x was considerable. For a while, the motivation to move forward waned as the engineering team focused on other projects. They claimed the upgrade wasn’t worth the effort since the existing version was sufficient. Eventually, the pain of being on a fork became more pronounced, and GitHub engineers began feeling suffocated by the weight of technical debt.
00:27:01.150 By now, the engineers reported that upgrading Rails was becoming increasingly difficult. As the team fought against this technical debt, they recognized that they needed to escape the clutches of the fork. So, in 2014, they assembled a group of engineers, wrote an upgrade plan, and tirelessly worked to deploy Rails 3.2 into production. Interestingly, during this time, Rails 5 was released, and many felt it was impossible for GitHub to catch up to the constantly evolving Rails. By late 2017, I joined GitHub, and it was evident that the Rails 4 upgrade was in dire need of attention. There was no dedicated team working on it at that time.
00:30:12.000 I quickly requested that Hugh run the tests for me on Rails 4 to assess the severity of the situation. We discovered there were over 4,000 errors. Luckily, Hugh's counting capabilities weren’t very accurate—he doubled the number of errors he reported! By this point, GitHub engineers had implemented tools to facilitate Rails upgrades. They deployed a system that enabled us to run the application on dual Rails versions simultaneously. This meant we didn’t have to maintain long-running branches—which was crucial as GitHub often shipped updates across hundreds of branches.
00:33:10.000 By allowing dual booting, we concentrated exclusively on test failures rather than merge conflicts, although this method required modifying Bundler. This adjustment enabled us to boot the server and console in both Rails 3.2 and 4, allowing easy comparison and behavior analysis. This process ultimately helped us incrementally upgrade Rails and prevent regressions. Once we achieved a stable build, we enforced that everyone at GitHub adhere to writing code that passes in both 3.2 and 4. Once the 4.1 build was stable, everyone had to write code that passed in both 3.2 and 4. Once we successfully deployed 4 to production, we transitioned to 5.2. In March 2018, one year and three months after I began at GitHub, we deployed Rails 4.2 to production with zero downtime and minimal customer impact.
00:35:47.100 Although this was a tremendous achievement, the Rails 3.2 to 4.2 upgrade took a significant amount of time. After deploying 4.2, I was eager to get started on 5.0 immediately to maintain our momentum. However, I was also incredibly burned out from being the only engineer working full-time on the Rails upgrade. While we had wonderful volunteers, this wasn’t their only responsibility. Upgrading Rails proved to be a challenging and lonely task. Thus, I told my manager I refused to undertake the 5-series upgrade alone.
00:37:25.800 For the 5-series, I led a team of four dedicated engineers who focused solely on the Rails upgrade. I learned which processes were efficient and which ones failed. As a team, we gathered every failed unique error and tracked each individual failure through a GitHub project. This way, we could monitor our progress. Thanks to this streamlined process and our dedicated team, the upgrade from 4.2 to 5.2 was achieved within five months. In August 2018, we successfully deployed Rails 5.2 to production without any downtime and only minimal customer impact. This taught us a lot! Through experience, we learned how to successfully deploy major versions.
00:40:50.100 The upgrade to version 5.2 represented a significant milestone; it marked the first time in ten years that GitHub was no longer on a fork of Rails. We had finally started to pay off ten years of cumulative technical debt—all while fighting against the internal challenges we faced. The experience might have seemed daunting, but the goal of this talk is not merely to share horror stories. Rather, it is to illustrate the tangible costs of neglecting upgrades and how that debt compounds over time. After learning about our story, engineers from other companies approached me for advice on persuading their leadership teams to prioritize upgrades alongside other features and resources.
00:43:41.260 Upgrading is undoubtedly time-consuming and expensive, a fact I won't deny. However, it can be more beneficial to frame the upgrade's cost in quantifiable terms: you can measure the total cost of the required engineers, resources, and time, determining whether it's too pricey. Ultimately, though, it doesn’t matter how expensive the upgrade might seem because the costs associated with not upgrading become immeasurable. Neglecting upgrades will eventually incur higher expenses than executing them. If you remain stagnant, you must become a security expert to manage vulnerabilities since the Rails Core team typically only supports patching current major versions and the latest version of the previous one.
00:46:21.590 For teams on versions lower than 4.0, they bear the burden of understanding and addressing security vulnerabilities themselves. This intricate balance can be tough to maintain because, within the Rails Core team, we don’t disclose how vulnerabilities can be exploited to protect those who can’t upgrade immediately. When you don’t upgrade, you limit yourself in acquiring great talent. New engineers entering the field, including boot camp grads and career changers, are not learning Rails 2 or 3; they won't have the skills or understanding of how those versions function. I might not have accepted a position at GitHub had we still been on a fork of 2.3.
00:49:56.700 Additionally, some gems you rely on will become abandoned or deprecated without upgrades, leaving you with the choices to cope with bugs or fork another dependency. New gems may not support outdated versions of Rails, resulting in additional work or the need to create forks, further complicating the development process. This accumulation of technical debt can drain your energy and resources as you find yourself constructing more revisions on top of an already fragile application. At GitHub, we've amassed a variety of infrastructure code on top of our application, implementing multiple databases, CI tooling, job queues, and more. Ideally, your application should consist solely of the code that defines your product. GitHub's primary value lies within our community, repositories, issues, and the architecture that keeps them running.
00:52:22.620 This infrastructure layer introduces unnecessary complexities, making it harder to maintain clean separation between Rails internals and our product code. The most significant risk of not upgrading Rails is that members of your team may eventually decide that Rails is not the right choice and propose migrating your application to microservices. I want to clarify that this isn't a critique of the Go language or other methods of development; rather, this is a Ruby conference! I want to affirm my love for Ruby and my aspiration to continue being compensated for writing it. It may seem ludicrous to say, but if we don't upgrade Rails, we risk losing the ability to continue writing Ruby.
00:55:58.900 Moreover, the Rails ecosystem won't flourish, and your applications will degrade, ultimately leading to costly rewrites. While the cost of upgrading is tangible, the price of a complete rewrite is astronomical and could jeopardize your career. The key to upgrading Rails is to incrementally pay off the cumulative technical debt you've accumulated and devise a plan for sustaining manageable debt. I won't tell you that upgrading Rails will be a walk in the park; I'll leave that to the person on Hacker News who wonders why we couldn’t execute the upgrade more swiftly. Upgrades do take considerable time, but they weren't our only endeavor as we also removed unused features, redesigned our test framework, and improved database handling for development and test environments.
01:00:11.210 It’s unfair for critics to look at our upgrade timeline and decide that Rails is simply too expensive. We at GitHub also made choices over the years that complicated our upgrades. Nevertheless, this does not imply that Rails is a poor decision or that we, as engineers, were incompetent. Technical debt is real, and each team must grapple with what debt is acceptable and what must be eliminated. At GitHub, we finally came to a consensus: being behind on Rails could no longer be tolerated. You can incrementally work on technical debt while upgrading Rails, positioning your application for success.
01:03:28.870 To help facilitate this process, let us now delve into the considerations for upgrading and the mistakes to avoid to prevent an extensive upgrade process like GitHub's seven-year journey. The first and most critical step is to build a dedicated team. Upgrades can be difficult, and having a supportive team to share ideas and maintain momentum is crucial. If your only upgrade resource leaves, your upgrade process may stall or come to a halt entirely. Make redundancy and support a priority for such an important project. If you have a small team but need to upgrade, consider hiring a contracting firm for assistance. They can help get you out of challenging circumstances and provide guidance on best practices to prevent future stagnation.
01:06:43.560 Planning is another critical aspect of making upgrades easier. Your team should consider whether to upgrade incrementally from Rails 3 to 4 to 5, or if you want to jump directly from 3 to 5. An incremental approach will make it easier to manage deprecation warnings, while a direct upgrade may expedite the process. Since our upgrade timeline was significantly lengthy, we believed an incremental approach was best to maintain motivation. Additionally, you'll need to evaluate whether to use long-running feature branches or allow booting in multiple Rails versions, as adding additional CI builds can enhance future upgrade processes.
01:09:37.800 Investing in your team's upgrade process will yield significant returns down the line. One effective strategy is to tackle deprecation warnings as they arise instead of leaving them unresolved. By addressing issues like the alias method chain—deprecated in 5.0 and removed in 5.1—you eliminate future roadblocks. You should also implement linting practices to catch regressions, ensuring your team does not introduce new problems into the codebase that will require correcting after each upgrade. Once you have successfully upgraded to the most recent Rails version, it's vital to devise a future upgrade plan. Determine how frequently your team will update Rails and whether you want to test new releases proactively to ensure smoother upgrades.
01:13:13.900 If you invest in upgrading, you'll also want to establish dual-booting CI systems to test different Rails versions simultaneously. As you work through your upgrades, it’s essential to steer clear of certain pitfalls that may lead to regret in the future. The choice to fork Rails and deviate from the framework was one of GitHub's most expensive decisions regarding our application. This choice had profound implications that compounded over time, ultimately causing a seven-year upgrade process. If you absolutely must fork Rails, closely track upstream changes and only use it to backport critical fixes or features needed for your application.
01:17:09.300 Adding features to your fork that you do not intend to contribute back can lead to greater difficulties during future upgrades, as your version diverges from the upstream Rails repository. If your Rails version becomes unsupported, consider using Rails Long-Term Support, as they can maintain your fork and ensure accurate security patches. Falling behind on upgrades may lead to potential mistakes, leaving your application vulnerable. Ideally, your application should closely track the latest Rails version to minimize surprises during security updates.
01:20:42.800 In addition, regularly updating your dependencies is crucial to align with Rails requirements. This will help you avoid being stuck with gems that are obsolete or unsupported, which can make upgrades more complicated. At GitHub, we had a challenging experience while upgrading Sinatra due to its compatibility with Rails. Regularly upgrading your dependencies can prevent you from ending up with abandoned gems. Besides this, be cautious of using Rails private APIs, as they can change unexpectedly without prior notice. Public APIs are documented, and you'll be informed about any deprecations or changes. Lastly, refraining from monkey patching is advisable; if necessary, ensure that the patch is limited to the Rails version it addresses and is well documented.
01:24:30.400 We had a significant number of outdated monkey patches from Rails 3, which made it challenging to clean them up. These choices contributed to the accumulation of technical debt within our applications. I want to instill confidence in you that you too can accomplish an upgrade. Although difficult, there are resources and support available. Remember that you can pay down your technical debt incrementally. Start with classes that utilize private Rails APIs, document or eliminate your monkey patches, and actively work to upgrade outdated dependencies.
01:28:58.640 Keep in mind that Rails upgrades are an ongoing marathon, not a sprint. If you and your team need to steadily address the buildup of technical debt, you can achieve the eventual upgrade. It’s also crucial to remember that you’re not in this alone. Many have already navigated their Rails upgrades, and countless will follow in your footsteps. Seek out individuals who are going through similar upgrades or have accomplished them in the past; they can be an invaluable source of support. I drew encouragement from Shopify engineers who had managed their own tough upgrade, demonstrating that it could be done.
01:33:02.350 When upgrades are completed, the benefits are worth it. Though challenging, don't let fear hold you back from upgrading. After all, proper upgrades produce numerous advantages beyond merely avoiding past pitfalls. Apart from improved security, upgrading leads to more manageable dependencies, enhanced APIs, and access to new libraries. Major version updates allow us to rethink and refine how previous features are designed and ensure seamless integration of new functionalities. For example, Rails 6 includes improved database handling, making it significantly easier to manage multiple databases compared to version 5.
01:37:45.830 Upgrading not only streamlines the development process but enables contributions upstream. Fixing bugs and adding features in outdated versions can be highly restrictive. As DHH noted, frameworks thrive through extraction—we build Rails around existing needs. If those needs are set back in outdated Rails versions, making a case for necessary extractions becomes nearly impossible. Thus, being part of this framework evolution is a strong motivation for many developers, including myself, who invested significant time upgrading Rails at GitHub.
01:41:20.600 Today, GitHub engineers have made countless contributions of over 60 pull requests that enhance performance and functionality in Rails 6. We are moving beyond merely using Rails or developing a Rails-based application; we are actively shaping the framework and pioneering its future. By extracting code from GitHub and helping establish new features that will aid in scaling applications, we reinforce our commitment to Rails for the long haul. This virtuous cycle enhances both the Rails framework and GitHub, augmenting the benefits for the entire community.
01:45:20.000 This investment in Rails upgrading is pivotal for our future at GitHub. Our journey initiated in 2007, when Rails was born, and it took until now—11 years later—to finally track master closely. Getting from starting the Rails 3.0 upgrade to the completion of the 5.2 upgrade took seven years. However, the future shines bright, and I’m excited to witness how much more we can extract and build with GitHub and Rails together over the next decade. As long as GitHub is utilizing Rails, we will consistently invest in the framework and uphold our commitment to supporting it.
01:49:30.000 Thank you!
See Slides on speakerdeck.com
Edit
Explore all talks recorded at Ruby on Ice 2019
+8