Get Your Ass to 1.9

Talks

Get Your Ass to 1.9

Nic Benders

1 talk

#ruby-1-9

#rails-upgrades

#application-performance-monitoring

#continuous-deployment

#legacy-applications

#bundler

#testing-strategies

Get Your Ass to 1.9

by Nic Benders

In the presentation "Get Your Ass to 1.9" at Ruby on Ales 2013, Nic Benders and Ralph, both team members from New Relic, discuss the transition from Ruby 1.8 to Ruby 1.9 and share their experiences during this critical upgrade. The session aims to guide those still using Ruby 1.8, as support for it is dwindling, particularly highlighting performance improvements and support factors pushing for migration.

Key Points Discussed:

- Introduction to Upgrade: The presentation opens with an acknowledgment that many production apps still use Ruby 1.8 and emphasize New Relic's prior challenges running on Ruby 1.9.

- Collaboration Importance: The success of the upgrade relied on collaboration between the site engineering and core app teams, where resource sharing was crucial.

- Annotated Project: A project monitored the application performance metrics that indicated lagging performance, prompting the upgrade. Benders and Ralph underline that their sizable team of around 25 engineers contributed to the process.

- Reasons for Hesitation: Attendees are prompted to consider reasons holding them back from migration, including concerns about substantial legacy applications and the complexity of the upgrade amid ongoing developments.

- Upgrade Strategy: The speakers outline their strategy to upgrade all at once using feature flags, while also integrating community tools like rbenv to manage multiple Ruby versions effectively.

- Testing and Compatibility: They emphasize maintaining a solid testing framework to catch potential issues early, especially with third-party gems and legacy code challenges encountered throughout the upgrade process.

- Performance Improvements: Post-upgrade, the team observed significant enhancements in application performance, including a 75% reduction in workload for garbage collection and reduced CPU usage, which positively impacted user experience.

- Lessons Learned: The presentation concludes with vital takeaways on the importance of collaboration, testing comprehensiveness, and preparing to tackle future upgrades, including transitioning to Ruby and Rails 2.0.

00:00:20.640 Today, we're going to discuss Ruby 1.9. While Ruby 1.9 is not new, I suspect that many of you with large production apps still haven't migrated. A year ago, New Relic was not running on Ruby 1.9 either. It was a challenge, but today we're here to show you how we accomplished it and how you might begin the transition yourself. It's crucial to understand that the patterns we discuss today are applicable not only to Ruby upgrades but also to Rails and other upgrades. It's time to upgrade to 2.0. I'm Nick from the site engineering team at New Relic.

00:01:11.920 I'm Ralph, and I work on the core app team at New Relic. We built the significant Rails app that we’ll be focusing on today, ensuring it runs smoothly in a production environment. The collaboration between our two teams, especially sharing resources and knowledge, was key during this process. The site engineering team is not isolated from development; although they often are, they are skilled Rubyists who understand Ruby. The development team is familiar with operational aspects, enabling us to overcome hurdles together.

00:01:36.400 As Kobe briefly introduced, New Relic offers application performance monitoring for various languages, including Ruby, Python, Java, and .NET. Our monitoring tools indicated that our app was slow, highlighting the need for significant improvements. Ralph and I contributed to this upgrade project, but we were not the only contributors; around 25 people worked on that app, and several teams, including the Ruby agent team and the site engineering team, were involved. Notably, four individuals played crucial roles: John Guyman, our spiritual leader Bill Kaiser, Jonathan Owens, and Andrew Bloomgarden. They're busy with hard work while we share our experiences here.

00:02:21.760 You may already be aware of the syntax changes, performance advantages, and other compelling reasons to upgrade to Ruby 1.9. Let's take a moment to encourage those who may still rely on Ruby 1.8 to think about making the switch. Is anyone still using Ruby 1.8? That's alright, but it's important to note that Ruby 1.9 has been available for a long time and is fully supported. Meanwhile, support for Ruby 1.8 is coming to an end, and if you’re using it in production, you might be relying on Ruby Enterprise Edition due to performance needs. This version is no longer actively supported, and users submitting bugs feel the impact of that lack of attention. So, things are shifting towards an inevitable end of life for Ruby 1.8.

00:03:05.920 This data from New Relic illustrates the situation: many Ruby users are transitioning to Ruby 1.9 or even Ruby 2.0. However, substantial numbers of applications still run on Ruby 1.8, indicating that there's still work to be done. Let’s remind ourselves of the reasons holding people back from upgrading to Ruby 1.9. Common objections include concerns about the size of their legacy applications and fear of the difficulties involved in upgrading while still maintaining ongoing development. Our app, too, is sizable and has legacy elements; for reference, our first commit was made in July 2007 with Rails version 1.2.3. Since then, we have consistently upgraded and are now running on Rails 3.0.

00:04:05.600 At the time we switched to Ruby 1.9, our codebase included around seventy thousand lines of code and nearly sixty thousand lines of tests. We were running Ruby Enterprise Edition, which had become increasingly outdated. Our main worry wasn't just about the size of the codebase; we were concerned about halting ongoing development. Shopify’s migration, for instance, involved dedicating multiple engineers who couldn’t ship features during that time, a scenario we sought to avoid. Initially, we thought about splitting our app to make it simpler to upgrade, but we mostly ended up just talking without significant action.

00:05:10.560 Eventually, we concluded that we would have to take the plunge and attempt the upgrade all at once, freezing our ongoing work to focus on the transition. John Guyman, who is here today, initiated a project aiming to convert the codebase in just one day — albeit, it didn't go as smoothly as planned. After several attempts to create branches, we found that merging back changes was extremely painful. Our branch needed constant updates against master, incorporating its own set of tests and changes that had been made. This only highlighted how difficult it was to maintain that approach.

00:06:09.120 With this realization, we acknowledged that we needed a continuous deployment standard: everything stays on master. By utilizing feature flags, we could continuously integrate any changes into master at all times, regardless of whether they were live or not. Thus, we established a system where we had a branch for Ruby 1.9 and a parallel CI job to monitor its health. We employed Jenkins to run two jobs simultaneously: one for Ruby 1.8 and another for Ruby 1.9, with early results from the Ruby 1.9 job being quite enlightening, indicating problems even before we formally completed the upgrade.

00:07:18.080 The next step was to work on smaller components within the application that would allow us to ease into the upgrade. However, a significant portion of our workload was focused on ensuring our foundational systems were in order. A critical task involved ensuring that our app could effectively run multiple Ruby versions side by side. Initially, we needed to upgrade our Rails version to at least 2.3 to ensure compatibility with Ruby 1.9. All the while, ensuring security was another important consideration to improve our application's stability.

00:08:18.560 Additionally, we needed an infrastructure in place for switching Ruby versions. Thankfully, the community provided excellent tools for this, particularly rbenv, which we integrated into our workstations, CI systems, staging, and production environments. As we migrated our production environment to use rbenv for managing Ruby versions, it enabled us to switch back and forth seamlessly across our stack. Crucially, the introduction of Bundler significantly aided in managing the Ruby gems, ensuring compatibility between Ruby 1.8 and Ruby 1.9.

00:09:05.680 To ensure a successful rollout, we had to maintain solid test coverage that would shield us from unforeseen complications following changes. Prior to the upgrade, we had 76% test coverage, which was better than 50%, but we quickly uncovered gaps in our testing framework, exposing areas in need of improvement. With that understanding, we pushed forward into the more challenging aspects of ensuring our complete test suite was operational under Ruby 1.9.

00:10:07.280 A significant challenge was handling third-party gems and plugins that were incompatible with Ruby 1.9. Specifically, we had to replace Ruport and Mongrel, which were not compatible with the newer Ruby version. Instead, we transitioned to Unicorn. We worked through all these changes on our master environment, pushing everything to production incrementally. This approach helped us slowly shed our dependencies on Ruby 1.8's environment while we concurrently ran on Ruby 1.9.

00:11:02.400 One of the biggest time investments during this transition was addressing test failures. The initial work began in March, and we reached completion by early September 2011. This didn’t include the time we spent originally resisting the need for the upgrade. We kept the project moving without halting everything else in the pipeline. Over time, new commits were consistently being checked against Ruby 1.9 compatibility, ultimately preventing further regressions.

00:12:09.840 We learned that Ruby 1.9 introduced several behavioral changes—small in isolation but significant in a larger application context. This metric meant we needed to carefully check for compatibility while testing. One of the biggest hurdles was sorting out complex syntax and structural issues we had throughout our codebase. By applying various strategies, we were able to tackle these nuanced changes as they arose.

00:13:00.000 As we delved into our upgrade project, we began encountering peculiar issues—like load paths suddenly not functioning as expected under Ruby 1.9. Each of our support scripts and background processes broke down as we moved along the upgrade path. We spent considerable efforts rectifying these issues as they appeared across our extensive codebase. We ensured our code remained flexible, accommodating both Ruby versions during the rollout, while keeping the application as stable as possible.

00:14:05.680 In addition to focusing on gem compatibility and making the necessary syntax changes, we were also mindful of employing best practices throughout our upgrade process. For instance, the introduction of ordered hashes in Ruby 1.9 forced us to reevaluate how we were handling collections throughout the application. Countless smaller bugs emerged based on our app's previous dependencies on the behavior of earlier Ruby versions.

00:15:10.800 Ultimately, after ensuring our codebase was clean and functional across versions, we rolled into the enhanced performance metrics. We eventually saw a dramatic uptick in performance—a testament to our effort and patience during the upgrade journey. The improvements in garbage collection times reflected a 75% reduction in workload, alongside a steep drop in CPU usage on our app servers. This level of optimization contributed significantly to the efficiency and speed that user experience underwent.

00:16:24.800 Reflecting on the entire process, we realized we learned valuable lessons along the way. Among them was the critical importance of collaboration and consistent communication across both teams involved in the upgrade. These experiences positioned us to apply similar practices for future upgrades as we look ahead to upgrading to Rails and Ruby 2.0.

00:17:39.200 Both Andrew and Julian are team members who contributed to the Rails 3 upgrade and faced their own unique challenges. They’ll be discussing their methodologies at RailsConf this year, so keep an eye out for their sessions. As we continuously work to enhance our applications, hiring is also something we prioritize within the engineering teams—so if you're interested, we'd love to chat.

00:18:33.760 To conclude, we appreciate your attention and look forward to any questions you may have regarding our upgrade experiences or broader discussions about Ruby and Rails. Our journey towards Ruby 1.9 might have had its challenges, but the outcomes significantly improved our application, and we hope to encourage others to follow this transformative path.

00:19:26.640 Thank you very much!

Ruby on Ales 2013