Busting Performance Bottlenecks: Improving Boot Time by 60%

Talks

Julian Nadeau

#developer-experience-dx

#profiling

Busting Performance Bottlenecks: Improving Boot Time by 60%

by Julian Nadeau

The talk by Julian Nadeau at RubyKaigi 2017, titled "Busting Performance Bottlenecks: Improving Boot Time by 60%", addresses the challenges faced by developers due to lengthy application boot times and presents strategies to optimize performance. Boot time is crucial, as it affects developers' productivity and negatively impacts perceptions of the application. Nadeau shares insights from his experience at Shopify, where improving boot time became essential given the vast number of developers and repeated tasks performed daily. Key points discussed in the presentation include:

Definition of Boot Time: Application boot time is defined as the duration taken for an application to start post-command execution. Even moderate boot times can accumulate significantly with frequent testing and command execution.
Impact of Boot Time on Developer Experience: Lengthy boot times frustrate developers, leading to decreased productivity, trust issues, and potential financial losses for organizations. Nadeau notes that Shopify faced boot times exceeding 25 seconds, stressing the importance of solutions to mitigate this issue.
Identified Performance Bottlenecks: The main areas causing delays included:
- Compilation of Ruby Bytecode: Dynamic compilation increases boot time as applications grow. Transitioning to Ruby 2.3’s instruction sequence methods allows caching of bytecode, thus expediting the process.
- Serialization: Previous use of YAML for configurations resulted in slower load times. Switching to faster serialization techniques using Ruby’s Marshal and MessagePack improved efficiency.
- Constant Lookup: The expensive and complex process of constant lookup can exacerbate delays. Implementing caching strategies can significantly reduce lookup time complexities.
Boot Snap Gem: Nadeau introduces the Boot Snap gem, designed to cache files improving boot time efficiency. Notable benefits include:
- Efficient caching of stable files, preventing unnecessary recompilation.
- Reduction of application boot time; for instance, Discourse's boot time improved from 6 seconds to 3 seconds.
- Shopify’s core application saw a decrease from 25 seconds to 6.5 seconds.
Optimizations in Bundler: Nadeau discusses performance enhancements in Bundler 1.15, which collectively reduced boot time by approximately 60%, enhancing efficiency across Ruby applications.

In conclusion, embracing effective caching and optimization strategies is crucial for improving boot times and the overall developer experience. Nadeau encourages attendees to explore these methods and contribute to the broader Ruby community in pursuit of better performance.

This talk provides valuable insights for developers seeking to enhance their applications' startup speeds using practical techniques and shared community knowledge.

00:00:00.120 The next speaker is Julian Nadeau. He's coming from Shopify, where he is a production engineer. Our talk today is titled 'Busting Performance Bottlenecks: Improving Boot Time by 60%.' I can understand the struggles of working with lengthy application boot times, as they often cause developers to lose context and view applications negatively.

00:00:16.770 Let me tell you a little about Shopify. Shopify is a Canadian company founded to sell snowboards online back in 2004 by our CEO and Founder, Toby. At the time, the software available for e-commerce was lacking, and it could cost around one hundred thousand dollars just to set up a payment gateway. Therefore, Toby utilized Ruby, particularly Ruby on Rails, starting with version 0.18 to create the Shopify application. Today, Shopify powers over five hundred thousand merchants across the globe, with many merchants situated in Japan, thanks to our recently opened Tokyo office. We now boast a multi-million line code base running on Rails 5.1 with about a thousand engineers at work. This topic is especially dear to us because enhancing boot time has significant implications for our time and money.

00:00:42.000 So, what is application boot time? We're going to start by discussing that. We'll touch on why it matters, how it impacts you, and how it affects your organization. Specifically, we'll talk about how it impacted our Shopify application. Following that, we’ll discuss the areas of impact and how we managed to mitigate those issues, particularly with the Boot Snap gem. Some of you may be familiar with Boot Snap as it’s a gem we've recently released, and I also worked on performance enhancements within Bundler 1.15. Most of this presentation will center around application boot time and, more importantly, the straightforward but powerful techniques we utilized to identify inefficiencies in our own code. My goal is to arm you with insights so you can return and address similar problems in your own codebase. Additionally, I've prepared a website and GitHub repository containing demo information; I encourage you to reference those resources throughout the session or afterward.

00:01:42.000 Let's define application boot time simply as the duration it takes for your application to start after the start command has been issued. For instance, when I run 'bin/rails server' in a Rails application, it may take 8 seconds for the application to fully initialize. Therefore, an 8-second boot time may not seem excessive, but it becomes significant when you consider this is the boot time every time you run tests. Most of us write tests, leading to running numerous commands such as 'bin/rails test' multiple times each day. The total time quickly accumulates.

00:03:04.319 Why does this matter? A developer's experience with your application hinges on how easy it is to work with it. For Shopify, our product involves an e-commerce platform, but the developer experience is also a critical product. I'm part of a production engineering team that runs and maintains the product affecting our developer experience. Hence, we must provide our developers with the same quality experience we deliver to our merchants. Lengthy application boot times can hinder this experience, causing frustration and a negative perception. Negative interactions lead to decreased productivity and trust among developers, which ultimately results in wasted time and diminished resources, impacting our bottom line. At Shopify, we faced application boot times for tests exceeding 25 seconds, alongside server times. With approximately a thousand developers executing countless commands daily, we were potentially losing tens of thousands of dollars each day due to inefficiency.

00:05:02.080 With these challenges in mind, let’s explore the areas of impact we identified within our application before diving into how you can detect these inefficiencies yourself. The main areas include compiling, serialization, and constant lookup. Compiling in Ruby is done dynamically, but as the size of applications grow, more code needs to be compiled, particularly with required statements at the top of each file. This results in increased boot time relative to your application's size. Fortunately, Ruby 2.3 introduced a few instruction sequence methods, which we'll cover later.

00:06:01.270 Serialization is another area where we observed inefficiencies. In our case, we utilized YAML for our test fixtures and configurations, but it caused significant serialization time during the boot process. We addressed this issue by compiling and serializing the contents of our MO files using Ruby's Marshal and MessagePack. I'll elaborate on this process shortly. Another significant bottleneck we observed was in constant lookup. It's vital to note that looking up a constant can become highly expensive, with O(n²) complexity as you traverse the load path for each required item. As your application and load path grow, this inefficiency compounds, affecting overall performance. However, we can alleviate this by caching the first iteration of the lookup to transform O(n²) complexity into O(1) for cache hits or misses, which I will discuss now.

00:08:17.110 Let’s discuss how caching operates within Boot Snap. First, is everyone familiar with Boot Snap? All right, since most of you are acquainted with it, I’ll start by explaining the basic caching protocol we deploy. We conduct stability checks determining if a file or constant is stable or volatile, meaning it may change or not. In essence, we treat Ruby installs and gems as stable since they typically don’t change in the file system. While Ruby core contributors might modify Ruby often, this is less of a consideration for most applications. Once we identify a file as stable, we cache it in a designated folder using Ruby iseq (instruction sequence), which we do similarly with other files, but with an additional cache key. In the stable cache, we retain it indefinitely, while in the volatile cache, we keep it for merely 30 seconds. This strategy allows us to cache the application during boot while ensuring we identify instances where compilation had caused delays.

00:14:21.860 We've noted issues with compilation and as a solution, began compiling bytecode. Since Ruby 1.9, Ruby has translated source into bytecode, improving execution speed dramatically. Following Ruby 2.3’s introduction of the instruction sequence (ISEQ), we effectively bypassed a costly compilation step that recurs every time we load a file. If we continually require a file, it undergoes repeated compilation unless we cache it after the first iteration. Utilizing Boot Snap enables us to cache it in its ISEC form, preventing unnecessary recompilation and significantly speeding up our process. The result? Drastic performance improvements.

00:19:34.500 Another critical area we optimized was the serialization of test fixtures and configuration files. We found that using YAML for loading files is rather slow, even when leveraging fast C extensions. When we compared the performance, we discovered that MessagePack or marshaling provided substantially faster serialization than the YAML approach. Given that YAML is inherently dynamic, it can create overhead due to merging and importing processes. Implementing caching under volatility checks enabled us to enhance boot time effectively. Additionally, we redefined the Kernel.require and load methods to prioritize using the cache first. This way, we eliminated the dual penalty of constant lookups and expensive compilation, streamlining file access and reducing system calls significantly.

00:23:02.700 The Boot Snap library is simple to integrate into your Rails application. Just add it to your Gemfile, ensuring you set `require: false` and include it right after the Bundler setup. Doing this can yield a significant boost in boot time. For instance, Discourse, a popular Rails application, experienced a boot time reduction from 6 seconds to 3 seconds, while one of our smaller internal applications decreased from 3.6 seconds to 1.8 seconds. The core Shopify platform, a large codebase, improved its boot time from approximately 25 seconds to just over 6.5 seconds. The impact of these changes is substantial, and we've received positive feedback from others who witnessed similar gains in their projects, reducing long boot times by up to 60 seconds.

00:26:12.980 Let's shift our focus to Bundler; a tool almost every Ruby developer employs for managing dependencies. Surprisingly, it contributed about a second to our boot time. While not an excessive duration, it mattered when we aimed to reduce our boot time to under 6 seconds. We identified several optimizations released in Bundler 1.15, resulting in a 60% decrease in boot time. Post-implementation, we watched the duration drop to about 300 milliseconds. Upgrading to Bundler 1.15 has proven reliable and useful across Shopify and the wider community. Our success with these updates encourages continual improvement and maintenance efforts to ensure performance remains optimal.

00:31:10.780 Another critical improvement revolved around gem specification validation and how Gemfiles handle dependencies. Since Git gems don’t actually compose native code, they behave like path-based gems. Bundler clones repositories and builds gems from scratch, yet validating gem specifications can be slow. Implementing validation only during the initial load optimizes the process. Moreover, for serializing Gemfiles, we transformed the array of dependencies into a hashed structure, reducing lookup complexity from O(n²) to a more efficient O(1). This shift not only improved efficiency but also slashed the time spent on these operations.

00:36:00.000 In summary, Boot Snap significantly bolstered our performance by implementing effective caching, optimization of constant lookups, and serialization techniques. The enhancements we made to Bundler further optimized our Ruby applications. Encouragingly, in collaboration with the Ruby community, we hope to enhance boot performance within the ecosystem as a whole. I invite you all to explore these methods and share in the open-source community effort, contributing to a faster, more efficient Ruby experience. Thank you for your time, and I'm open to any questions you might have.