Building RailsPerf, a toolkit to detect performance regressions

by Kir Shatrov

In this talk by Kir Shatrov at RailsConf 2015, the focus is on performance regressions in Rails applications and the development of RailsPerf, a tool designed to detect these regressions. Performance regressions occur when software operates correctly but becomes slower or uses more resources than previous versions. Shatrov highlights several significant points related to this issue:

Definition and Impact: A performance regression is characterized by increased page load times or memory consumption despite correct functionality. An example discussed is the upgrade to Rails 4.2, which resulted in slower performance for the Discourse application compared to Rails 4.1.
Performance Metrics: The two primary metrics for evaluating performance regressions are timing (how long code takes to execute) and allocations (the number of objects created in memory). Reducing both can significantly enhance application efficiency.
Key Examples: Numerous cases demonstrate how small code optimizations can lead to substantial performance improvements, such as utilizing string interpolation over concatenation to decrease unnecessary object allocations.
Benchmarking: To track performance changes in Rails, Shatrov proposes a service concept that incorporates existing benchmarks, especially those observed in the Discourse platform. The benchmarking framework aims to identify performance issues continuously during the development process.
Automation and Integration: Shatrov emphasizes the importance of automating benchmark tests, automating results reporting, and integrating solutions like RubyBench for better infrastructure in Rails performance testing. RailsPerf was conceived as a prototype, evolving into a more sophisticated tool.
Future Directions: The overall goal is to implement benchmarks for every pull request to ensure contributors receive timely feedback on performance changes. This includes exploring benchmarks for other Ruby implementations, such as JRuby.

Concluding with actionable insights, Shatrov encourages developers not involved in Rails contributions to monitor their applications' performance metrics, build proper benchmarks, and stay informed through community resources. With a focus on community contribution and collaboration, RailsPerf represents a significant step towards more efficient Rails application development and performance management.

00:00:12.080 Hello, my name is Kir Shatrov. I come from Russia, but I now live in Finland. I work as a maintainer on Capistrano and I am also a Rails committer. I work at an awesome Rails shop called Evil Merchants.

00:00:20.480 We do Rails development and consulting. We developed Groupon Russia since its early days, and currently, we are doing a lot of work for eBay, which is my current project. Today, I will talk about performance regressions in Rails.

00:00:43.120 I’ll explain what performance regression is, how to catch them, how to solve them, and what tricks you can use to avoid them. I will also discuss the tool I have been working on for the last few months, which automatically catches performance regressions in Rails. Finally, I will analyze the current state of this tool and discuss its future.

00:01:14.960 Before we start, let’s define what a performance regression is. A performance regression is a situation where the software still functions correctly but takes more time to compute or consumes more memory compared to previous versions.

00:01:38.000 In the context of Rails, imagine a scenario where, after upgrading to a shiny new version, the page load time increases from 250 milliseconds to four seconds. That's quite significant. This actually happened in Rails 3.2.13 due to a bug in Sprockets.

00:02:07.520 Now, I want to share with you a story about why we should care about performance regressions in Rails and why this issue affects all of us. Right after the release of Rails 4.2 release candidate 1, the Discourse team, which has a large application, decided to upgrade and try Rails 4.2.

00:02:30.960 They were excited about all the active record patches by Aaron and expected a performance boost. However, they discovered that Rails 4.2 RC1 was actually two times slower than Rails 4.1, which was surprising considering the expectations. Fortunately, this was just a release candidate.

00:03:09.840 After applying numerous patches by both the Discourse team and the Rails team, they managed to make Rails 4.2 RC3 perform as fast as Rails 4.1, if not faster. So, it might seem like a hassle for Rails committers and contributors, but what about regular developers?

00:03:27.519 It is a problem for everyone because a faster framework means a faster application. By speeding up Active Record with adequate patches and improvements, your Rails app can benefit significantly since Active Record is usually tightly integrated with most Rails applications.

00:04:13.799 To get a better understanding, let’s review some examples from Rails commits that have affected performance and how we solved these regressions. First, we need to consider what metrics we focus on when discussing performance regressions in Rails.

00:04:46.880 The two primary metrics are timing and allocations. Timing refers to the amount of time that a method or a piece of code takes to execute. Reducing this timing makes our methods faster and, consequently, our applications faster.

00:05:19.759 An example of measuring timing is straightforward. On the other hand, we also want to reduce the number of allocated objects in the Ruby interpreter's memory because garbage collection is a costly operation in Ruby. Allocating fewer objects minimizes the workload for the garbage collector.

00:05:41.599 Many performance issues can be alleviated just by understanding how basic data primitives in Ruby work, like strings, hashes, blocks, and arrays. Now, let’s dive into specific examples of Rails comments that worked with strings and resolved performance issues.

00:06:02.560 One noteworthy comment by Aaron Patterson highlighted a problem with string concatenation using the '+' method. This approach was inefficient because interpolation is significantly faster, and by using interpolation, we avoid the unnecessary allocation of an extra string.

00:06:43.440 Another interesting case comes from Sam Saffron, who modified a piece of code that was called on every Active Record attribute read. By moving a string constant outside of the method and freezing it, he reduced the number of allocated objects by 5000 on the Discourse front-end application.

00:07:02.480 There’s also a comment not from Rails itself but from the Direct Gem. This code allocated new strings for variables on every request, which was inefficient. By applying the same method of moving strings into constants, we significantly reduced allocated objects.

00:07:38.560 For instance, this approach lowered allocated objects per request by 40—almost half the total objects inside the gem. Such popularity of performance fixes indicates how impactful these optimizations can be.

00:08:05.599 In Active Support, we encountered an issue with time formatting that used 'gsub' four times, which allocated a new string each time. This could have been resolved with just one regex to handle the replacements, enhancing performance.

00:08:43.440 Another case in Ruby 2.2 involved switching from using regular expressions for string replacements to a process that is 1.5 times faster. This optimally utilized the up-to-date Ruby regex engine and eliminated unnecessary conversion steps.

00:09:13.760 Next, we have examples regarding hashes. An example from URL4 helper allocated a new array to override just one hash option. Instead of this method, we could employ the 'Hash#[]=' method to set a new value without creating a new array.

00:09:52.320 By avoiding unnecessary hash allocations, we can improve the code’s efficiency. Aaron Patterson also showcased that we need not allocate new hashes either; we can simply set a value to an existing hash.

00:10:35.360 Effective performance tuning often concerns optimizing 'hot code'—that is, code that runs frequently on every request. Optimizing such code provides substantial benefits, as Rails operates across various applications heavily.

00:11:19.840 Now, let's discuss how we can track Rails’ performance changes. The idea was proposed during a Rails base camp, suggesting building a service that tracks performance changes. This would help optimize time for Rails contributors.

00:11:53.440 To effectively track these changes, we would need some benchmarks. Are there existing benchmarks in Rails? Yes, there's a remarkable benchmark within the Discourse application that accurately measures performance when upgrading Rails versions.

00:12:22.480 In this benchmark, Discourse sets up the database, populates it with records, starts Unicorn in a production environment, and makes numerous requests using Apache Bench to analyze timing and memory usage.

00:12:51.200 This benchmarking method was utilized even by Koichi to optimize garbage collection algorithms in MRI. It demonstrates a fantastic way to ensure any new Rails version is indeed an improvement performance-wise.

00:13:14.880 However, to properly measure performance changes, we want to assess Rails components separately. I compiled a list of components such as Action Controller and Action View, which I ran benchmarks against various major Rails versions.

00:13:43.440 Using D3.js for visualization, I created a static page to showcase these benchmarks. The higher the bar indicates better performance, and you can see trends reflecting how Rails has become faster over time.

00:14:40.080 For example, one of the benchmarks focuses on ActiveRecord finders. In this benchmark, the creation of a single user record is done, and after that, there’s an attempt to retrieve it using the `find` method, which is done 100 times, serving as a micro-benchmark.

00:15:06.000 I also want to stress the importance of running accurate benchmarks in an environment close to production. For example, when making requests to a small Rails app, the initial version of the benchmark showed unexpected results, making me realize that the environment needed alterations.

00:15:37.760 After making the necessary adjustments, the benchmark results began to reflect true real-life performance. Therefore, having a set of benchmarks is crucial, and it’s time to automate their execution against new versions of Rails.

00:16:05.920 Automation is widely embraced in the Ruby community; for instance, Travis CI automates builds and tests while Hound CI provides code review based on Ruby guidelines. Implementing such automation for benchmarks is the next logical step.

00:16:32.800 I started creating a prototype of this service in January, which was relatively simple at first, running just two benchmarks against updates in the Rails repository. However, as time went on, I envisioned it evolving into something akin to Travis CI with more complex infrastructure.

00:17:00.960 At that same time, I discovered a service called RubyBench, which automates benchmark testing for Ruby MRI with nice infrastructure powered by Docker and dedicated hardware, ensuring accurate and consistent benchmark results.

00:17:44.559 After connecting with the authors of RubyBench, we decided to integrate Rails support into RubyBench. Each new commit to the Ruby repository triggers a webhook that starts a background job within our web application.

00:18:16.080 In this setup, a new Docker container is created with the exact version of Ruby and the necessary benchmarks, which are then executed. After finishing the benchmarks, the results are reported back and displayed on the web application for review.

00:18:46.720 The web application serves as a simple Rails 4.2 app hosted on Heroku using delayed job. It allows users to choose benchmarks from the sidebar and visualize the results on the charts displayed.

00:19:15.920 Regarding the architecture of RubyBench, there are three components available on GitHub: a web application, a repository for Docker files, and a repository for the benchmark suite itself. Docker is used to isolate environments, making it much easier to manage dependencies.

00:19:54.720 We have a benchmark suite that includes many micro-benchmarks pertaining to both Rails and Ruby. We are actively working on adding new benchmarks for Rails as we continue to enhance Active Record and Action Controller and Action Dispatch.

00:20:30.560 We run builds for Rails with each new version release and are looking forward to launching per commit benchmarks shortly. Since November, there have been numerous benchmarks run, making it meaningful.

00:20:55.440 In fact, we have identified a performance regression in Ruby core. The regression was located quickly, and the team successfully addressed it the next day, showcasing how important these tracking methods can be.

00:21:31.600 Looking forward, we aim to establish benchmarks for every pull request, ensuring that performance feedback is relayed to the author while generating weekly reports on performance changes across the framework.

00:22:02.920 We are also planning to run these benchmarks against JRuby, as we have already developed a set of benchmarks for Ruby that can be easily adapted.

00:22:40.960 Today, we have extensively covered the concept of performance regressions, how they manifest in Rails, methods to identify and solve them, and the tools available for this work. We've also explored the importance of maintaining a robust benchmarking suite and automating performance tracking.

00:23:17.440 But you may wonder: what if you're not a Rails contributor and how can these insights assist you with your applications? Tracking your own app's performance is entirely feasible.

00:23:44.920 Start by studying how Ruby’s basic objects function and managing them properly. A recommended resource is 'Ruby Under a Microscope' by Pat Shaughnessy. Additionally, create a Discourse-style benchmark for your Rails app to test your endpoints during development.

00:24:23.520 You can automate running these benchmarks after each commit. Also, subscribe to the Rails weekly mailing list to stay informed about the latest features and regression fixes.

00:25:00.960 I have also included links in my talk regarding my benchmarks for Rails, detailing the benchmarks running since Rails 3.2 to 4.2, as well as the prototype of the RailsPerf service that I’ve created. The RubyBench components are also accessible on GitHub.

00:25:27.840 Please feel free to follow me on Twitter or GitHub, and do not hesitate to reach out if you're interested in Rails benchmarks or contributing in any way. I deeply appreciate your time and attention today.

00:25:55.720 Thank you for attending this talk!