00:00:12.080
Hello, my name is Kir Shatrov. I come from Russia, but I now live in Finland. I work as a maintainer on Capistrano and I am also a Rails committer. I work at an awesome Rails shop called Evil Merchants.
00:00:20.480
We do Rails development and consulting. We developed Groupon Russia since its early days, and currently, we are doing a lot of work for eBay, which is my current project. Today, I will talk about performance regressions in Rails.
00:00:43.120
I’ll explain what performance regression is, how to catch them, how to solve them, and what tricks you can use to avoid them. I will also discuss the tool I have been working on for the last few months, which automatically catches performance regressions in Rails. Finally, I will analyze the current state of this tool and discuss its future.
00:01:14.960
Before we start, let’s define what a performance regression is. A performance regression is a situation where the software still functions correctly but takes more time to compute or consumes more memory compared to previous versions.
00:01:38.000
In the context of Rails, imagine a scenario where, after upgrading to a shiny new version, the page load time increases from 250 milliseconds to four seconds. That's quite significant. This actually happened in Rails 3.2.13 due to a bug in Sprockets.
00:02:07.520
Now, I want to share with you a story about why we should care about performance regressions in Rails and why this issue affects all of us. Right after the release of Rails 4.2 release candidate 1, the Discourse team, which has a large application, decided to upgrade and try Rails 4.2.
00:02:30.960
They were excited about all the active record patches by Aaron and expected a performance boost. However, they discovered that Rails 4.2 RC1 was actually two times slower than Rails 4.1, which was surprising considering the expectations. Fortunately, this was just a release candidate.
00:03:09.840
After applying numerous patches by both the Discourse team and the Rails team, they managed to make Rails 4.2 RC3 perform as fast as Rails 4.1, if not faster. So, it might seem like a hassle for Rails committers and contributors, but what about regular developers?
00:03:27.519
It is a problem for everyone because a faster framework means a faster application. By speeding up Active Record with adequate patches and improvements, your Rails app can benefit significantly since Active Record is usually tightly integrated with most Rails applications.
00:04:13.799
To get a better understanding, let’s review some examples from Rails commits that have affected performance and how we solved these regressions. First, we need to consider what metrics we focus on when discussing performance regressions in Rails.
00:04:46.880
The two primary metrics are timing and allocations. Timing refers to the amount of time that a method or a piece of code takes to execute. Reducing this timing makes our methods faster and, consequently, our applications faster.
00:05:19.759
An example of measuring timing is straightforward. On the other hand, we also want to reduce the number of allocated objects in the Ruby interpreter's memory because garbage collection is a costly operation in Ruby. Allocating fewer objects minimizes the workload for the garbage collector.
00:05:41.599
Many performance issues can be alleviated just by understanding how basic data primitives in Ruby work, like strings, hashes, blocks, and arrays. Now, let’s dive into specific examples of Rails comments that worked with strings and resolved performance issues.
00:06:02.560
One noteworthy comment by Aaron Patterson highlighted a problem with string concatenation using the '+' method. This approach was inefficient because interpolation is significantly faster, and by using interpolation, we avoid the unnecessary allocation of an extra string.
00:06:43.440
Another interesting case comes from Sam Saffron, who modified a piece of code that was called on every Active Record attribute read. By moving a string constant outside of the method and freezing it, he reduced the number of allocated objects by 5000 on the Discourse front-end application.
00:07:02.480
There’s also a comment not from Rails itself but from the Direct Gem. This code allocated new strings for variables on every request, which was inefficient. By applying the same method of moving strings into constants, we significantly reduced allocated objects.
00:07:38.560
For instance, this approach lowered allocated objects per request by 40—almost half the total objects inside the gem. Such popularity of performance fixes indicates how impactful these optimizations can be.
00:08:05.599
In Active Support, we encountered an issue with time formatting that used 'gsub' four times, which allocated a new string each time. This could have been resolved with just one regex to handle the replacements, enhancing performance.
00:08:43.440
Another case in Ruby 2.2 involved switching from using regular expressions for string replacements to a process that is 1.5 times faster. This optimally utilized the up-to-date Ruby regex engine and eliminated unnecessary conversion steps.
00:09:13.760
Next, we have examples regarding hashes. An example from URL4 helper allocated a new array to override just one hash option. Instead of this method, we could employ the 'Hash#[]=' method to set a new value without creating a new array.
00:09:52.320
By avoiding unnecessary hash allocations, we can improve the code’s efficiency. Aaron Patterson also showcased that we need not allocate new hashes either; we can simply set a value to an existing hash.
00:10:35.360
Effective performance tuning often concerns optimizing 'hot code'—that is, code that runs frequently on every request. Optimizing such code provides substantial benefits, as Rails operates across various applications heavily.
00:11:19.840
Now, let's discuss how we can track Rails’ performance changes. The idea was proposed during a Rails base camp, suggesting building a service that tracks performance changes. This would help optimize time for Rails contributors.
00:11:53.440
To effectively track these changes, we would need some benchmarks. Are there existing benchmarks in Rails? Yes, there's a remarkable benchmark within the Discourse application that accurately measures performance when upgrading Rails versions.
00:12:22.480
In this benchmark, Discourse sets up the database, populates it with records, starts Unicorn in a production environment, and makes numerous requests using Apache Bench to analyze timing and memory usage.
00:12:51.200
This benchmarking method was utilized even by Koichi to optimize garbage collection algorithms in MRI. It demonstrates a fantastic way to ensure any new Rails version is indeed an improvement performance-wise.
00:13:14.880
However, to properly measure performance changes, we want to assess Rails components separately. I compiled a list of components such as Action Controller and Action View, which I ran benchmarks against various major Rails versions.
00:13:43.440
Using D3.js for visualization, I created a static page to showcase these benchmarks. The higher the bar indicates better performance, and you can see trends reflecting how Rails has become faster over time.
00:14:40.080
For example, one of the benchmarks focuses on ActiveRecord finders. In this benchmark, the creation of a single user record is done, and after that, there’s an attempt to retrieve it using the `find` method, which is done 100 times, serving as a micro-benchmark.
00:15:06.000
I also want to stress the importance of running accurate benchmarks in an environment close to production. For example, when making requests to a small Rails app, the initial version of the benchmark showed unexpected results, making me realize that the environment needed alterations.
00:15:37.760
After making the necessary adjustments, the benchmark results began to reflect true real-life performance. Therefore, having a set of benchmarks is crucial, and it’s time to automate their execution against new versions of Rails.
00:16:05.920
Automation is widely embraced in the Ruby community; for instance, Travis CI automates builds and tests while Hound CI provides code review based on Ruby guidelines. Implementing such automation for benchmarks is the next logical step.
00:16:32.800
I started creating a prototype of this service in January, which was relatively simple at first, running just two benchmarks against updates in the Rails repository. However, as time went on, I envisioned it evolving into something akin to Travis CI with more complex infrastructure.
00:17:00.960
At that same time, I discovered a service called RubyBench, which automates benchmark testing for Ruby MRI with nice infrastructure powered by Docker and dedicated hardware, ensuring accurate and consistent benchmark results.
00:17:44.559
After connecting with the authors of RubyBench, we decided to integrate Rails support into RubyBench. Each new commit to the Ruby repository triggers a webhook that starts a background job within our web application.
00:18:16.080
In this setup, a new Docker container is created with the exact version of Ruby and the necessary benchmarks, which are then executed. After finishing the benchmarks, the results are reported back and displayed on the web application for review.
00:18:46.720
The web application serves as a simple Rails 4.2 app hosted on Heroku using delayed job. It allows users to choose benchmarks from the sidebar and visualize the results on the charts displayed.
00:19:15.920
Regarding the architecture of RubyBench, there are three components available on GitHub: a web application, a repository for Docker files, and a repository for the benchmark suite itself. Docker is used to isolate environments, making it much easier to manage dependencies.
00:19:54.720
We have a benchmark suite that includes many micro-benchmarks pertaining to both Rails and Ruby. We are actively working on adding new benchmarks for Rails as we continue to enhance Active Record and Action Controller and Action Dispatch.
00:20:30.560
We run builds for Rails with each new version release and are looking forward to launching per commit benchmarks shortly. Since November, there have been numerous benchmarks run, making it meaningful.
00:20:55.440
In fact, we have identified a performance regression in Ruby core. The regression was located quickly, and the team successfully addressed it the next day, showcasing how important these tracking methods can be.
00:21:31.600
Looking forward, we aim to establish benchmarks for every pull request, ensuring that performance feedback is relayed to the author while generating weekly reports on performance changes across the framework.
00:22:02.920
We are also planning to run these benchmarks against JRuby, as we have already developed a set of benchmarks for Ruby that can be easily adapted.
00:22:40.960
Today, we have extensively covered the concept of performance regressions, how they manifest in Rails, methods to identify and solve them, and the tools available for this work. We've also explored the importance of maintaining a robust benchmarking suite and automating performance tracking.
00:23:17.440
But you may wonder: what if you're not a Rails contributor and how can these insights assist you with your applications? Tracking your own app's performance is entirely feasible.
00:23:44.920
Start by studying how Ruby’s basic objects function and managing them properly. A recommended resource is 'Ruby Under a Microscope' by Pat Shaughnessy. Additionally, create a Discourse-style benchmark for your Rails app to test your endpoints during development.
00:24:23.520
You can automate running these benchmarks after each commit. Also, subscribe to the Rails weekly mailing list to stay informed about the latest features and regression fixes.
00:25:00.960
I have also included links in my talk regarding my benchmarks for Rails, detailing the benchmarks running since Rails 3.2 to 4.2, as well as the prototype of the RailsPerf service that I’ve created. The RubyBench components are also accessible on GitHub.
00:25:27.840
Please feel free to follow me on Twitter or GitHub, and do not hesitate to reach out if you're interested in Rails benchmarks or contributing in any way. I deeply appreciate your time and attention today.
00:25:55.720
Thank you for attending this talk!
