Profiling Ruby tests with Swiss precision

Vladimir Dementyev

Vladimir Dementyev

#test-driven-development

Profiling Ruby tests with Swiss precision

by Vladimir Dementyev

In the talk "Profiling Ruby tests with Swiss precision", Vladimir Dementyev addresses the challenges posed by slow Ruby tests within development environments and CI builds, particularly in the Ruby and Rails ecosystem. The presentation focuses on strategies to profile and optimize test performance, emphasizing the importance of quick feedback loops in enhancing developer productivity and overall satisfaction.

Key Points Discussed:

Importance of Profiling Tests:
- While application code performance is critical for user experience, optimizing test performance is equally important to reduce developer waiting times on CI builds.
- Profiling tests can help reduce CI costs, but the effort must be weighed against potential savings. A substantial number of builds are often required for such optimization to be worthwhile.
Feedback Loop Efficiency:
- Slow tests extend the feedback loop, causing frustration and leading to potential bypassing of tests, which impacts reliability in the development process.
- Raviating slow tests restores their utility for debugging, refactoring, and overall software quality assurance.
Profiling Tools and Strategies:
- Built-in profiling support: Popular frameworks like RSpec and Rails now include profiling tools that help identify slow tests. However, deeper issues may require more specialized approaches.
- Test Prof:
- A dedicated Ruby test profiler developed to enhance performance. It includes features to assist in both profiling and efficiently optimizing test suites.
- General Profiling:
- Utilize Ruby profilers such as StackProf and RubyProf to assess overall suite performance, enabling identification of bottlenecks and opportunities for improvements across frequently used components.
Specialized Profilers:
- Factory Doctor: Identifies tests that can be optimized by avoiding unnecessary database calls, leading to performance gains.
- Event Prof: Tags slow tests for further analysis and overall streamlining of the testing process.
- RSPEC Dissect: Measures test setup times to identify redundancy and potential optimization strategies.

Significant Examples:

GitHub Case Study: Dive into a story from a GitHub engineer indicating improved production application performance after profiling test suite bottlenecks.
Test Prof Implementation: The development of Test Prof was driven by practical experience with slow CI runtimes and emphasizes the need for gradual but efficient optimizations in existing testing frameworks.

Main Takeaways:

Optimizing test performance leads to faster feedback loops, increasing developer happiness and productivity within the Ruby community.
Regular profiling and the use of specialized tools like Test Prof can significantly enhance testing processes, making it easier to manage and identify slow-running tests.
Continual profiling can prevent the reemergence of slow tests, ensuring that teams maintain efficient testing infrastructures moving forward.

In conclusion, improving Ruby test performance is crucial for operational efficiency, developer satisfaction, and maintaining the integrity of the development process. Careful application of profiling tools can provide the insights necessary to enhance test suite performance significantly.

00:00:08.480 I don’t want to spoil the party, but I would like to start by talking about Ruby tests, specifically slow Ruby tests, and how to find the best ways to speed them up. I want to kick off my talk with a question: Why do we need to profile tests at all?

00:00:25.039 When it comes to our application code, it's fairly obvious—we want the application to be fast. We aim to achieve a higher number of requests per second because we want our users to be happy with its performance. We prioritize optimizing and profiling our application code for these reasons. However, tests are something different. We primarily use them locally or on continuous integration (CI), and they don't directly affect our users. So, why should we invest in their performance?

00:01:04.640 There are multiple reasons for that, and I’ve collected some responses from the community that I’d like to discuss. This will lead us to understand why we need this and how to approach test profiling. The first answer I received was not the most popular one, but it was quite interesting: to reduce CI costs. However, this sounds questionable.

00:01:24.680 Optimization work requires effort and investment from the team. You're spending work hours on the test suite instead of building features, and it's debatable whether it's worth it. I tried to formulate a way to assess whether we should optimize for cost reduction or not. As a mathematician, I enjoy writing formulas, but don't worry; you don’t need to understand them right now. The idea is that optimizing for reducing CI costs only makes sense if the potential reduction is significant and the effort to achieve it is low.

00:01:50.240 Let's consider an example: if we aim for a 20% optimization, we want to cut the CI build time by 20%. Say we have an average build time of 30 minutes. We can derive how many builds we need over a year to make this optimization worthwhile. If we want to spend at least two days on test optimization, which is a reasonable minimum time, we need to ensure we conduct a substantial number of builds throughout the year. If you calculate the numbers, it shows that you would need over 50,000 builds to make this worth the investment of two days.

00:02:30.280 Investing time just to reduce CI costs can lead to a negative balance because you're wasting your engineers' time. However, that doesn't mean you should avoid optimizing tests for CI cost reduction. It's simply an indicator that your work is slower. CI builds have other positive impacts on the team's work, such as saving time and minimizing distractions.

00:02:47.840 The most common reason for optimizing tests is to make the feedback loop quicker. The feedback loop defines our productivity. We can describe our work as a series of actions we perform during the day, each of which can be split into loop operations: we do something, assess the change, decide what to do next, and then move on.

00:03:05.104 The longer the assessment phase—which includes running tests and waiting for feedback—the fewer actions we can perform during the workday. If that phase becomes too long, it introduces distractions, much like the common scenario of 'while it compiles, let’s play games.' (You know what I mean.) Similar distractions happen with slow tests and CI builds, and we need to avoid this to maintain productivity.

00:03:25.320 This issue with a slow feedback loop can lead to tests becoming unusable locally. When tests are so slow, developers may choose to bypass running them locally, simply pushing their code to CI and hoping for a positive result. Unfortunately, they can be disappointed when they receive an error. Tests then become merely a tool for verification, and their broader usefulness for refactoring and debugging is lost. Tests are essential tools that developers rely on daily, and if they're slow, their utility decreases, leading to diminished productivity.

00:04:11.760 Another less popular but valuable perspective I encountered was that profiling tests can also help profile your production application code. Most of your tests execute the code you push into production, so when profiling tests, you may discover performance degradation in your application code. I recently heard an interesting story from a GitHub engineer about how they improved their production application's performance after identifying a bottleneck in their test suite.

00:04:45.240 To summarize what I've said so far, faster tests lead to happier developers. Particularly in the Ruby community, we focus on developer happiness in software engineering. Ruby is often associated with developer happiness, so we should apply the same philosophy to the code we write. Slow test suites are frequently cited as one of the primary areas developer teams wish to improve upon in their work.

00:05:21.240 I've learned this from my experience working with many clients, including Evil Martians. To clarify, it's time I formally introduce myself. My name is Vladimir Dementyev. I used to learn German, though I’m not sure how well I’ve retained it, particularly in Swiss dialect. You can find me on GitHub, where I work on several projects, including Test Prof.

00:05:53.760 Evil Martians is a consultancy focused on developer tools for startups. We engage in a lot of performance optimization work and pride ourselves on being open-source driven, publishing our work in the form of blog posts, videos, and open-source tools. Now that I’ve introduced myself, I also want to share that I recently published a book on Rails.

00:06:27.559 While this particular book isn't directly related to the topic at hand, I want to mention that I initially outlined another book on Rails testing a couple of years ago, which included a significant section on writing tests that run quickly. That’s essentially what we’re discussing today. One day, I hope to have time to turn that outline into a comprehensive project.

00:06:47.680 Today, we're focusing on existing slow test suites and how to approach them. The gist of my presentation is to demonstrate how to profile these tests—not to fix them, as that’s a different topic—and I only have about 15 minutes left.

00:07:12.080 To summarize the first part of my talk, the goal of profiling tests is to identify frequently used components to optimize them efficiently. This means seeking a balance between the time spent on optimization and the improvements gained from that work. If we continue this analogy, we’ll likely find that the most common responses to the question 'how to profile tests?' are centered around built-in profiling support in our testing frameworks.

00:07:49.560 For instance, RSpec has a profiling option, and Rails has recently introduced a profiling feature as well. These built-in tools can help you identify slow tests, but unfortunately, they often aren't enough to address the core issues. Optimizing merely the slowest test usually requires considerable effort, and we should weigh the effort against the potential improvements and cost reductions.

00:08:30.640 Let's consider a comparison between production and test profiling. Jason, during a recent podcast, highlighted key similarities and differences in this area, and I encourage you to check that out for a more in-depth discussion. But returning to our topic—it's crucial that we focus on frequently used components. If we can optimize these, we’ll gain performance boosts not just for individual tests but for the entire suite.

00:09:05.280 Thus, this is how we came up with the idea of Test Prof—a Ruby test profiler. What is Test Prof? It's a specialized profiler for your Ruby tests with a single focus: to improve performance. While today’s discussion will center on profiling, it also includes tools to assist in fixing tests with minimal refactoring effort.

00:09:38.560 Test Prof was developed around five or six years ago while we worked on a large monolith and faced issues with slow CI runs. We managed to optimize performance significantly—from 30 minutes down to about 8 minutes. We decided to publish our findings, combining our efforts into a single tool called Test Prof.

00:10:02.560 Today, Test Prof is utilized by numerous renowned Ruby and Rails projects. I hope that after this conference, you will consider using Test Prof in your own projects. It begins with simply adding the gem to your application, allowing you to leverage all its capabilities.

00:10:36.240 The first step in using Test Prof for profiling is to assess the test suite as a whole. You can treat it like any Ruby program to find global environmental conditions that affect performance. To assist with this, you can use general Ruby profilers such as StackProf, RubyProf, and a newer one called 'faer'.

00:11:02.760 Let me demonstrate how to get started with general profiling using Test Prof. Test Prof simplifies using these profilers; you need no additional code or configuration. For example, you can add StackProf to your test environment, and it will activate profiling for your tests.

00:11:30.240 Consider enabling sampling profilers—these collect stack traces at a defined sampling rate. While it can be cumbersome with a vast number of tests, sampling assists in identifying consistent issues more clearly. If, for example, your profiler shows slow stack traces related to logging, it indicates a significant area to address. Disabling logging in your tests is one easy step that can yield around a 6% performance improvement.

00:12:05.440 This demonstrates the principle of efficient test profiling: seek out small changes that lead to significant improvements. For instance, with StackProf, you can visualize collected stack traces as flame graphs and identify root causes of slowness. You may discover that certain operations, like capturing exceptions in tests, are unnecessarily slow and can be avoided.

00:12:51.520 To analyze profiling results further, I want to highlight some upcoming profiling tools. One example is GPR, the next-generation profiler being developed actively by John Hawthorne and Aaron Patterson, which integrates with various profilers as well. This tool offers unique features, such as the ability to add markers for specific events, facilitating clearer navigation within flame graphs.

00:13:25.520 In conclusion, general profilers can help identify common Ruby-level problems unrelated to specific application behavior. However, tests themselves often use unique patterns in their structure, which means we need to locate and fix those patterns in our test code to improve performance. We should also utilize specialized test profilers.

00:14:00.520 Now, let me move on to the next step. Instead of analyzing every single test, focus on what matters most. A typical large test suite might contain thousands of tests, and not every group warrants the same level of optimization effort.

00:14:37.520 We’ve hypothesized over the years that tests within similar categories—like all controller tests or model tests—tend to share performance issues. If we find common problems, we can often apply fixes across entire groups of tests simultaneously, rather than addressing each individually. Among the tools that can help with this analysis is a profiler called TPR.

00:15:11.920 TPR allows us to understand how much time each group of tests consumes. When we analyze the reports generated from TPR, we often find that a significant portion of our testing time is spent on models and controllers.

00:15:39.840 With this knowledge, we can prioritize optimization efforts. For example, if model specs consume 40% of total testing time and we can improve the performance of these tests by a factor of two, that would translate to a 20% overall improvement in test suite performance.

00:16:14.640 However, analyzing 8,000 model examples can still be challenging. To narrow our focus further, we can utilize another profiler from Test Prof called Event Prof, which uses instrumentation to highlight specific problem areas.

00:16:54.320 This profiler can automatically tag slow tests, allowing you to create a manageable list of tests to analyze further. With this focused approach, we can delve deeper into aspects of testing performance, and for instance, profile how our factories are set up.

00:17:27.280 Factory Doctor is one of the most valuable profilers for factory setups. It identifies tests that do not require real database calls, which can save a lot of time during test execution. Certain practices, such as using build stubs instead of creating actual records, can lead to substantial performance enhancements.

00:18:07.760 Factory Prof helps in understanding the relationships between your factory-created objects and identifying a phenomenon known as factory cascades. Resolving these cascades can lead to considerable performance boosts.

00:18:41.840 Another useful profiler is RSpec dissect, which measures how much time is spent in a test's setup phase. With this data, you can develop plans for optimization. Many times, common setup contexts can be shared among multiple tests, so creating them repeatedly is inefficient. You can utilize lazy setup options offered by RSpec or use tools like Let It Be or Before O to manage setup data more effectively.

00:19:18.640 Therefore, once you identify bottlenecks within your slow test suite, you can refer to the Test Prof ecosystem for recipes that may help you fix the issues with minimal effort. After addressing the largest bottlenecks, reinforce what you've learned by developing good practices for writing tests upfront.

00:19:57.680 It's essential to regularly repeat the profiling process to prevent slow tests from creeping back into your workflow. Though this can become monotonous, the results are often worth the trouble.

00:20:34.760 To that end, we recently released an autopilot project, designed to help facilitate the profiling process. It isn't a true autopilot but rather assists in scripting out profiling tasks in Ruby language. For example, you can ask it to analyze the slowest tests or files by aggregating their execution times.

00:21:06.720 Today, Test Prof has grown into more than just a single gem. We have the autopilot tool and additional resources to improve code in Rails and Ruby projects. Stay tuned, and remember to take care of your tests. Please feel free to grab some stickers and approach me with any test-related questions after this talk.

00:21:47.480 Thank you!

See Slides on speakerdeck.com

Helvetic Ruby 2023