Ruby Video
Talks
Workshop: From slow to go: Rails test profiling hands-on
Vladimir Dementyev
@palkan
#ruby-on-rails
#profiling
#test-driven-development
#performance
#rspec
#continuous-integration-ci
#testing
Summarized using AI

Workshop: From slow to go: Rails test profiling hands-on

by Vladimir Dementyev

The workshop "From Slow to Go: Rails Test Profiling" led by Vladimir Dementyev focuses on enhancing the performance of Rails test suites to save time and increase productivity. It addresses common frustrations developers face with slow test executions both in local environments and CI (Continuous Integration).

Key Points Discussed:

  • Test Profiling Basics: Dmitry begins by explaining the objective of profiling tests to identify and address performance bottlenecks. Effective profiling can dramatically reduce test execution time, thereby improving productivity.
  • Challenges with Slow Tests: Participants often struggle with lengthy test times. Possible reasons include inefficient database configurations, lengthy integration tests, or race conditions leading to delayed feedback loops.
  • Introduction of TestProf: TestProf is discussed as an effective Ruby testing toolbox designed to identify and fix problems within test suites. It helps streamline the optimization process, focusing on speed and enhancing the testing infrastructure.
  • Practical Profiling Techniques: The workshop covers various profiling tools, including StackProf and Vernier, which gather data on test execution. Techniques for using these profiling tools effectively are shared, including sampling strategies to analyze test performance adequately.
  • Masteron Project: Dmitry introduces the Masteron project as a practical example where TestProf has been successfully integrated to identify performance issues in an open-source codebase. Participants work hands-on to apply the principles learned on real-world challenges.
  • Iterative Improvement: The importance of iterating on profiling results is emphasized, encouraging attendees to constantly refine their approaches based on insights gained through profiling data.

Key Takeaways:

  • Optimize your test suites to enhance productivity by minimizing run times in both local setups and CI environments.
  • Utilize tools like TestProf to gain a deeper understanding of testing pipelines and pinpoint issues that lead to inefficiencies.
  • Keep a collaborative mindset for continuous optimization, documenting changes to improve testing strategies over time.
  • Remember that effective profiling is an ongoing process that requires systematic approaches and regular adjustments based on findings.

In conclusion, attendees are encouraged to implement the techniques discussed to achieve faster testing cycles and better coding practices, reinforcing the goal of becoming more efficient Rails developers.

00:00:10.320 Welcome to the hands-on workshop titled 'From Slow to Go: Rails Test Profiling' with Vladimir Dementyev.
00:00:16.160 Vladimir is a mathematician who found his passion in programming Ruby and Erlang, contributing to open source, and being an author at Evil Martians. He is the creator of TestProf, a Ruby testing toolbox, among other projects.
00:00:23.320 He also enjoys creating many yet unknown ukulele melodies, so please welcome him to the stage.
00:00:35.600 Before we begin, I want to remind everyone that if you're having trouble installing or setting up due to a slow internet connection, I have a few USB sticks that you can grab to copy everything we need for today's workshop.
00:00:49.199 So feel free to come and grab one. Now, let's get started—we have two hours ahead and a lot to cover regarding test profiling.
00:01:03.920 In this workshop, we'll focus on profiling tests. It was great to have a keynote this morning discussing general profiling for Ruby, so I'm excited to continue that conversation here.
00:01:16.640 I hope you were able to catch it because we will address important topics such as sampling profiling and how to analyze flame graphs. That’s the plan for the first part of the workshop.
00:01:31.320 While you’re setting up and running tests, let’s discuss why we’re here today. If you’re attending this workshop, it’s likely because you find your application tests to be slow and have encountered problems with that.
00:02:06.200 You're likely wishing they could run faster either locally or on CI. The primary reasons to optimize tests are to avoid longer CI builds, which can increase expenses, and to reduce longer local build times.
00:02:30.160 If you check the current results for this code base, you'll see that running tests can take anywhere from 5 to 15 minutes. Yes, we can simply provide everyone with a free Mac laptop. However, 15 minutes is definitely not feasible, so we want to improve that.
00:03:04.239 When we talk about local builds, our goal is to optimize both aspects: making sure that even running a small subset of tests locally should be quick enough to avoid distractions and to make your feedback loop faster, ultimately increasing productivity.
00:03:37.639 This drive for speed and efficiency is why many of us choose Rails in the first place, as it allows us to be productive. The process of making tests faster resembles any other optimization work: we need to identify what causes the slowness, apply fixes, and repeat until we achieve satisfactory results.
00:04:03.240 I want to begin our session by discussing how to identify problems in your tests. One might assume we can simply use the profile flag in RSpec, which reveals the top ten slowest tests, and then go and fix those. However, that’s not the most effective way to tackle this.
00:04:32.960 Typically, optimizing slow tests can be time-consuming. Unlike production code optimization, you don’t have much time budget for this, as it doesn’t directly affect users. It’s not easy to persuade product managers to allocate resources to optimize tests either.
00:05:03.240 Instead, test optimization requires you to fix issues quickly with minimal refactoring. This approach leads to a focus on enhancing the speed of your testing framework and systems, which is why we introduced the TestProf library.
00:05:34.960 TestProf has been used effectively in many Rails applications over the years because dealing with tests is different from production code. There are common patterns that are often tough to identify, and we need the right tooling to help us track those down.
00:06:21.720 As the author of TestProf, I am committed to supporting open source work related to this and other projects. I work for Evil Martians, and during office hours at Hack Day, feel free to come and ask any questions you may have.
00:07:12.800 By the way, I wrote a book on the story of graph optimization—it’s a solid technical read and received good reviews. I’ll have some copies with me at lunch, as well as stickers from my book.
00:07:42.760 Though today we are primarily focused on tests and how to handle mastering them effectively. With that, I will be showing you a few techniques that leverage TestProf to improve our testing library.
00:08:02.600 We also have a project called Masteron, where we can experiment with TestProf on an open-sourced codebase. Our objective is to discover real-world problems—instead of working with an artificial example—and extract reusable solutions.
00:08:40.360 I want to give a quick disclaimer: we're not utilizing the master branch of Masteron as our source code; we're relying on a history-less repository to showcase our findings.
00:09:02.160 The Masteron project has already implemented some of TestProf's principles and optimizations. We intentionally reverted those to make our workshop more engaging, as most of the optimizations present will be new to you today, allowing you to actively participate in solving the issues as they arise.
00:09:51.520 We will group the day's profiling work into three parts: general profiling, using tools like StackProf and Vernier to identify hotspots in our code; then moving to specific profiling with TestProf; and finally addressing factories to clean up after our tests. We will see if we can make improvements, okay?
00:10:31.520 For example, I maximized the efficiency of a project—which I spent over two hours working on—by taking the local execution time from 8 minutes and 30 seconds down to 2 minutes and 30 seconds by applying these techniques.
00:10:41.440 As we go forward, I’ll share paper blanks so you can track your progress throughout the workshop and maybe even share it on social media to document the process. Interactive components will make sessions more lively and rewarding.
00:10:54.360 I also have the CI pipeline running tests using the same setup that Masteron utilizes, and we can compare results, so stay tuned for that.
00:11:05.520 The first step in optimizing Ruby tests is understanding the three primary reasons contributing to sluggishness: those elements can be longer CI runs caused by factors like database configurations, long-running integration tests, or race conditions.
00:11:57.600 When your tests take two or three times longer than expected, focus your attention on the troublesome spots that slow down everything else instead of spreading yourself too thin by trying to optimize everything simultaneously. Our goal is to ensure CI components finish in a more synchronous manner without having significant outliers.
00:12:47.440 Let's move towards practical profiling with StackProf and the various techniques we've established. The slide you see is meant to explain how sampling profilers like StackProf collect data by gathering stack traces over time, which we will analyze together.
00:13:16.640 These tools allow us to identify where time is being consumed in our tests and to examine common issues within configurations or patterns in execution that could be slowing things down. We should pay close attention to identifying any misconfigurations in your test setup that leads to inefficiencies throughout.
00:14:36.179 Now let's interact with our environment. If we start the container and launch the terminal, we will execute some model tests and make sure everything works smoothly. After that, we'll proceed to begin sampling our tests in a more systematic way.
00:15:51.600 Before diving into the profiling with StackProf, let’s make sure we understand some nuances like how to run tests efficiently and how each acts with profiling data over time. Successfully profiling the whole suite is unlikely due to time constraints, so I’ll illustrate the best approach for narrowing focus on a subset.
00:16:17.679 If we run tests against the entire suite, we’ll face a few limitations like a high overhead when profiling tests that take too long individually. So what’s the right method? First, we can select a small size sample of tests and repeat it to analyze and aggregate the results effectively.
00:17:12.640 To analyze progress and guide yourself throughout the profiling process, utilize papers and graphs to compare results visually, while leveraging the benefits of profiling tools like TestProf to enhance speed without delving deeply into the depths of individual slow tests.
00:18:02.640 Now we'll run our first profiling command using TestProf. First we introduce a brief explanation of how TestProf profiling works and how to efficiently analyze your test results to improve overall execution time.
00:18:51.680 By sampling based on selected filters, we can tune our Ruby profiler to retrieve structural information that helps explore each layer of our builds efficiently. This technique can pinpoint unnecessary delays while focusing on solution improvements as we proceed.
00:19:27.760 As our testing suite continues to run under profiling, take notes not just on run times but how well results measure against expectations. As tests conclude, charting these values will allow us deeper insights into how we can navigate down to specifics that impact overall performance.
00:20:15.040 Interacting with the results after profiling provides opportunities to inspect call graphs and determine what areas dominate usage. Functions that show excessive overhead can be addressed while profiling to clarify necessary changes needed to speed up execution time.
00:21:05.760 As we continue analyzing our test performances and profiling the vital areas for optimization, refining our approach will pay dividends—it’s only a matter of time before those small changes contribute to significant results. Let's keep sharpening our evaluation processes.
00:21:45.840 As we engage FAQ and further inquiries at our roundtable sessions, we will touch more on enhancements yielded by improved profiling approaches and how to build sustainability into our testing architecture with continuous optimization.
00:22:27.320 With this understanding, let's move into detailed profiling of specific function calls, allowing us to focus our optimizations on the areas truly impacting overall test performance. Identifying inefficiencies in individual functions will also contribute to a rounded solution.
00:23:00.640 We're aiming to enhance our existing unit tests by leveraging the profiles we've generated using TestProf. Through analyzing reports and call graphs, let’s explore those methods that require urgent attention and structure our optimization efforts around them.
00:23:39.520 Before we conclude this session, let’s review our key takeaways. Remember, optimizing tests in Rails is a multi-fold process involving strategy, profiling tools, and collaboration with your team. Reviewing this video material can provide additional introspections into how you approach profiling in your own work.
00:24:18.640 So to wrap things up, always be forward-thinking with how you approach optimization. Documenting changes can help improve practices over time. Test faster, more efficiently—that should be our goal as Rails developers.
00:25:05.600 Thank you for participating in today's workshop! I look forward to seeing the improvements you make in your coding practices through our discussions today and the techniques covered in this session. Let’s move forward together and keep refining our skills.
Edit
Explore all talks recorded at RailsConf 2024
+29