00:00:10.320
Welcome to the hands-on workshop titled 'From Slow to Go: Rails Test Profiling' with Vladimir Dementyev.
00:00:16.160
Vladimir is a mathematician who found his passion in programming Ruby and Erlang, contributing to open source, and being an author at Evil Martians. He is the creator of TestProf, a Ruby testing toolbox, among other projects.
00:00:23.320
He also enjoys creating many yet unknown ukulele melodies, so please welcome him to the stage.
00:00:35.600
Before we begin, I want to remind everyone that if you're having trouble installing or setting up due to a slow internet connection, I have a few USB sticks that you can grab to copy everything we need for today's workshop.
00:00:49.199
So feel free to come and grab one. Now, let's get started—we have two hours ahead and a lot to cover regarding test profiling.
00:01:03.920
In this workshop, we'll focus on profiling tests. It was great to have a keynote this morning discussing general profiling for Ruby, so I'm excited to continue that conversation here.
00:01:16.640
I hope you were able to catch it because we will address important topics such as sampling profiling and how to analyze flame graphs. That’s the plan for the first part of the workshop.
00:01:31.320
While you’re setting up and running tests, let’s discuss why we’re here today. If you’re attending this workshop, it’s likely because you find your application tests to be slow and have encountered problems with that.
00:02:06.200
You're likely wishing they could run faster either locally or on CI. The primary reasons to optimize tests are to avoid longer CI builds, which can increase expenses, and to reduce longer local build times.
00:02:30.160
If you check the current results for this code base, you'll see that running tests can take anywhere from 5 to 15 minutes. Yes, we can simply provide everyone with a free Mac laptop. However, 15 minutes is definitely not feasible, so we want to improve that.
00:03:04.239
When we talk about local builds, our goal is to optimize both aspects: making sure that even running a small subset of tests locally should be quick enough to avoid distractions and to make your feedback loop faster, ultimately increasing productivity.
00:03:37.639
This drive for speed and efficiency is why many of us choose Rails in the first place, as it allows us to be productive. The process of making tests faster resembles any other optimization work: we need to identify what causes the slowness, apply fixes, and repeat until we achieve satisfactory results.
00:04:03.240
I want to begin our session by discussing how to identify problems in your tests. One might assume we can simply use the profile flag in RSpec, which reveals the top ten slowest tests, and then go and fix those. However, that’s not the most effective way to tackle this.
00:04:32.960
Typically, optimizing slow tests can be time-consuming. Unlike production code optimization, you don’t have much time budget for this, as it doesn’t directly affect users. It’s not easy to persuade product managers to allocate resources to optimize tests either.
00:05:03.240
Instead, test optimization requires you to fix issues quickly with minimal refactoring. This approach leads to a focus on enhancing the speed of your testing framework and systems, which is why we introduced the TestProf library.
00:05:34.960
TestProf has been used effectively in many Rails applications over the years because dealing with tests is different from production code. There are common patterns that are often tough to identify, and we need the right tooling to help us track those down.
00:06:21.720
As the author of TestProf, I am committed to supporting open source work related to this and other projects. I work for Evil Martians, and during office hours at Hack Day, feel free to come and ask any questions you may have.
00:07:12.800
By the way, I wrote a book on the story of graph optimization—it’s a solid technical read and received good reviews. I’ll have some copies with me at lunch, as well as stickers from my book.
00:07:42.760
Though today we are primarily focused on tests and how to handle mastering them effectively. With that, I will be showing you a few techniques that leverage TestProf to improve our testing library.
00:08:02.600
We also have a project called Masteron, where we can experiment with TestProf on an open-sourced codebase. Our objective is to discover real-world problems—instead of working with an artificial example—and extract reusable solutions.
00:08:40.360
I want to give a quick disclaimer: we're not utilizing the master branch of Masteron as our source code; we're relying on a history-less repository to showcase our findings.
00:09:02.160
The Masteron project has already implemented some of TestProf's principles and optimizations. We intentionally reverted those to make our workshop more engaging, as most of the optimizations present will be new to you today, allowing you to actively participate in solving the issues as they arise.
00:09:51.520
We will group the day's profiling work into three parts: general profiling, using tools like StackProf and Vernier to identify hotspots in our code; then moving to specific profiling with TestProf; and finally addressing factories to clean up after our tests. We will see if we can make improvements, okay?
00:10:31.520
For example, I maximized the efficiency of a project—which I spent over two hours working on—by taking the local execution time from 8 minutes and 30 seconds down to 2 minutes and 30 seconds by applying these techniques.
00:10:41.440
As we go forward, I’ll share paper blanks so you can track your progress throughout the workshop and maybe even share it on social media to document the process. Interactive components will make sessions more lively and rewarding.
00:10:54.360
I also have the CI pipeline running tests using the same setup that Masteron utilizes, and we can compare results, so stay tuned for that.
00:11:05.520
The first step in optimizing Ruby tests is understanding the three primary reasons contributing to sluggishness: those elements can be longer CI runs caused by factors like database configurations, long-running integration tests, or race conditions.
00:11:57.600
When your tests take two or three times longer than expected, focus your attention on the troublesome spots that slow down everything else instead of spreading yourself too thin by trying to optimize everything simultaneously. Our goal is to ensure CI components finish in a more synchronous manner without having significant outliers.
00:12:47.440
Let's move towards practical profiling with StackProf and the various techniques we've established. The slide you see is meant to explain how sampling profilers like StackProf collect data by gathering stack traces over time, which we will analyze together.
00:13:16.640
These tools allow us to identify where time is being consumed in our tests and to examine common issues within configurations or patterns in execution that could be slowing things down. We should pay close attention to identifying any misconfigurations in your test setup that leads to inefficiencies throughout.
00:14:36.179
Now let's interact with our environment. If we start the container and launch the terminal, we will execute some model tests and make sure everything works smoothly. After that, we'll proceed to begin sampling our tests in a more systematic way.
00:15:51.600
Before diving into the profiling with StackProf, let’s make sure we understand some nuances like how to run tests efficiently and how each acts with profiling data over time. Successfully profiling the whole suite is unlikely due to time constraints, so I’ll illustrate the best approach for narrowing focus on a subset.
00:16:17.679
If we run tests against the entire suite, we’ll face a few limitations like a high overhead when profiling tests that take too long individually. So what’s the right method? First, we can select a small size sample of tests and repeat it to analyze and aggregate the results effectively.
00:17:12.640
To analyze progress and guide yourself throughout the profiling process, utilize papers and graphs to compare results visually, while leveraging the benefits of profiling tools like TestProf to enhance speed without delving deeply into the depths of individual slow tests.
00:18:02.640
Now we'll run our first profiling command using TestProf. First we introduce a brief explanation of how TestProf profiling works and how to efficiently analyze your test results to improve overall execution time.
00:18:51.680
By sampling based on selected filters, we can tune our Ruby profiler to retrieve structural information that helps explore each layer of our builds efficiently. This technique can pinpoint unnecessary delays while focusing on solution improvements as we proceed.
00:19:27.760
As our testing suite continues to run under profiling, take notes not just on run times but how well results measure against expectations. As tests conclude, charting these values will allow us deeper insights into how we can navigate down to specifics that impact overall performance.
00:20:15.040
Interacting with the results after profiling provides opportunities to inspect call graphs and determine what areas dominate usage. Functions that show excessive overhead can be addressed while profiling to clarify necessary changes needed to speed up execution time.
00:21:05.760
As we continue analyzing our test performances and profiling the vital areas for optimization, refining our approach will pay dividends—it’s only a matter of time before those small changes contribute to significant results. Let's keep sharpening our evaluation processes.
00:21:45.840
As we engage FAQ and further inquiries at our roundtable sessions, we will touch more on enhancements yielded by improved profiling approaches and how to build sustainability into our testing architecture with continuous optimization.
00:22:27.320
With this understanding, let's move into detailed profiling of specific function calls, allowing us to focus our optimizations on the areas truly impacting overall test performance. Identifying inefficiencies in individual functions will also contribute to a rounded solution.
00:23:00.640
We're aiming to enhance our existing unit tests by leveraging the profiles we've generated using TestProf. Through analyzing reports and call graphs, let’s explore those methods that require urgent attention and structure our optimization efforts around them.
00:23:39.520
Before we conclude this session, let’s review our key takeaways. Remember, optimizing tests in Rails is a multi-fold process involving strategy, profiling tools, and collaboration with your team. Reviewing this video material can provide additional introspections into how you approach profiling in your own work.
00:24:18.640
So to wrap things up, always be forward-thinking with how you approach optimization. Documenting changes can help improve practices over time. Test faster, more efficiently—that should be our goal as Rails developers.
00:25:05.600
Thank you for participating in today's workshop! I look forward to seeing the improvements you make in your coding practices through our discussions today and the techniques covered in this session. Let’s move forward together and keep refining our skills.