00:00:11.360
Welcome to "Parallel Testing with Ractors: Putting CPUs to Work." My name is Vinicius Stock, or Vinnie. I'm a senior developer in the Ruby type scene at Shopify, where we work on leveraging static typing to enhance our developers' productivity. We've been doing a lot of work around Survey and Tapioca, and we are hiring. So if you're interested in joining our team, please reach out. If you want to know more about my work, you can find me at "vinistock" on both Twitter and GitHub.
00:00:39.239
In the last edition of RubyConf, we got the first Ractor demonstration from Koichi. I was really eager to try Ractors out in particular for running tests in parallel. Today, I want to share some of what I learned while trying to do so.
00:00:46.140
We'll take a look at a few test execution strategies to parallelize tests, an introduction to Ractors and how they communicate. We'll build a mini test-like framework, parallelize it using Ractors, and then examine a few features, advantages, and limitations of using Ractors.
00:01:12.659
What does it mean to run tests? Fundamentally, tests are just pieces of code we want to organize and execute. It doesn't matter if, in MiniTest, they are test methods, and in RSpec, they are Ruby blocks. They are just blocks of code we want to run in an organized manner. The key is to do this as quickly as possible to reduce the feedback loop for our developers, leading us to strategies for parallelizing test execution.
00:01:39.659
The first strategy we'll discuss involves using groups. We start by figuring out the tests we want to run, splitting them into different groups, and assigning each group to a separate worker or CPU for execution. While this is a simple strategy, it does not guarantee an even distribution of workload. If one CPU ends up with a larger group or slower tests, it may finish much later than the other CPUs, resulting in idle time.
00:02:28.319
Execution strategy number two uses a queue and attempts to address the workload distribution issue. Again, we organize the tests into a queue, which is a list of all the tests we need to run. Here, workers can pop items from this queue as they become available to do more work. In MiniTest, this would look like a queue where each item represents test methods, allowing workers to pop items and run them, avoiding idleness.
00:03:06.599
The last strategy we’ll explore is called work stealing, common in various computing areas. In this case, we again start by figuring out the tests we want to run, randomizing them into groups, and initially assigning each group to a specific worker. The difference is that if any worker becomes idle, they can signal the main process to take work from another worker and pass it to the idle one. This ensures that no worker remains idle while others are still working.
00:04:07.019
Both, the queue-based execution strategy and work stealing try to ensure an even workload distribution. The main difference is that work stealing reduces the communication overhead involved with a shared queue. In the queue-based strategy, each time a worker needs to run a test, it has to communicate with the shared queue, while in work stealing, each worker starts with a predetermined list and only needs to communicate when they become idle.
00:04:50.580
Originally, Ruby did not support utilizing all the CPUs in our computers effectively for parallel test execution. For instance, Rails ships with a MiniTest plugin that can parallelize tests for our applications. There are also several other plugins available for different test frameworks to accomplish the same.
00:05:21.000
In the case of Rails, it implements the second strategy using a queue object that holds all of the test examples defined by the application. It then forks multiple Ruby processes to create a worker pool for executing tests in parallel. The last piece we need to address is how to communicate between the main process holding the shared queue object and all the workers.
00:05:56.600
Rails handles this by using a gem called DRb, which stands for Distributed Ruby. This gem provides an abstraction for sharing Ruby objects across different processes. In Rails, the queue object is shared with all the workers, allowing them to pop items from the queue similarly to the main process.
00:06:37.380
Now, in Ruby 3, we have Ractors. Can they perform any differently than processes? We begin by creating a new Ractor using the "new" method, just like any other Ruby object. The code passed into the block runs in parallel. In our case, it's a worker receiving an item and processing it. However, at this point, the work is halted, waiting for the item to arrive.
00:07:09.900
To kick off processing, we send it some initial information and then use "take" to wait until the worker finishes processing. At that point, we can retrieve the return value from the block passed to the Ractor's initialization.
00:08:02.220
In a more involved example for our test framework, we create a worker pool. We start by creating a queue of numbers from 0 to 99, which will serve as our queue of tests. We also create an array to store the results of our calculations and build a pool of ten Ractors.
00:08:46.920
Each Ractor will be in a loop, receiving a number sent to it and yielding that number multiplied by five back to the main Ractor. This form of communication is often used when we need to return values to the main Ractor at multiple points. As we kick off all Ractors with an item from our queue, we ensure they can process numbers based on our given logic.
00:09:13.200
When our queue of numbers is completely empty, the loop will still run in the background, ensuring that we handle each worker, confirming they're finished processing, to avoid incomplete results.
00:10:00.060
Next, we’ll dive into the details of executing the tests within our framework. We will begin by keeping track of all the defined test classes when we require the test files. We can define a class method, 'inherited,' on a base class 'Test.' All tests must inherit from this base class, allowing us to gather a list of them easily.
00:10:47.580
To build our queue of tests, we need to create a structure that can hold tuples. Each tuple would comprise the test method and its corresponding class. By looping through the classes, we can leverage the instance methods defined in them while filtering out any inherited methods.
00:11:46.760
Once we have our queue of tuples, we need to execute the tests. For each test method received from the queue, we create an instance of the test class to ensure isolation between test examples. If instance variables change within one example, they won't impact others, ensuring reliable results.
00:12:34.800
As we execute the tests, we can invoke the necessary lifecycle methods: setup, the test method, and teardown. This outlines our basic execution strategy. However, we need assertions to verify that our application works as expected.
00:13:12.540
We'll implement a fundamental assertion to check if a value is truthy or falsy. If an assertion fails, we need to register that failure so we can display it at the end, as tracking failure information is crucial for feedback.
00:14:27.740
In our execution flow, we track statistics such as the total number of assertions and test runs. If an assertion fails, we skip to the next item in the queue since the current test cannot succeed. The approach allows us to provide meaningful output for the user at the end of the test run.
00:15:29.220
We aggregate our test run results, reporting successes and failures while ensuring each instance maintains relevant details. This allows us to print out the results and highlight the failures.
00:16:26.160
With all the pieces in place, we need to connect them within an executable framework that requires the test files and processes the tuples. Each completed test updates the Singleton instance of the reporter, which aggregates everything for the final summary.
00:17:24.660
Now that our sequential test framework works, we can focus on parallelizing it, which is the primary goal. We replicate our earlier logic while implementing Ractors for greater efficiency.
00:18:11.400
Upon execution, Ractors will run tests in parallel. This ensures less overhead during initialization compared to forking multiple processes. However, certain limitations arise from relying on class variables, which are not accessible from within child Ractors.
00:19:03.060
To collect reporting information, we define temporary reporters that yield data for each test execution. Once all workers finish, we merge them into a single aggregated report.
00:19:56.900
Throughout testing, we have encountered some bugs in Ruby related to garbage collection and object management within Ractors. Surprisingly, the need for careful coding practices was underscored by these experiences.
00:20:38.880
Despite the limitations of Ractors, they present exciting opportunities for parallelism within testing frameworks. This environment allows for flexibility in testing experimental technologies.
00:21:11.160
Thank you very much for listening to my talk! I sincerely hope you enjoyed it as much as I enjoyed working with Ractors. Enjoy the rest of RubyConf!