Look out! Gotchas of Using Threads in Ruby

00:00:10.920 Welcome, everyone, and thanks for coming to my talk, "Look Out! Gotchas of Using Threads in Ruby." I feel like Matt's keynote was amazing, and I was thinking it could either be the best keynote to have before my talk or the worst because he touched on a bunch of things I’m also going to talk about. Hopefully, it’s the best! So, let me get started. Who am I to be talking to you about this? My name is Ivo Anjo, and I’m a senior software engineer at Datadog. I’ve been using Ruby since 2014, and ever since I started, I fell in love with it. I love the expressibility, the community, and the tools; it’s just amazing to use Ruby.

00:00:38.719 I've always enjoyed exploring different language runtimes, such as C Ruby, JRuby, Truffle Ruby, and the Java Virtual Machine, among others. I still have my Euruko 2017 t-shirt, but it’s a bit old now, so I use it as pajamas. I also enjoy playing with concurrency, which is where a lot of the insights I will share today come from. Additionally, I really enjoy working on application performance and specifically on building tools that help you look at performance in a new way and uncover new insights. That’s how I ended up working on the Datadog continuous profiler for Ruby, which I am currently building.

00:01:13.400 Also here today is my colleague Gustavo, who works on application security management, so if you want to talk a bit about security, feel free to reach out to him as well. Okay, let’s go over a bit of an outline. We'll discuss why you should learn about concurrency and why it’s interesting. Then, we’ll talk a bit about concurrency in Ruby and threads, which are one form of concurrency that Ruby offers. I will share a few gotchas using threads, then we will discuss what atomic and non-atomic code is, covering examples of things you might think should be atomic but actually aren’t. Finally, we will finish with a recap.

00:01:55.880 Let’s start by discussing concurrency. Concurrency is not dark magic, and I feel like a lot of times people shy away from it or are a bit afraid because it seems like something only highly experienced Ruby wizards can master. That’s not true. Concurrency is actually a powerful and useful tool that you can use to solve many problems in your applications. So, it’s worth having this tool in your toolkit rather than saying, 'I shall not go there.' You can have fun with concurrency without being a mega wizard who knows everything about it.

00:02:28.080 One thing to be careful about with concurrency is the saying, 'With great power comes great responsibility.' While concurrency can be a solution to many problems, it also presents a few challenges and potential pitfalls. However, there are a lot of problems that you can solve effectively with concurrency, so don’t shy away from using it if it's the right solution for your problem. Think of concurrency as a skill you can learn. The whole point of this talk is to share some insights based on experiences I’ve run into in production — either myself or at companies I've worked at. By discussing these insights and understanding how they occurred, you can improve your mental model of what to expect when using concurrency.

00:03:43.480 Now, let’s talk about concurrency in Ruby. Concurrency in Ruby has never been in better shape. Nowadays, we have processes, reactors, threads, and fibers, all of which are undergoing continuous improvement. Developers are working on lowering memory usage, making them more efficient, and increasing performance, which allows you to create more threads with lower overhead. Therefore, concurrency in Ruby is in excellent shape with the latest versions and those on the horizon.

00:04:22.199 If you think a bit about this, you can notice that they form a hierarchy. Inside a Ruby process, you can have multiple processes. Within one process, you can have one or more reactors. If you're using Ruby 3E and not explicitly using reactors, you are using the main reactor that Ruby automatically creates when you start your Ruby application. Similarly, you can have one or more threads within each reactor. If you haven’t explicitly created a thread, you are just using the default thread created by Ruby. The same applies to fibers, which are tied to specific threads, enabling you to have one or more fibers inside a thread.

00:05:08.440 In this talk, I’m mostly focusing on threads, so let’s compare threads with other concurrency models. Threads versus processes: Threads typically provide lower memory usage because they share the same memory set, which means you don’t have to duplicate stuff in memory. With multiple processes, you often end up duplicating memory. For instance, if you have YJIT, it will compile things, and if you have multiple processes, YJIT will be duplicated across those processes, compiling again. Threads, on the other hand, allow for easier communication and cooperation.

00:06:05.839 However, threads provide significantly less isolation than processes. For example, if something goes wrong with a web request inside a process, you can safely terminate that process, eliminating the issue. Conversely, if something fails within a thread, terminating that thread may leave the system in an inconsistent state. Additionally, there’s latency impact; threads can affect each other's latency, while processes, being more isolated, typically handle this better.

00:07:00.200 Now, let’s look at threads versus reactors. To be fair, reactors are still experimental, while threads are mature and stable. Threads allow you to share mutable objects, which can be both an advantage and a disadvantage. The power of threads comes with the risk of potential concurrency issues due to shared states. On the other hand, reactors offer a more limited model, which reduces some of the pitfalls associated with threads. Threads share the same Global VM Lock (GVL), which we’ve heard Matt talk about a lot today.

00:07:50.119 Let’s discuss threads versus fibers. Threads use preemptive scheduling, while fibers utilize cooperative scheduling. Preemptive scheduling means that Ruby automatically switches between threads. In contrast, unless you're using something like the fiber scheduler, fibers require manual switching. This can introduce latency since if a fiber does not switch to another, it can block others from running effectively. On the other hand, the operating system generally manages process scheduling more efficiently, ensuring that each process receives its fair share of CPU time.

00:08:55.159 Unfortunately, threads are more expensive to create and require more memory. Usually, applications can handle dozens or maybe hundreds of threads effectively. However, if you find yourself approaching hundreds, you may be pushing the limits, causing suboptimal performance from either Ruby or your operating system. In contrast, fibers are cheaper to create and can be managed in greater numbers without straining system resources.

00:09:19.399 Let’s delve into some gotchas of using threads and what we can learn from them. Gotcha number one: Threads in Ruby are concurrent but not parallel. We will discuss this risk shortly. In Ruby, if you have a method that takes five seconds to execute and you run it twice sequentially, that will take ten seconds. However, if you create two threads where each thread calls the same method, it will still take ten seconds total. The key insight is that while you may have two threads running, Ruby will switch between them, causing the total execution time to remain unchanged. This is because Ruby does not run multiple threads at the exact same time with CPU-bound tasks; it merely alternates between them.

00:10:23.919 This becomes evident when you execute concurrent tasks that do not heavily rely on CPU usage. For example, if you’re making multiple database queries or making web requests, Ruby can switch to another thread while one is waiting for an external resource. This means threads can still be extremely useful in Ruby, especially when they handle tasks that involve waiting for IO operations. However, one must be cautious, as utilizing multiple threads can inadvertently add latency to your application.

00:10:59.119 Gotcha number two: Threads in JRuby and Truffle Ruby are indeed parallel. If we revisit our previous example, where a method takes five seconds to execute, running it on two threads in JRuby or Truffle Ruby can result in only five seconds of total execution time. This means if you are using a multicore machine, threads can execute concurrently across different threads and actually complete faster. The same is true for multiple reactors in C Ruby, as they can also offer effective parallelism when designed using processes.

00:12:30.839 Some insights on this point: it is possible to achieve parallelism in Ruby, so don’t let anyone tell you otherwise. For a long time, developers have been achieving parallelism in Ruby using multiple processes through web servers like Puma configured in multiprocess mode or Unicorn. These are great ways to handle parallelism in Ruby applications. Gotcha number three: When utilizing threads and sleep in your tests, you can inadvertently end up with flaky tests. For example, if you have a background thread running some result calculation and you add a sleep in your test, it may work perfectly on your machine, but it can start failing intermittently on another machine because the timing may differ.

00:14:07.680 This issue might lead you to increase your sleep duration, and while it seems to fix the problem temporarily, it can introduce more instability across different environments.

00:15:34.640 Instead of relying on sleep walks for synchronization, use synchronization mechanisms to wait for the background task to complete.

00:15:48.640 One solution is calling 'join' on your threads. Calling join will ensure that the main thread waits until the background thread has finished its task, eliminating the need for artificial sleep time. However, in some scenarios, you may not control the lifetime of the thread, or it may be nested in some framework. Therefore, using Ruby's Queue class for communication across threads can be an effective way of managing thread synchronization.

00:16:36.439 In this case, you can create a 'done queue.' When you spawn your thread, you can send a signal to the queue to relay that the work is complete, allowing the main thread to wait for that signal instead of relying on sleep. You can extend this pattern further to communicate across multiple threads effectively, and it is crucial to utilize Thread and Sized Queue classes designed for safe concurrency in Ruby.

00:17:32.640 To fix flaky tests, an additional solution involves the concept of a ‘better sleep solution,’ which can be implemented using an approach where you continuously check a condition after small sleep intervals rather than introducing arbitrary static sleeps. This way, you maintain responsiveness in your tests while reducing the chances of flaky behavior.

00:18:14.879 Gotcha number four: Let’s discuss what happens when mutating a hash across threads. Suppose we have one thread writing to a hash while another iterates through it. This is a race condition situation; when threads interact concurrently, you may encounter a runtime error stating that you can’t add a new key into a hash during iteration.

00:19:02.200 In production, this error can be challenging to debug since it’s not immediately clear which thread is interacting with the hash, leading to confusion about whose fault it was. The primary takeaway here is to avoid writing to shared objects when they are also being iterated upon.

00:19:59.639 To address this issue, one solution is to create the hash, mutate it as needed, and then freeze it before sharing it with other threads. Freezing the hash will prevent any further modifications, making it easier to spot unintended writes since Ruby will raise an exception if someone tries to do so. Another alternative is simply to create a copy of the hash instead of sharing the original.

00:20:47.580 If you want to engage with more advanced techniques, consider utilizing the Concurrent Ruby gem, which offers powerful tools for managing concurrency. For example, instead of a regular hash, you can opt for a concurrent map instance from this gem, providing safe operations suited for concurrent contexts.

00:21:33.440 If you're considering rolling your implementation of concurrency using mutexes, be aware that a mutex enforces one-at-a-time access to an object. In this case, you would create a mutex, and when you want to write to the hash, you would call 'mutex.synchronize' on it. This ensures that when one thread is accessing the object, no other thread can write to it.

00:22:10.439 However, it is easy to forget to call synchronize, which is why it’s advisable to create a wrapper class that contains both the hash and its associated mutex. This prevents direct access to shared resources, restricting potential pitfalls.

00:22:54.240 A few hints to keep in mind: Avoid sharing mutable objects among threads in general, not just hashes, and be cautious of situations where you write while iterating since this can lead to difficult-to-debug scenarios in production.

00:23:54.400 Now, let’s talk about atomic behavior. Atomic operations are executed completely or not at all, ensuring consistency in your code. Many tools, like databases, utilize atomic transactions to prevent inconsistent states. Unfortunately, most operations in Ruby are not atomic, as Ruby can switch threads at almost any point, even mid-operation. For example, if you are modifying multiple variables in the same method, other threads might interject, causing unexpectedly inconsistent behavior.

00:25:11.520 Gotcha number five: Resolving race conditions can be difficult, especially if you have tasks that rely on a shared resource. For example, if you create a database client instance in a multithreaded environment, you could inadvertently end up with multiple database client instances. This can lead to excessive open connections, which can exhaust your limits, leading to application failures. It's essential to wrap your client initialization with a mutex to prevent concurrent creation.

00:27:32.400 You must ensure that all threads use the same mutex instance, thus controlling access to the resource. In cases where you initialize the mutex within each thread incorrectly, you’ll find that you again run into the risk of multiple threads creating resources you did not intend.

00:28:41.840 As a note, avoid using lazy-loaded or on-demand loading of classes in multithreaded contexts. When threads access classes or modules that may not yet be fully loaded, it can lead to undefined methods being called upon incomplete classes. This can create perplexing errors during execution because the call may seem valid at first glance.

00:29:59.680 As we recap, remember that Ruby can switch threads at any point. Thus, when writing concurrent code, it's critical to account for potential context switching unless you’re only dealing with independent threads. If you are sharing state or variables, you must ensure you are taking the necessary precautions to manage shared states.

00:31:04.920 If possible, avoid mutable objects being shared across threads, but when needed, use the tools Ruby provides. As mentioned, the Mutex, Queue, and Sized Queue are great for ensuring safe operations. Additionally, the Concurrent Ruby gem is incredibly valuable, offering atomic classes and thread pools to manage threads efficiently. It works well across different implementations, including JRuby and Truffle Ruby.

00:32:43.560 Don't be afraid of concurrency; just approach it with caution and use the appropriate tools. I encourage you to have fun with it, experiment with it, and hopefully, this talk has given you essential insights into the potential challenges and clever solutions to implement concurrency effectively in Ruby.

00:34:00.160 Thank you very much for your attention today! If you’d like to find more about my work, feel free to check out my blog. If you have any questions, don’t hesitate to reach out!

00:38:01.240 Lunch is happening right now. Thank you for being part of the first half of this day, and make sure to catch up with others to discuss Ruby!