Don't Wait For Me! Scalable Concurrency for Ruby 3!

Talks

Samuel Williams

Don't Wait For Me! Scalable Concurrency for Ruby 3!

by Samuel Williams

In the presentation titled "Don't Wait For Me! Scalable Concurrency for Ruby 3!" delivered by Samuel Williams at RubyKaigi Takeout 2020, the focus was on improving Ruby's scalability through the use of fibers and a new Ruby fiber scheduler. The discussion began with the essentials of scalability and its critical role in maximizing application efficiency, particularly in handling large traffic volumes that Ruby can face. Key points discussed included:

Understanding Scalability: Scalability is about efficient resource utilization as system demands grow. It's essential to avoid overwhelming systems, especially in cloud environments where inefficiencies can incur higher costs.
Concurrency vs. Parallelism: Concurrency allows interleaving tasks, improving hardware utilization and reducing the overhead associated with traditional threading and processes. Efficient task scheduling can minimize waiting times and enhance application throughput.
Introducing the Fiber Scheduler: The implementation of a fiber-based scheduler that seamlessly integrates with existing Ruby code without requiring major rewrites was emphasized. This scheduler can handle non-blocking I/O operations, improving responsiveness and scalability.
The Async Gem: The Async framework allows developers to write non-blocking code with minimal changes to existing structures. It provides wrappers around traditional I/O operations, enabling concurrent execution and improving performance under I/O-bound conditions.
Falcon Application Server: The talk also introduced Falcon, a web server for Ruby designed as a proof of concept for scalability. With capabilities to manage numerous simultaneous connections efficiently using fibers, Falcon demonstrates significant improvement over traditional web servers.
Challenges with Existing Libraries: The discussion highlighted issues with existing libraries, such as Active Record, which may not efficiently support concurrency due to their connection management strategies. However, ongoing discussions aim to resolve these limitations as Ruby 3's concurrency model evolves.

In conclusion, Samuel Williams stressed the importance of community involvement in developing these new scalability features. He encouraged developers to experiment with the Async gem and Falcon server while considering financial support for further advancements. The objective is clear: to improve Ruby’s ecosystem, allowing developers to build more scalable and efficient applications without significant changes to existing codebases. This work not only enhances application performance but also aligns with broader goals of resource efficiency in technology architecture.

00:00:00.560 Hello and welcome to my talk: "Don't Wait For Me! Scalable Concurrency for Ruby 3!" Today, we are going to discuss Ruby scalability, what it is, and how it impacts your applications. Then, we will look at the new Ruby fiber scheduler, how it is implemented, and how it can improve existing applications with minimal changes.

00:00:24.640 So, what makes Ruby a great language? Ruby has many good qualities, but I wanted to gather some opinions from others. Naturally, I asked Twitter, and here are the results: syntax was the most important factor, followed closely by semantics, libraries, and finally scalability. Someone even voted for math—thank you, Matt, for creating Ruby and supporting my work!

00:00:54.879 Looking at these results, we might wonder: is scalability important? Why did it come last? Many companies handle large amounts of traffic with Ruby, yet it appears scalability might be a weak point. Before we delve deeper, we should discuss what scalability is, so we have a clear idea of the problems we are trying to solve and how we can improve Ruby in this regard.

00:01:30.640 Scalability is fundamentally about the efficiency of a system and how many resources are required as the size of the problem increases. For example, we can compare the efficiency of English and Kanji: English requires six times more characters to convey the same meaning, making it less efficient.

00:01:52.560 As we increase the size of our system, efficiency issues manifest in many different ways. One significant issue is resource utilization: how much time we spend processing versus waiting. If we do not efficiently utilize the hardware, our scalability will be negatively impacted, leading to higher running costs. Cloud providers thrive on inefficient systems because they profit from it, but we want to avoid this situation to enhance the cost-effectiveness of our business growth.

00:02:26.720 Modern processes are so quick that they often spend a lot of time waiting on memory, storage, and network I/O. To utilize hardware more efficiently, we can time-share between multiple tasks; when one task is waiting, another can execute. This is one way to improve scalability.

00:02:51.120 Typical operating systems use processors and threads to execute tasks. While threads are similar to processors, they have less isolation and fault tolerance because they share memory and address space. A single hardware processor can context-switch between these tasks to complete all required work as quickly as possible.

00:03:16.879 We can add more hardware processes to increase system scale, allowing multiple tasks to execute simultaneously. If one task is waiting, the processor can move to another. However, switching between tasks is not free; context switching, as it is called, is an expensive operation at the system level. As processors dance between tasks, the overhead of switching can become significant.

00:03:48.640 We refer to simultaneous execution as parallelism, which is an important aspect of any scalable system. However, parallelism introduces significant complexities, including non-determinism, race conditions, and fault tolerance. Application code should not have to manage these complexities, as it has repeatedly been shown that thread-safety issues are a significant source of bugs. Used carefully, isolated parallelism provides the foundation for scalable systems.

00:04:25.440 Another way to improve hardware utilization is to interleave tasks. This can be achieved through several methods: callbacks, promises, and fibers. When application code would execute a blocking operation, the state is saved, and instead of waiting, another task can be executed. When the application code is ready to process again, it will resume.

00:05:01.039 This approach allows us to combine tasks back to back within a single process, minimizing the overhead of system-level context switches by implementing an event loop that schedules work more efficiently. We refer to interleaved execution as concurrency, which is an important tool for highly scalable systems.

00:05:20.080 Threads and processes are designed to expose hardware-level parallelism, limited by the number of processor cores in your system. In practice, having one process or thread per connection typically implies an upper bound of 10 to 100 connections per system, which is simply not practical for building highly scalable applications.

00:05:43.840 How does scalability impact cost? A lack of scalability in an application can be costly for businesses. Not only does it increase your hardware costs, but it can also negatively impact user experience if systems become overloaded. Additionally, you may find yourself building more complex distributed systems to compensate for a lack of scalability in your core application. Such designs bring many hidden costs, including operational and maintenance overheads, making it difficult to pursue new opportunities due to scalability bottlenecks.

00:06:20.160 Efficient and scalable systems are also beneficial for the planet, as fewer computers are needed to perform the same work. The technology sector is responsible for approximately 2% of global carbon emissions, and civilization faces unprecedented challenges related to climate change. We can help by improving the scalability of our code to reduce natural resource consumption and energy utilization, thus contributing to the development of more efficient systems for society.

00:06:46.639 I believe this is our responsibility as software engineers. Now, is Ruby fast enough? Significant efforts go into improving Ruby, including enhancements to the interpreter, garbage collection, and just-in-time compilation. Yet, we are still striving to improve performance.

00:07:07.120 Let's consider a hypothetical request that includes several blocking operations. I have split it into 10 milliseconds of processing time and 190 milliseconds of waiting time. The total time is 200 milliseconds, a very typical situation. For instance, at its peak, a real Ruby web application might spend 25 milliseconds executing Ruby code and 150 milliseconds waiting on Redis and external HTTP requests.

00:07:46.720 So, what happens if we make Ruby ten times faster? In our hypothetical scenario, we reduce the processing time from 10 milliseconds to 1 millisecond—a tenfold improvement—but the total time is still dominated by waiting, which remains unchanged. Now, what if we could make Ruby's execution a hundred times faster?

00:08:09.679 The results are clear: even if we improve Ruby's performance, so much time is still wasted waiting. Thus, we need to find ways to make Ruby faster and avoid long waiting times.

00:08:25.760 Let's revisit the example of a single request within the context of a small program. This request is one among many, and such sequential processing happens frequently when clients interact with multiple remote systems or fetch multiple records on an application server handling various incoming requests.

00:08:46.640 Imagine if we could change this: while waiting for one request to complete, we could start another. Although we cannot change the latency of individual requests, we can significantly reduce the total latency of all requests together, thereby improving the application's throughput.

00:09:22.640 So, how do we implement this? Let's examine the changes needed in the code. We will start from our sequential program and add a worker to wrap our requests, allowing us to represent individual jobs that can run concurrently.

00:10:12.320 Next, let's expand what a request looks like. First, we will make a connection to the service, then write a request, and finally read the response.

00:10:31.440 Let's make our read operation non-blocking. Primero, we will change 'read' to 'non-block,' and then we need to check the result; if the data is unavailable, we'll wait for the response to become readable. In that case, we will use a specific system call—IO.select—that waits until the data is available. This operation accounts for the waiting time in our program. Instead of waiting, we can use a special operation called 'yield', which passes control out of the current worker, allowing another worker to execute.

00:11:27.919 Additionally, we store the current worker into a waiting table, indexed by the I/O we are waiting on. Finally, we implement an event loop that waits for any I/O to become ready using IO.select, then resumes the corresponding worker so it can continue processing the response.

00:12:01.040 Congratulations! This is the entire implementation of non-blocking event-driven I/O for Ruby.

00:12:05.760 As you can see, the implementation is a bit verbose. We don’t want developers to have to rewrite their code like this because it exposes many details they shouldn’t have to worry about. In fact, I believe that you shouldn’t need to rewrite code at all; scalability should be a property of the runtime system, not your application code.

00:12:42.640 Let's simplify this code and hide some of the details behind a tidy interface. First, we will encapsulate the waiting list into a scheduler object. This scheduler will provide the necessary interface to wait for I/O and other blocking operations, including sleep, while concealing all event loop and operating system details.

00:13:09.440 Next, we will move the specifics of waiting on a particular I/O into the scheduler implementation. There is already a standard method for doing this called 'wait_readable,' which allows the scheduler to handle the scheduling of individual tasks without the user needing to yield control or manage the waiting list—these are typically implementation-dependent details.

00:13:51.760 Finally, we move the entire event loop into the scheduler implementation and replace it with a single entry point: 'run.' Since some operating systems have specific requirements regarding how this loop is implemented, encapsulating it within the scheduler enhances the flexibility of user code across different platforms.

00:14:46.559 As I mentioned earlier, I don't believe we should need to rewrite existing code to take advantage of the scheduler. Other programming languages often require explicit syntax like 'async' or 'wait' to achieve this, but thanks to Ruby's fibers, we can avoid such explicit syntax and conceal the 'await' within the method implementations.

00:15:15.360 In our code, we expanded the read operation to be non-blocking, but this detail can be hidden within Ruby's implementation itself. Therefore, we revert the implementation back to the original blocking read while still needing to call into the scheduler. How can we do this? In our implementation, 'scheduler' is a local variable, but we can introduce a new special thread-local variable to store the current scheduler.

00:16:02.080 This thread-local variable allows us to pass the scheduler as an implicit argument to methods invoked on the same thread. The Ruby implementation of I/O read is mostly similar to our non-blocking implementation, and it internally invokes 'rb_io_wait_readable.' In this method, we add a bit of magic: at the start, we check if a thread-local scheduler is set and, if so, defer to its implementation of 'wait_readable.'

00:16:44.960 This reroutes all blocking I/O operations to the scheduler if it is defined, all without necessitating any changes to application code.

00:17:26.060 Finally, we haven’t revealed the implementation of 'worker.' It is, in fact, nothing more than a Ruby fiber. While the practical implementation has additional internal details, this represents the complete implementation of non-blocking event-driven I/O for Ruby.

00:18:06.720 The original concept of this design began in 2017, and finally, in 2020, after extensive testing of these ideas, we made a formal proposal for Ruby 3. This design outlines the scheduler interface, which must work across different interpreters, including CRuby, JRuby, and TruffleRuby, and describes how existing blocking operations should be redirected.

00:19:00.160 The implementation of the scheduler may vary based on the underlying implementations, allowing for great flexibility. This proposal included a working implementation for CRuby and was recently approved by Matz for experimentation in the master branch, marking a significant milestone for the project.

00:19:35.680 We hope to iterate on the current proposal and deliver a robust implementation for Ruby 3. But how do we support existing web applications? Fundamentally, the scheduler is designed to require minimal changes in application code.

00:19:59.280 However, we must still provide a model to execute application code concurrently. Not all blocking operations are due to I/O; this is especially true for current database gems, which often conceal I/O operations.

00:20:18.720 Despite the challenges, this presents an exciting development: we have found that existing maintainers are willing to begin implementing the necessary improvements.

00:20:41.360 For applications requiring concurrency, you can start using Async. The core abstraction of Async is the asynchronous task, which provides fiber-based concurrency.

00:21:05.959 We introduce new non-blocking event-driven wrappers that have the same interface as Ruby's I/O classes. In many cases, you can inject them into existing code, and everything will function smoothly without requiring changes.

00:21:46.800 In Async, tasks execute predictably from top to bottom, so no special keywords or changes to program logic are necessary. In Ruby 2, you must use the async wrappers, but in Ruby 3, the scheduler interface allows us to hook into Ruby's blocking operations without altering application code.

00:22:06.960 At the core of Async is the non-blocking reactor, which implements the event loop. It supports various blocking operations, including I/O, timers, queues, and semaphores, all of which yield control back to the reactor while it schedules other tasks until the operation can proceed.

00:22:39.920 The Async gem is under 1,000 lines of code with substantial unit test coverage. We believe that reaching a stable 1.0 release with a well-defined interface and a robust test suite is essential because this establishes a strong foundation for building reliable applications.

00:23:01.920 Lately, I have been looking into Async because one of my projects would greatly benefit from non-blocking I/O, and it is designed beautifully. Janko Maron’s thoughts on the general design of Async have been very informative.

00:23:19.840 Async has a growing ecosystem of scalable components, including support for Postgres, Redis, HTTP, and WebSockets. One of our goals is to bring high-quality implementations of common network protocols to Ruby, focusing on both consistency and scalability.

00:23:56.961 Almost all components are tested for performance regressions. Since the Ruby 3 scheduler interface has been merged, we now have experimental support for their interface in Async. Once this interface is finalized, you will be able to make existing Ruby I/O non-blocking when used within an Async task.

00:24:27.200 Currently, this is experimental, but we hope to release it soon. If you write existing code with Async, you won't need to make any changes to see improved concurrency with Ruby 3.

00:24:49.920 One of the most critical components for many businesses using Ruby is the web application server.

00:24:56.640 We built Falcon as a scalability proof of concept, and its success has led us to develop it into a mature application server for Ruby web applications.

00:25:24.960 Falcon is implemented in pure Ruby, yet it rivals the performance of other servers that include native components. This reinforces our original assumptions about the diminishing returns of improving the Ruby interpreter.

00:25:51.520 Out of the box, it supports HTTP/1 and HTTP/2 without any configuration, providing a modern localhost development environment. It can also be used in production without NGINX, featuring a robust model for WebSockets. Recently, we demonstrated a single instance managing one million active WebSocket connections!

00:26:32.240 To achieve this level of scalability, Falcon employs one fiber per request, scheduling your application requests concurrently if they perform blocking operations, such as database queries, web requests, or interacting with Redis.

00:27:07.680 To facilitate parallelism across multiple CPU cores, Falcon uses a hybrid multi-process and multi-threaded model, allowing seamless support for CRuby, JRuby, and TruffleRuby.

00:27:44.160 So, is it scalable? Well, like anything, it depends on how you use it. Various micro-benchmarks have demonstrated that it can perform exceptionally well, especially when compared to operations that are blocking on other systems.

00:28:03.040 For instance, the graph shows Permalink's performance, where Falcon plateaued at 16 workers while Falcon continued to scale upward until all CPU cores were maximized.

00:28:39.200 We regard this as a fantastic outcome, especially for companies that encounter substantial traffic volume. Using Async is the right model because web applications are almost always I/O-bound, and the Ruby web ecosystem has been lacking in scalability.

00:29:16.640 For instance, WebSockets on Permalink unlock the next layer of scalability in the most Ruby-like way possible. Brian Powell has insights on migrating from Permalink to Falcon.

00:29:42.160 Now, the important question: does it work with Rails? Yes, it is Rack-compatible. However, some caveats come into play that we will discuss next.

00:30:08.640 Active Record is a commonly used gem for accessing databases, but unfortunately, it currently utilizes one connection per thread. This leads to issues with multiple independent fibers sharing the same underlying connection.

00:30:40.560 This presents a design flaw within Active Record, and discussions are ongoing on how to improve the situation. I believe Ruby 3 will drive these changes as the model for event-driven concurrency becomes clearer.

00:31:11.920 Another challenge is that database drivers do not use native Ruby I/O and are generally opaque C interfaces. However, the maintainer of the Postgres gem is eager to add support for non-blocking I/O.

00:31:30.560 I anticipate expanded support from other maintainers once the Ruby 3 concurrency model stabilizes, so these changes will hopefully propagate to higher-level libraries and drive adoption.

00:31:54.400 Alternately, we recently began implementing a low-level database interface gem called 'DB,' which currently provides non-blocking access to PostgreSQL and MariaDB/MySQL.

00:32:14.320 While we believe existing gems may eventually adopt the event-driven model, we didn’t want to wait for our implementation.

00:32:49.440 Redis is another commonly used system for swift access to in-memory data structures, widely utilized for job queues, caches, and similar tasks. We initiated a pull request to support event-driven I/O in Redis.rb, but, unfortunately, it was rejected.

00:33:13.440 Thus, we have continued to maintain the pure Ruby async Redis, which is almost as fast as the native high Redis implementation.

00:33:36.320 One essential aspect of the async gems is that they are guaranteed to be safe in a concurrent context, as our research identified many gems that are not thread-safe—including, until recently, the Redis gem.

00:34:09.840 HTTP is a fundamental protocol for the modern web, and we provide a gem called async HTTP, which features full client and server support for HTTP/1 and HTTP/2, and we hope HTTP/3 support will follow.

00:34:34.560 This gem fully supports persistent connections and HTTP/2 multiplexing transparently, thus delivering excellent performance right out of the box when paired with Falcon.

00:34:54.160 We recently added Faraday support with the async HTTP Faraday adapter, which we tested on various projects.

00:35:20.400 The GitHub changelog generator makes numerous requests to GitHub to retrieve detailed information about a project. While we haven't addressed every area of the code, we observed a notable performance improvement by fetching commits in pull requests concurrently.

00:35:55.440 For larger projects, the improvements resulting from multiplexing and persistent connections are even more significant.

00:36:15.680 In another script, a traditional HTTP gem was compared with async HTTP Faraday when fetching job status across a large build matrix from Travis, comprising about 60 to 70 requests.

00:36:45.280 Because those requests could now be multiplexed over a single connection, the overhead was significantly reduced. This highlights the advantages of leveraging async methods and underscores my excitement about the future of Async, Falcon, and non-blocking event-driven Ruby.

00:37:48.160 As a fun note, I've come across an intriguing experiment involving the FFI gem, which we utilize for implementing database drivers. This experiment offloads FFI calls to a background thread, subsequently yielding control back to the scheduler. While this approach is somewhat risky overall, it may be beneficial in certain cases.

00:38:05.680 What are the next steps? It depends on you, the community. Please test out these new gems and features within Ruby and provide feedback. The best thing you can do is enjoy experimenting with these new projects.

00:38:22.560 To businesses, please support this vital work. We recently introduced a paid model for Falcon, and if you're interested, please reach out.

00:38:45.440 We wish to support the Ruby ecosystem and foster its growth, especially regarding scalability. If this is essential for your business, you can assist us in further developing these projects through financial investment.

00:39:09.920 Thank you for listening to my talk!

00:39:16.640 You.

RubyKaigi 2020 Takeout