Letting Concurrency Help You Today

by Richard Bishop

The video "Letting Concurrency Help You Today" by Richard Bishop, presented at RubyConf 2014, delves into the often misunderstood topic of concurrency in programming, particularly focusing on its significance for Ruby developers. Bishop begins by defining concurrency and distinguishing it from parallelism, highlighting that concurrency is about managing multiple tasks concurrently, while parallelism involves executing multiple tasks simultaneously.

Key Points Discussed:
- Importance of Concurrency: Bishop emphasizes the need for concurrency in a multi-core world, arguing that maximizing hardware resources can improve performance and user experience. He notes that better designs often stem from considering concurrency in application development.
- Basic Concepts: The talk covers foundational concepts of concurrency, including processes, threads, and events, explaining how they relate to communication and parallelism in programming.
- Ruby's Concurrency Challenges: Bishop addresses the limitations placed by Ruby’s Global Interpreter Lock (GIL) and the cost of forking processes, which complicates effective concurrency utilization.
- Practical Applications: He introduces several libraries and tools to enhance concurrency in Ruby applications:
- Celluloid: A library that implements the actor model for concurrency, allowing for communication through messages between lightweight, concurrent objects.
- Sidekiq: A background processing tool that uses Celluloid to efficiently manage jobs in a multi-threaded environment.
- Sucker Punch: An in-memory job processing tool suitable for prototyping.
- Puma: A multi-threaded web server that integrates concurrency in its architecture.
- Design Benefits: The video discusses how concurrent programming leads to better software designs by promoting modularity and flexibility. Bishop compares Ruby's approach to concurrency with Elixir, a language designed with concurrency as a core principle, showcasing the advantages of concurrency-oriented design.
- Future Directions: Bishop mentions ongoing efforts in the Ruby community to enhance concurrency through better abstractions and improved garbage collection, signaling potential growth in Ruby’s concurrency capabilities.

Conclusions and Takeaways:
- Concurrency is fundamentally about coordination and communication in programming, which can lead to more efficient and scalable applications.
- Understanding and utilizing concurrency can vastly improve software design and performance in Ruby applications.
- The Ruby community is evolving towards better abstraction and support for concurrency, promising a brighter future for Ruby developers looking to harness these capabilities.

00:00:18.600 Hello, I'm Richard Bishop, and I'm here to talk to you about concurrency. Specifically, I'll discuss what concurrency is, what it isn't, and hopefully, provide you with some actionable insights for your applications today.

00:00:25.400 A little bit more about me: I’m a self-proclaimed polyglot with a problem. I love programming languages.

00:00:31.640 I don't know if it's appropriate to say this at RubyConf, but if you're not exploring another language to stretch your thinking in different ways, you really should. There's a lot of interesting stuff happening right now, particularly in how different languages tackle concurrency.

00:00:36.840 These experiences have significantly impacted me as a Rubyist. Away from the keyboard, I’m also a big coffee and beer enthusiast, and it's great that this event is in San Diego because I used to live here. In my opinion, we also have the best beer in the world, so if you come across some local brews, definitely give them a try. Just make sure you don't indulge too much, as you are at a conference, and you don't want to make a fool of yourself.

00:00:45.360 Since this is a conference, I should give a shout-out to my employer: I'm a freelance programmer. We are not hiring.

00:00:50.840 For me, learning about concurrency was somewhat accidental. I like to refer to it as 'the other side of Rack.' Most of our time is spent on the application call side, while on the completely opposite side, we navigate through web servers and system calls.

00:00:58.719 One night, I encountered an issue with bind2, and at the time, I was wondering, 'What's bind2?' It looked like just another method call, and I wasn’t even fully aware of what a system call was. This curiosity led me deeper into understanding the patterns that web servers like Unicorn implement.

00:01:05.880 It turned out to be a lot of fun, and Ruby is actually a great language to explore concurrency because it provides all the necessary primitives. That said, it does leave you wishing for more, which is part of the story of Ruby and concurrency.

00:01:12.240 So, let’s dive into what I'm covering today. We will start with some basic concepts of concurrency. Some of you might already be familiar with a lot of this, while for others, it might be brand new.

00:01:19.360 We’ll also look at practical uses of concurrency in Ruby, focusing on abstractions because that's really what concurrency is about.

00:01:24.640 While this is a Ruby conference, about 80% of this talk will be dedicated to general concurrency, and the remaining 20% will be how it relates specifically to Ruby.

00:01:32.320 Concurrency is a vast and complex topic. To kick things off, I'll start with the basics.

00:01:39.560 There’s a common exercise in concurrency talks that involves spawning threads and measuring how quickly you can populate an array with a million integers. But honestly, that never captured my interest in concurrency. It’s important to differentiate between concurrency and parallelism, and we’ll talk about those differences later. I’m not going to critique the Global Interpreter Lock (GIL) because that’s a common topic and it's here, so we have to deal with it.

00:01:50.760 Most of our work is I/O bound anyway, and I'm not going to delve into the other runtimes because we, as Rubyists, are pretty spoiled with our ability to run our code on several runtimes.

00:01:56.000 However, each of these runtimes could each represent a talk of its own.

00:02:06.759 So, why should we care about concurrency? We’ve all heard the adage that 'the free lunch is over,' and we now live in a multi-core world. We need to make better use of our hardware.

00:02:12.959 As Rubyists, we know that forking is quite expensive. If you don’t receive the server bill every month, you might not care, but I believe we should be maximizing the potential of our hardware.

00:02:18.800 Another reason is better user experience. If you examine highly concurrent, parallel runtimes, they consistently deliver stable response times, avoiding those extreme 99th percentiles and long-tail latencies.

00:02:24.879 You can’t just consider requests per second or average response times; it’s essential to look at the entire picture.

00:02:31.599 One major surprise for me while learning about concurrency was noticing that better designs arise from designing programs concurrently. Now let’s delve into what concurrency actually is.

00:02:37.879 First, it’s important to clarify this point. Concurrency often gets confused with parallelism, but these concepts are actually quite different.

00:02:42.920 Thankfully, in recent years, a clear definition has emerged, courtesy of Rob Pike, one of the creators of Go. He outlined that concurrency is about dealing with many things at once, whereas parallelism is about doing many things at once.

00:02:48.319 That's a solid definition, but if you’re like me, upon first encountering it, you might understand what concurrency isn’t, but you still won’t fully grasp what it is.

00:02:54.640 To make this more concrete, let’s consider application development. If you're working on a multicore computer using threads or a multitude of processes, those can indeed be CPU-bound up to the number of cores you have running at the same time.

00:03:01.599 However, the important thing is that those are merely points of execution; there’s no communication occurring among them.

00:03:06.720 In the context of a web server, a request acts as a boundary for concurrency. A prime example would be batch processing: even on something like MRI, you could fire up 30 Sidekiq processors. They may not all consume CPU time simultaneously, but they are indeed running concurrently.

00:03:12.360 When you group tasks together, there can be communication present. For instance, if you have a parent job, it can update its status to indicate success or failure or trigger additional actions once all tasks are completed.

00:03:19.480 So, the reason there’s so much focus on concurrency today is that we desire the benefits of parallelism, but we want a clear and concise approach to achieve it.

00:03:24.879 Ultimately, concurrency is about composition and communication; it sits at the nexus of performance and clarity. That’s why there are so many various models to approach it.

00:03:30.439 Before diving deeper, I came across an interesting tweet while preparing this presentation that I think is worth sharing. It stated that programmers love two things: making generalizations and dismissing other people's generalizations since they are often outliers.

00:03:37.760 I’m mentioning this because concurrency is an expansive topic; within 45 minutes, I can't possibly cover everything there is to know. You may have had experiences that don’t align with what I'm sharing today.

00:03:44.319 In covering the basics of concurrency, there are three essential concepts provided by operating systems and hardware developers: processes, threads, and events.

00:03:50.280 These are the foundational elements that you’ll encounter. If we examine these in the context of parallelism, on a multicore machine, processes allow for both CPU and I/O parallelism.

00:03:55.360 Threads offer the same benefits, but events, which are kernel events, are solely designated for I/O; there’s no CPU concurrency or parallelism associated with events.

00:04:01.200 The last column in this context pertains to communication, which, again, is what concurrency is fundamentally about. When numerous processes are operating, it’s crucial to establish a means of communication among them.

00:04:06.920 With processes, communication is handled via inter-process communication on the same machine, which can involve pipes, sockets, or message queues. That’s the sole way for them to relay information.

00:04:12.880 In contrast, with threads, we share memory to facilitate communication, which introduces its own set of challenges. Unfortunately, events by themselves do not offer a communication path since callbacks are simply executed as they complete.

00:04:18.840 Hence, the need for substantial abstraction when dealing with events; hence the jab at Node.js. Alone, events can become quite complex.

00:04:24.280 In other languages like Clojure and Erlang, the handling of events is usually concealed by the runtime, which simplifies interactions considerably.

00:04:30.560 In summary, concurrency revolves around composition and communication. It’s how your programs are structured versus how they are executed simultaneously.

00:04:36.240 In this talk, we will focus mainly on threads within Ruby, which have been designated as the fundamental unit of concurrency.

00:04:42.720 We have not fully embraced this concept yet, thus lacking great support for threading in Ruby. However, if we engage with threads correctly—meaning not using them directly, or not being aware we’re utilizing them—the experience becomes significantly better.

00:04:49.760 When I consider threading and concurrency, it resembles several other complex challenges we’ve faced as developers. The essence is that something should be difficult to accomplish, yet important enough to pursue, and if that’s the case, we should tackle it head-on.

00:04:55.400 Let’s take an example: memory management. We are all familiar—especially this year—with issues surrounding manual memory management, like the Heartbleed vulnerability.

00:05:01.700 However, this process was abstracted long ago; Lisp was known as the first garbage-collected language, and Ruby, of course, utilizes garbage collection.

00:05:06.960 Most of our work occurs in garbage-collected environments, and there are languages like Rust that have an innovative ownership model addressing memory management.

00:05:13.280 The key takeaway is that we must manage memory effectively, and while it is not necessary to manually handle it, understanding how it works is important.

00:05:19.440 Of course, we encounter SQL as well—who doesn’t love the occasional SQL injection?

00:05:25.920 In Ruby, we have query generators and ORM (like Active Record) to help mitigate these issues for us.

00:05:32.560 And when it comes to threading, some languages offer excellent abstractions, but Ruby unfortunately does not.

00:05:39.920 When comparing memory management to threading, I think of threads and mutexes as akin to malloc and free. We should be aware that they are in use somewhere, but ideally not have to deal with them daily.

00:05:47.040 In Ruby, we certainly love processes. We take joy in forking and utilizing all that RAM, but this comes with high costs.

00:05:54.280 Communicating effectively between several processes can be challenging. I refer to this form of communication as passive-aggressive because one process sends information to a message queue without knowing which other process will ultimately receive that message.

00:06:00.320 Without direct communication, it's difficult to construct truly robust and composable systems.

00:06:07.160 Fortunately, due to our community, there are some libraries available today that we can utilize to ease these challenges. I want to highlight a few, with hopes of motivating you to start using them.

00:06:14.440 The first library I'd like to discuss is Celluloid. Developed by Tony Arcieri, it's an excellent library for concurrency.

00:06:20.000 If you’re familiar with the actor model, as seen in languages like Erlang and Elixir, Celluloid implements this model well. It allows you to harness concurrency primitives in a familiar manner, where actors execute concurrently and communicate through messages.

00:06:26.760 Some people argue that methods are messages in a way, but I’ll show how this view can be inaccurate later on.

00:06:33.160 Celluloid integrates the actor model into Ruby objects, providing a cohesive library for concurrency.

00:06:39.200 Interestingly, this has become one of the most downloaded gems that many users may not even be aware they are using, as it’s employed by several other outstanding libraries.

00:06:45.320 The library also includes handy built-in features like pooling actor objects for work distribution and supervisors designed for fault tolerance and reliability.

00:06:51.600 To illustrate an actor in Celluloid, you can envision it having a mailbox in which messages are stored. The mailbox functions as an advanced queue that processes incoming messages in order.

00:06:58.120 When messages are retrieved from the queue, the actor processes them and proceeds with its functionality.

00:07:04.560 In Celluloid, the message contains the method to invoke and the respective arguments to provide.

00:07:10.240 Here’s a simple, albeit contrived, example of using Celluloid. When mixing it in, you can instantiate an object.

00:07:16.480 Notice that the result isn't the defined class but rather a wrapped instance inside a Celluloid proxy, through which you can communicate by sending messages asynchronously.

00:07:22.000 You utilize the async method to get back an async proxy, allowing you to call methods like greet on it.

00:07:28.080 However, it’s vital to emphasize that method calls aren't messages; this distinction is crucial.

00:07:34.440 Method calls are synchronous by nature and must wait for completion before proceeding.

00:07:40.560 One way to bypass this delay is to spawn a thread to 'fire and forget.' However, this approach is not something recommended for frequent use.

00:07:46.760 In the actor model, messages can be both asynchronous and synchronous. You can opt to require a reply after sending a message, but the initial state remains asynchronous.

00:07:53.160 Celluloid implements this functionality through metaprogramming. When invoking async along with a method, the async proxy carries out method_missing, wrapping the call and forwarding it for later processing.

00:07:59.760 The reason I wanted to highlight this library first is that the next one relies on it, and I hope everyone here is already using it: Sidekiq.

00:08:05.440 Sidekiq is a phenomenal background processing library developed by Mike Perham. In my experience, one Sidekiq worker can replace between 8 to 20 classic Resque workers.

00:08:11.480 This efficiency stems from its multi-threaded nature, offering great benefits even in MRI because of I/O parallelism.

00:08:18.360 Examining Sidekiq’s design reveals that it utilizes Celluloid behind the scenes, featuring a launcher object that initializes fetchers and managers.

00:08:24.920 The manager oversees these processors, which are responsible for instantiating and executing your jobs. This clear separation of concerns is remarkable.

00:08:31.160 All these elements operate concurrently, fostering effective communication through messages. The manager is able to send requests to the job processor and the fetcher sends tasks to the manager.

00:08:37.600 This leads me to another question: who here is currently utilizing Sidekiq? That’s a good number of you!

00:08:44.040 Ruby is popular among startups, and I understand that startups often feel incredibly busy, leaving little time to consider concurrency. This brings me to another intriguing background job library called Sucker Punch.

00:08:51.520 Developed by Brandon Hilyer, it also uses Celluloid and operates entirely in memory, sharing the same API as Sidekiq for easy upgrades.

00:08:58.320 However, Sucker Punch is not recommended for crucial tasks since it is purely in-memory; if your process crashes, the queue is lost.

00:09:05.840 Nonetheless, it’s quite effective for prototyping and similar scenarios. The last library I’d like to discuss is Puma, a multi-threaded web server.

00:09:12.240 Puma supports forking and includes a reactor, offering a comprehensive concurrency model.

00:09:18.920 Examining Puma’s design reveals a clear separation of concerns. It features a thread pool, a popular concurrency abstraction with shared resources among threads, guarded by a mutex around a request queue.

00:09:25.600 On the contrary, an acceptor thread handles TCP requests and transfers them to a request queue that gets processed by the threads.

00:09:32.720 This design fosters straightforward separation of concerns, essential for effective threading. However, potential bottlenecks exist, with one being the mutex synchronizing access to the queue.

00:09:39.520 Fortunately, this isn't a major concern because it’s often overshadowed by the real bottleneck within Ruby itself.

00:09:46.560 To recap this section, everything we've discussed revolves around abstractions. You've seen actors and thread pools, and I hope I’ve encouraged you to explore libraries that can enhance your efficiency significantly.

00:09:52.960 The next segment delves into what I've found particularly fascinating about concurrency: the design benefits that arise when you leverage concurrent programming.

00:09:59.840 In today’s landscape, there’s an enormous emphasis on software design—many discussions revolve around concepts like designing in small scopes, service-oriented architecture, service objects, or microservices.

00:10:06.720 Essentially, what design boils down to is the pain experienced when working on a single-threaded synchronous application with poor structure; the resulting unexpected bugs and challenges in making changes lead to slower iterations.

00:10:14.840 Conversely, when dealing with a concurrent application, flawed design introduces not just the same issues, but also deadlocks, race conditions, and additional complications.

00:10:21.600 Recognizing this enables you to adopt a more merciless approach to your design process.

00:10:26.960 I want to briefly introduce you to another language called Elixir. While I can't thoroughly cover its intricacies at this moment, I’d like to provide some high-level insights.

00:10:34.720 Elixir, created by José Valim, runs on the Erlang virtual machine. If you've met an Erlang programmer, they've likely extolled the numerous advantages of the Erlang VM within moments of conversation.

00:10:40.880 The striking aspect of both Elixir and Erlang is their paradigmatic approach to concurrency. They treat concurrency as a standalone paradigm, much like object-oriented or functional programming.

00:10:48.080 Within these languages, you'll find structured applications defined by supervision trees, where individual components run concurrently and communicate through message passing.

00:10:54.000 What’s notable is that the red circles, known as supervisors, manage these actors. These structural distinctions enhance flexibility and adaptability in your designs.

00:11:00.160 When crafting Elixir applications, you begin by categorizing your library code—your models, parsing logic—and progressively implementing the concurrently running actors.

00:11:06.960 This method increases scalability rates tenfold. In a typical four or eight-core machine, one can run 300,000 to 1,000,000 actors concurrently.

00:11:13.200 In the Erlang world, while some naming conventions may be obscure, this structure greatly enhances modularity. Each actor, or process as they refer to them, can be independently deployed.

00:11:19.440 In contrast to the Rails world, which often revolves around monolithic deployments, this modular design allows for granular scalability.

00:11:25.920 With a traditional Ruby application, a large component could hinder deployments as scaling typically involves deploying the entire code base. However, in the Erlang context, individual components like caches can be upgraded effortlessly.

00:11:32.080 A common challenge in Ruby applications is balancing simplicity in code structure against the complexities arising from deployment strategies.

00:11:39.680 We typically favor simplicity during design, but this often leads to convoluted deployment structures, with multiple processes necessitating varying commands.

00:11:46.280 Elixir’s architecture contrasts with Ruby's historical approach. The simplistic view of single-threaded code conceals a complex deployment reality that involves multiple processes.

00:11:53.520 Thus, while single-threaded synchronous programs might seem easier to manage, they introduce complexities that have a cascading effect on deployment configurations.

00:11:59.840 To reiterate the elegance of Elixir's concurrent design, all components are executed concurrently within a single code base, enabling individual scaling.

00:12:06.640 This modular approach is desirable in any programming scenario, especially when seeking to leverage the advantages of concurrent design.

00:12:13.720 In Ruby, there’s often a tendency to simulate that kind of scalable design, often resulting in various services that complicate our operational landscape.

00:12:20.360 When deploying services, several teams often create redundant systems that might only slightly partition a monolithic codebase.

00:12:27.600 A prevalent notion is that you need services to handle individual aspects because your Ruby application is deployed too frequently.

00:12:34.000 However, what ends up happening is that you cobble together message queues and provide HTTP communication between services that were meant to be decoupled.

00:12:40.640 In practice, this often leads to what I would describe as repository-oriented architecture, with a haphazard web of interdependencies.

00:12:47.920 While SOA is often justified by the promise of clean separations and individual scaling, the process of using message queues often complicates these aspirations.

00:12:54.720 With messages crossing various components through HTTP, you can find yourself coupling services unnecessarily over the network.

00:13:01.920 To further recapitulate, concurrent design mandates that you start with modular architecture; it’s a fundamental requirement in languages like Elixir and Erlang.

00:13:08.240 With adequate scaling, independent deployment becomes possible for each actor or process, allowing teams to adjust units without compromising a central application.

00:13:15.200 In contrast, designing for concurrency in Ruby often leads to intricate service-oriented architectures where pulling one thread can unravel the entire structure.

00:13:22.240 The final aspect I want to cover relates to discussing the Ruby concurrency model, particularly as it pertains to the GIL.

00:13:28.960 While I haven't criticized the GIL yet, I do want to highlight that JRuby is an excellent option if GIL is a significant issue for you.

00:13:35.680 To comprehend Ruby's structure, we should acknowledge its design history dating back to the '90s. I wasn’t programming back then, but looking back, one can see the patterns.

00:13:42.360 Many languages from that time share common traits. A programming language often starts off by selecting a paradigm, like object-oriented programming, of which Matt chose a beneficial version.

00:13:49.480 Towards the end of that design phase, someone slaps on some basic threads and considers that the concurrency model. Ruby had green threads from the outset, albeit not really built for concurrency.

00:13:56.640 In that period, concurrency didn’t even register on most coding radars, hence the GIL's existence.

00:14:04.880 Flash-forward to the 2000s and beyond, you notice a shift. Languages like Go actively prioritize concurrency right from their design.

00:14:11.520 The design is such that steps toward concurrency are inherently integrated, involving aspects like immutability that frame the core model.

00:14:18.520 In Ruby, the perception is that concurrency and parallelism should mesh seamlessly, as if Matt were to simply flip a switch.

00:14:25.760 However, the reality is that other elements require addressing first; primarily, a competent garbage collector.

00:14:32.680 If concurrency and parallelism are introduced without an efficient garbage collector, you risk stalling due to world stops and facing significant performance issues.

00:14:39.040 Currently, considerable efforts are underway to improve Ruby's garbage collector, as seen with versions 2.1 and 2.2.

00:14:46.520 Secondly, we need to develop abstractions beyond the current, limited threading and mutexes we have in Ruby today.

00:14:54.400 Currently, the core provides basic tools like threads, mutexes, and condition variables, falling short compared to many other languages.

00:15:02.800 Rubyists often find themselves envious when comparing their threading structures against those in languages like Java.

00:15:10.520 One ultimate factor influencing Ruby's concurrency struggles is its culture of simplicity. We leverage Ruby because it eases our programming tasks and allows for enjoyable coding experiences.

00:15:17.640 However, much of the current concurrency discussion is rooted in cutting-edge theories, often dating back to the 70s and 80s, which frequently lack clear implementation guidance.

00:15:25.560 Languages like Go have teams with extensive Unix programming experience, leading to a sound implementation process with multiple iterations based on learned experiences.

00:15:32.680 As a result, the journey toward implementing actionable concurrency models is intricate and demands time for exploration.

00:15:39.520 Matt recently expressed a commitment to introducing abstract concurrency units such as actors, signaling that a change may be on the horizon.

00:15:47.160 The community can anticipate the arrival of improved abstractions soon—when they emerge, they will provide the needed simplicity alongside advanced capabilities.

00:15:54.680 That concludes my talk. Thank you for your attention!