00:00:08.280
Hey everybody, my name is Gerred Dillon. Today, we're talking about messaging patterns in Ruby. As I said, I work with Quick Left as a software engineer, and you can find me on Twitter at Justice Rights.
00:00:22.619
Let's start off by welcoming you to the enterprise world. The big blue dot represents the money; it's all about enterprise.
00:00:29.939
When we look at the enterprise, we discuss a vast array of components—many of which we typically prefer not to interact with directly. This complex system is a modal mass of many parts. Who here has dealt with a massive system like this? Well, I have too, and it’s not always the best experience. For the purpose of this talk, we will treat it as a single mass, and I'm going to label these components—I owe thanks to Jeff for the inspiration.
00:01:02.640
We want to integrate with this system, but for various reasons—political, coding practices, or otherwise—we can’t connect directly to it. This forces us to operate outside the main system's sphere. Let's consider our application as a Rails application, which is a good starting point. Our goal is to find the best way to integrate multiple systems.
00:01:24.360
One of the first ideas that comes to mind is, 'Let’s create an API between the two systems.' This approach means sending requests out to fetch necessary information; we can handle a couple of these requests adequately. However, the complexity grows; as we need to interact with more endpoints, things can become messy very quickly.
00:01:41.100
It's not just our requests—everything inside the system may also require consuming information from us. We begin to create multiple endpoints, leading to a chaotic mess that no one wants to manage. The complexity increases as we attempt to abstract away this heterogeneous environment.
00:02:00.600
If we have a public API, it complicates things even more; managing multiple systems becomes necessary. Besides, our APIs, by their nature, tend to be unreliable. We need to get into the habit of error-checking all these connections, making sure everything runs smoothly, all while ensuring that operations remain synchronous. It's clear we need a different approach.
00:02:30.420
You might ask, 'Okay, so what can we do to resolve these complexities?' We need to address a few issues: we are blocking our main application, maintaining a heavy load, and struggling with scalability. One option is to talk about an intermediary broker that can manage all these interactions, allowing us to consume data at our own pace. This effectively shifts our system towards an asynchronous model.
00:02:59.640
Communicating through a message queue can help us achieve this. In today's talk, I will focus on RabbitMQ, which is a messaging queue based on the AMQP standard developed by JPMorgan Chase in 2005. RabbitMQ was specifically designed to resolve the integration of numerous systems with various code bases asynchronously. It’s proven to be effective and dependable.
00:04:23.820
Once we implement RabbitMQ, we can eliminate concerns about the wider system, allowing our Rails app to focus on interactions solely with the messaging server. We optimize our communication in a way that allows operations to run smoothly without direct dependencies on other systems.
00:04:42.660
This now lets our systems collaborate efficiently, keeping everything running smoothly. We transition from a convoluted mess to a more structured workflow, emphasizing the importance of how our Ruby application interacts with other components. We still have to uncover the larger narrative: our Rails application is complemented by a web server, and now, we’ve introduced workers—background processes specifically designed for handling tasks concurrently.
00:05:54.660
These workers can process tasks for us, pulling information from the queue and executing whatever is needed. Here’s a basic example using the Ruby AMQP gem in an event machine loop. The Ruby AMQP gem runs on top of an event machine, which by the way, is designed for concurrency through event registration.
00:06:10.680
To subscribe to the channel of interest, we can specify a transactions queue. When we receive a message, we simply put it out—it's straightforward. This model allows us to scale horizontally; if we have three workers that can process three messages at a time, what happens when they become overwhelmed? We can increase the number of workers, continuously scaling our resources to handle the growing load.
00:06:54.660
This async approach shifts thinking—no longer will everything happen directly, impacting system performance in a synchronous manner. Instead, we create a buffer that decouples our systems, addressing potential bottlenecks within our Rails application too. As we continue to run more workers, operational bottlenecks within our application can emerge.
00:08:53.820
In the event of an overload, traditionally we would diagnose and mitigate these bottlenecks, which is still valid. However, we could consider making our controller actions asynchronous. What if we outsource data generation, allowing it to be processed by background workers while our application remains responsive?
00:09:27.060
By deferring tasks to workers, we allow the web application to focus on what it does best—serving requests and business logic. Still, that introduces another layer of complexity as we must also consider how our database server interacts with this model. We must strike a balance between handling requests efficiently and ensuring database performance.
00:10:19.160
The performance bottleneck might occur here, and we need to devise a way to optimize interactions with our database server. Optimizing controller code could be as simple as ensuring that we perform operations asynchronously where possible. Using worker systems to offload tasks can lead to more scalable solutions.
00:10:57.480
For instance, the Delayed Job gem is quite user-friendly, allowing for ease of implementation. However, it still ties us back to the database’s performance limitations since it queuing is still reliant on that performance. An excellent alternative could be using something like Queue Classic, which utilizes PostgreSQL features to manage queues independently of the database constraints.
00:12:02.460
Queue Classic offers a decoupled approach to job processing, but let's not lose sight of the original problem. Adding processes can improve efficiency but often adds layers of complexity. This challenges developers to ensure that these idiosyncratic queues don’t lead to additional performance constraints.
00:13:05.620
Resque is another popular choice, built on top of Redis, a lightweight key-value store that facilitates rapid job processing. Redis possesses some advanced capabilities that allow developers to build robust worker systems while maintaining reliable performance. It’s particularly well-suited for scenarios requiring fast data retrieval.
00:14:11.560
In this asynchronous model, we can design our controllers to enqueue jobs efficiently. By doing so, we maintain a flexible approach that allows our application to serve responses quickly. While the setup appears straightforward, it necessitates a mindset shift regarding data management within our application.
00:14:55.360
In an asynchronous model, you are rendering responses while delegating job processing. This separation proves invaluable as we continue to scale our systems. Even as workloads increase, adding more workers allows us to efficiently manage queues and optimize requests, creating a seamless user experience.
00:16:39.680
Through this approach, users interact only with responsive components, free from the burdens of underlying processes. When it's time to deliver results, we can push data to the user without delay. This enhancement significantly shifts the thought process behind web applications, changing our focus from synchronous operations.
00:17:52.740
In closing, we have managed to tackle synchronization and blocking issues, particularly concerning our interactions with external systems. By turning to asynchronous structures, we reduce bottlenecks in both our external APIs and the flow within our applications.
00:18:00.780
We've also considered alternatives that improve our systems without overburdening our databases. This model not only enhances performance but fundamentally alters the way we approach application design.
00:18:07.500
I’m now open to any questions. Thank you for attending my talk.
00:18:22.680
The follow-up questions led to an insightful discussion about database constraints, queue management, and worker performance. Addressing their concerns, I emphasized the importance of redundancy and persistence in job queues.