RabbitMQ to the Rescue

Talks

Andreas Finger

#event-driven-architecture

#message-queues

#ruby

#asynchronous-processing

RabbitMQ to the Rescue

by Andreas Finger

This video titled 'RabbitMQ to the Rescue', presented by Andreas Finger at Ruby Unconf 2018, delves into the use of RabbitMQ as a message bus to achieve decoupling of services in software architecture. The main theme revolves around how message buses help in effective asynchronous processing, handling high volumes of messages, and enhancing overall system responsiveness.

Key points discussed include:
- Purpose of Message Buses: They aim to decouple services, transitioning from monolithic applications to microservices, thereby improving scalability and maintainability.
- Event-Driven Architecture: The example of a user registration process illustrates how RabbitMQ allows different systems (like email notifications and Salesforce) to react to events independently.
- Handling High Volume Registration: Emphasizes the importance of message buses during scenarios of sudden spikes in registrations, allowing systems to queue messages and process them as resources become available.
- Message Acknowledgments: Discusses the concept of positive and negative acknowledgments to assure message delivery, emphasizing the trade-off between speed and security.
- Types of Exchanges: Explains the different types of exchanges (direct, fan-out, topic, and headers) that manage routing messages based on specific use cases.
- Queue Management: Describes how consumers manage messages from queues with acknowledgments and how to handle rejected messages using dead letter exchanges.
- Practical Coding Examples: The presenter provides Ruby code examples using the Bunny gem to illustrate the setup of publishers and consumers, creating channels, exchanges, and binding queues with routing keys.
- Monitoring and Management: Stress the importance of monitoring queue lengths to maintain effective message processing and prevent system overload.

The major takeaway from the presentation is that RabbitMQ is a highly effective tool for managing message systems, allowing flexibility in architecture while ensuring reliability and performance in message handling. Finger encourages attendees to explore RabbitMQ further and consider its application for their systems, underlining its stability and vast client support.

00:00:17.630 Hello, can you all hear me? Roll this microphone, no pizza. Microphone. Speak up, speak up. Big ups because I have to shout in New York.

00:00:29.580 I'll try, please raise your hand if you can't understand me. Is anybody here? Okay, so this talk is about message buses.

00:00:39.949 I'm using RabbitMQ as an example. I hope all of you have been in casting storms first talk in this room here today.

00:00:49.920 She was talking about background processing and asynchronous processing. Because I said she already raised an understanding of why you want to use a message bus, why you want to decouple your systems.

00:01:00.979 That was a good introduction. I'd say one of the main goals of it is to decouple your services.

00:01:09.090 You often start with a monolith and then want to extract parts out of it. You end up with services, which you might call microservices. Most likely, they are not microservices because they are still really entangled with the main system, giving all those synchronous calls.

00:01:28.530 Imagine you have one application, and another user fills out a registration form. Then you have your main registration application, which communicates with the main application to get information about the user, and it also connects to the mail application to send the welcome email. You have all the control in one application; it's not decoupled.

00:01:50.850 If you could send messages, after a new user registration, you would simply pass that user's database ID to your email system, which would listen to this event. It would know, "Oh, a new user registration!" and get the user information.

00:02:10.470 The email system would then send out an email. You can even easily change the system; for instance, let's say you also want to push this data down to Salesforce. You just add a Salesforce listener that listens to the same event without changing any code on the other side.

00:02:30.739 That listener would add user information to Salesforce. So, that's a significant benefit. You can also use message buses to handle high volumes of messages.

00:02:50.920 Let's say this user registration happens after you run a TV commercial. When I worked for a gaming company, after a good spot, you might see 50,000 new registrations in just a few minutes. The system has to handle that load.

00:03:09.140 The user fills in a form, and the system saves that to a database, telling the user, "Great, you will receive a welcome email." That's fine if you have decoupled it. If your queue can handle all those messages while other systems need a bit more time to process them, to send out the emails or do whatever other tasks need to occur, your system will stay responsive.

00:03:27.380 It won't fail, and you won't lose any of those expensive registrations. You can also use it to broadcast messages—one event and just shout it out into the wild, adding listeners time after time.

00:03:44.000 You may run something like a news feed, which is currently only used by a few sites in Germany, but as you become more successful, other people will also want to use it. You don't have to do anything; you just shout out events, and you tell your clients, or consumers, to listen to those messages.

00:04:06.440 You can even use the same mechanism to replicate your data to another region in the world.

00:04:14.060 For example, you might have your RabbitMQ broker instance here in Germany and another one in Asia.

00:04:19.419 You can replicate the data there for faster access on both sides. So, I don't have time to go through all of that. I used this presentation before, and I have too much content on the slides.

00:04:31.030 On the left side, you see the publishers. On the right side, these are the consumers. In the center, we have basically RabbitMQ—the broker itself.

00:04:39.490 As you can see, the publishers publish to exchanges; they do not publish to queues. It always goes to an exchange, and the exchange routes the messages to a queue. I will explain how that happens.

00:04:54.400 Then, the queues connect to the consumers. In our case, queues will push messages to the consumers. This reminder leads me to explain 'ack'.

00:05:05.830 'Ack' is short for acknowledgment. Right now, we're just throwing all the messages down the pipeline.

00:05:12.520 At some point, you might want to have feedback: was this message received? Was it processed? Was there any issue? For that, we talk about acknowledgments.

00:05:25.690 You can have a positive acknowledgment, confirming receipt of a message, and you can also have a negative acknowledgment, which basically means to reject a message. Let’s look at some examples of how to handle that.

00:05:39.830 Starting with the left side, publishing messages: it's all about the trade-offs between speed and security. RabbitMQ in particular gives you lots of different options on how to handle sending messages and consuming messages.

00:05:55.060 What you do in between messages is all about trade-offs. Do you want to ensure that your message was received and handled, or do you not care so much? Because maybe you have locks, or you just want to push messages through the system as fast as possible.

00:06:08.200 If speed is more critical than reliability—if you don’t mind if a few messages are dropped—then you can go fast, knowing that there are no guarantees.

00:06:22.990 Messages are just sent from A to B, and you hope the consumer handles it. If a couple of messages drop, you don’t mind as much; it’s more important that the system doesn’t slow down.

00:06:33.220 However, there's an issue: you might overload the consumers and drop a lot of messages. In RabbitMQ, this is often handled through publisher confirmations.

00:06:45.430 The publisher confirmation happens on the left side of this diagram. When the broker is able to route a message from an exchange to a queue, and the message is sitting on that queue, a confirmation is sent to the publisher, signaling that all is well.

00:06:57.110 So, while the message is not consumed yet, the publisher knows it will be handled.

00:07:06.800 You can't really lose your messages this way; you can ensure that messages are usually routable in RabbitMQ. However, you need a lot of memory if your system is fast enough.

00:07:18.960 If the memory load is high, this means your queues should empty themselves really fast, which is fine. You can also persist them to disk if you’re afraid that your RabbitMQ instance could go down. It shouldn’t, but you know it can happen.

00:07:35.330 RabbitMQ doesn't use Redis or similar tools; it has a built-in database optimized for writing, while most other databases are optimized for speed.

00:07:46.620 That’s because it only writes a message to the database. It never reads it, and it deletes it when handling it. It may only read messages if something goes wrong.

00:08:00.060 You can create a cluster—you usually work with two types of nodes: read nodes and write nodes. It’s often used not for performance but for increased redundancy.

00:08:12.860 This means having one master that replicates data to other nodes. If the master goes down, replicas can take over.

00:08:23.820 All this configuration can happen automatically across the board.

00:08:30.600 We call these queues high availability queues, and that summarizes the fast left side.

00:08:36.720 Now let's move to the right side of consuming messages, keeping in mind the same trade-offs of speed and security.

00:08:46.250 When listening to messages, if it's too fast, we can overload the consumers, so that's something you may want to prevent.

00:08:55.070 In the fastest mode, there are no guarantees. There are no acknowledgments—nothing. Just give me all the messages you have, and I'll attempt to handle them.

00:09:14.190 You can set a quality of service level, which means you set a prefetch count—for example, ten messages, fifty or a hundred, whatever you choose.

00:09:32.350 After consuming that number of messages, the consumer would send an acknowledgment that it received those messages.

00:09:41.780 Both sides—the queue side (the broker) and the consumer—account for messages. The broker sends ID to confirm if messages are received.

00:09:58.200 If not all have been received, it will send only the messages that were not confirmed again so you never lose messages, and it's quite fast without too much overhead.

00:10:12.920 You can even be more granular. You can send acknowledgment after each message manually. An automatic acknowledgment would be sent when a message was received by the consumer.

00:10:28.220 The consumer can just send an acknowledgment, saying "I got it." The issue is that your consumer might crash or fail to process this message, and you would lose it.

00:10:46.580 So you usually want to send manual acknowledgment and do it after processing the message. This makes the whole process a bit slower, but it's a much safer way to do things.

00:10:57.990 This all depends on how important the information is that you're transferring.

00:11:12.920 When I'm talking about acknowledgment, I should mention non-acknowledgment or rejection. The consumer may try to handle a message that doesn't work; for instance, the data wasn’t formatted properly, or the external system it needed to communicate with is unavailable.

00:11:26.470 What happens when I reject a message? Theoretically, you could send the message all the way back to the publisher, but that's usually a bad idea. The publisher just gets a message that it sent like minutes ago or even seconds or an hour ago.

00:11:41.820 This creates excessive logic in the publisher. The consumer should handle it itself and determine how to manage messages that fail.

00:11:57.320 Typically, we reject messages and then provide instructions to another system about how to handle those rejected messages.

00:12:10.340 So, what are coherent nodes? I don’t have time for examples. We talked about the left and right sides: the publishers and the consumers. Now let’s come to the center: the broker.

00:12:29.470 The broker and exchanges route messages. To connect publishers and consumers, we have at least one exchange.

00:12:44.880 This exchange will bind queues. Each exchange will bind to at least one queue, and typically many of them.

00:12:58.080 The binding queues are handled in various ways, as I will explain. The broker routes messages from exchanges to queues.

00:13:09.140 There are different types of exchanges for different use cases. In a direct exchange, routing matches exactly with a binding key. If the string matches, then you receive this message.

00:13:27.160 With a fan-out exchange, every queue that binds to a fan-out exchange will receive all messages sent to that exchange. It's like broadcasting.

00:13:42.470 The most interesting one is the topic exchange, where routing keys use dot separators and queues find a binding key that matches the pattern.

00:13:59.560 For instance, you can match a word with asterisks, and hashtags can match a word followed by dots.

00:14:15.470 The whole string can emulate a direct exchange with the same string on both sides, or emulate a fan-out exchange with a hash symbol to receive all messages.

00:14:28.000 There are more types of exchanges, such as header exchanges that are part of message properties. It's basically a hash with key-value pairs you can use for routing.

00:14:39.250 RabbitMQ allows you to add plugins for more unusual types of exchanges you might need. You can also use exchanges to handle message failures.

00:14:56.060 We have alternate exchanges. Whenever routing doesn’t work, and a publisher sends a message to an exchange that cannot route it due to no queue defined with that key, it can happen.

00:15:11.320 In such cases, you can use alternate exchanges to handle those unrouteable messages.

00:15:25.130 For instance, the message was received by the exchange, but if it doesn't work, it can be sent to a different exchange, an alternate exchange, and you can monitor what happens with messages in that exchange.

00:15:39.000 For consumers, if a consumer receives a message but can't handle it, it rejects that message. We talked about what to do with rejected messages—it's a good way to deal with them using a dead letter exchange.

00:15:50.690 A dead letter exchange can have two exact same queues created from a normal exchange. It captures all the rejected messages. What happens next really depends on what you want to do.

00:16:06.890 You could implement a retry mechanism—maybe attempt to process them in an hour or a day—or you could handle them manually. You could go in, see what didn't work, and correct issues like broken source code.

00:16:28.840 Alternatively, you could have an automated process that routes those messages to an error logging or monitoring system.

00:16:39.720 So, you might end up deleting the queues if they didn’t work, but at least you collected the routing information.

00:16:57.030 Keeping in mind the previous examples of routing, I want to concentrate on traffic exchanges here.

00:17:09.280 Imagine we have an existing schema utilizing cities, topics, and categories for a service connecting tech communities around the world.

00:17:25.020 The schema could look like this: five cities, corresponding to topics and categories; this gives us 125 possibilities to create routing keys.

00:17:41.110 For example, Barcelona can serve as either an event or a job. If we have job postings, they could relate specifically to test jobs in Ruby.

00:17:58.100 These keys would start with ´Barcelona´ as I'm currently in Barcelona.

00:18:12.310 So, we can have various routing keys like 'Barcelona.event' or 'Barcelona.job' to capture particular events or job postings.

00:18:29.570 We can bind to queues with these keys, utilizing pattern matching. Thus, a binding key could be 'Barcelona.#', which captures all related events.

00:18:40.950 If we were to take it a step further, it might be like routing keys like 'registration.user.new' for registrations or updates.

00:18:57.000 That’s the central part: the exchange decides which queue receives which messages.

00:19:09.960 One message can go to multiple queues. It doesn’t need to go to just one. Every queue with a matching binding key will receive the message.

00:19:27.430 A queue can have multiple consumers, allowing a load balancing effect, ensuring tasks are handled more efficiently.

00:19:39.510 This load balancing is typically done in a round-robin manner. You can also have several consumers listening to different queues to receive more messages.

00:19:55.250 You have great flexibility in how you structure your system and how you extend it.

00:20:13.270 So, we’ve talked about publishers, the broker with exchanges and queues, and consumers, but we haven't discussed the individual messages.

00:20:32.980 In this example, the messages are indicated by arrows passing through. We have the method frame, not mentioned so far, which includes routing information like the routing key.

00:20:50.200 We also have a content header frame, which contains message properties. For instance, looking at the 'use by' column, we find information relevant for you, the user.

00:21:05.920 This isn’t used by RabbitMQ, allowing you the freedom to utilize it as you prefer. Important properties may include content type and content encoding.

00:21:23.730 The publisher should specify the content type—for example, application/json—and the encoding, such as UTF-8, which informs how the message should be read.

00:21:40.330 Usually, the consumer should deserialize the message properly. Instead of passing raw JSON to the application, you may want to handle it as a Ruby object or at least a hash.

00:21:56.960 The message also encompasses optional fields such as message ID, which you can create yourself, and timestamps.

00:22:15.300 You can define an expiration time, where the message would automatically be deleted or rejected if it's too old.

00:22:33.950 You can set a delivery mode, correlation ID, and priority properties. We haven’t discussed the payload part yet.

00:22:49.500 The payload is part of the body frame, which constitutes what we're sending—our actual message. This may be as simple as a user ID or a more complex object that depends on your system's setup.

00:23:06.760 It may contain more information if you plan to migrate to a different system later on, as other systems may not possess all the meta-information RabbitMQ offers.

00:23:22.469 The important thing to keep in mind is that the payload is the body of the actual message.

00:23:37.340 I realize I may have skipped some content; I have some coding examples which are written in Ruby and utilize the Bunny gem.

00:23:50.540 This is the most popular solution you'll find nowadays, so please bear with me. In the coding example, you can see my Ruby RabbitMQ system.

00:24:06.020 I initiate my Bunny object, establishing a new connection and creating a channel for the connection.

00:24:24.210 This process occurs once for the publisher and once for each consumer. Keep in mind, the consumer and publisher can be part of the same application.

00:24:41.450 You might encounter a monolith, but you still need background processing in between. So, you can publish a message and consume it later.

00:24:57.040 This setup is the same for declaring an exchange. The queues are initially connected, allowing us to bind them together.

00:25:12.700 In the last line, we see that the queues are created and the exchange bind themselves, utilizing the binding key.

00:25:31.360 In this case, I'm not extensively using pattern matching, but if you do utilize a topic exchange, you can also publish messages.

00:25:49.610 I’ll publish messages with the relevant information, including the message ID and any data properties that come into play.

00:26:04.440 This message publishing acts as a pairing between the sender and messages being correctly routed.

00:26:24.950 Now, when consuming messages, you'll create another channel connected to the same queues to subscribe consumers.

00:26:40.380 The consumer is defined by which channel and queue it belongs to. The consumer tag will be an ID generated by RabbitMQ, or you can set your own IDs.

00:26:56.930 There are various options to set here, including the automation of acknowledgment procedures.

00:27:09.970 After subscribing to the queue, you're creating a callback for message delivery, securing that you receive the proper properties and payload.

00:27:22.870 This example indicates that various methods of acquiring and managing messages are highlighted. The payload can have various other properties.

00:27:35.830 Lastly, you can cancel subscribing when needed—maybe during deployments or for maintenance.

00:27:49.420 When you take the consumer offline, all messages will then wait in the queue for you.

00:28:00.970 One system integrates all messages while keeping redundancy and allowing you to monitor the state of your queues.

00:28:13.300 In an ideal world, your queue lengths would always be zero, indicating that all messages arrive and get consumed immediately.

00:28:31.120 You may also want to monitor different processes, such as long-running jobs that may take longer than expected to finish.

00:28:50.230 Setting some thresholds is ideal. If the queues are filling up significantly, you might consider investigating the underlying processes.

00:29:02.610 When it comes to clusters, I won't delve into all setup practices. Some best practices will be marked, and I’ll send you a link after this.

00:29:11.990 It's worth examining the software that makes using RabbitMQ easier, as well as client libraries.

00:29:28.170 At this Ruby conference, I, of course, have to mention the Bunny gem. But you don't have to rely solely on the gem; frameworks also exist.

00:29:47.180 The variety of client applications in different programming languages increases, so as highlighted, RabbitMQ has been stable for over ten years.

00:30:00.680 Coming from banking and finance, it’s well-managed and documented. The top link is a great reference to the documentation.

00:30:15.580 Download it; you can try it easily. Thank you very much for your time, and I hope you enjoyed learning about RabbitMQ. You earned yourself a piece of cake and coffee. I’ll see you back in this room in 45 minutes; next up will be Anna talking.

Ruby Unconf 2018