00:00:17.630
Hello, can you all hear me? Roll this microphone, no pizza. Microphone. Speak up, speak up. Big ups because I have to shout in New York.
00:00:29.580
I'll try, please raise your hand if you can't understand me. Is anybody here? Okay, so this talk is about message buses.
00:00:39.949
I'm using RabbitMQ as an example. I hope all of you have been in casting storms first talk in this room here today.
00:00:49.920
She was talking about background processing and asynchronous processing. Because I said she already raised an understanding of why you want to use a message bus, why you want to decouple your systems.
00:01:00.979
That was a good introduction. I'd say one of the main goals of it is to decouple your services.
00:01:09.090
You often start with a monolith and then want to extract parts out of it. You end up with services, which you might call microservices. Most likely, they are not microservices because they are still really entangled with the main system, giving all those synchronous calls.
00:01:28.530
Imagine you have one application, and another user fills out a registration form. Then you have your main registration application, which communicates with the main application to get information about the user, and it also connects to the mail application to send the welcome email. You have all the control in one application; it's not decoupled.
00:01:50.850
If you could send messages, after a new user registration, you would simply pass that user's database ID to your email system, which would listen to this event. It would know, "Oh, a new user registration!" and get the user information.
00:02:10.470
The email system would then send out an email. You can even easily change the system; for instance, let's say you also want to push this data down to Salesforce. You just add a Salesforce listener that listens to the same event without changing any code on the other side.
00:02:30.739
That listener would add user information to Salesforce. So, that's a significant benefit. You can also use message buses to handle high volumes of messages.
00:02:50.920
Let's say this user registration happens after you run a TV commercial. When I worked for a gaming company, after a good spot, you might see 50,000 new registrations in just a few minutes. The system has to handle that load.
00:03:09.140
The user fills in a form, and the system saves that to a database, telling the user, "Great, you will receive a welcome email." That's fine if you have decoupled it. If your queue can handle all those messages while other systems need a bit more time to process them, to send out the emails or do whatever other tasks need to occur, your system will stay responsive.
00:03:27.380
It won't fail, and you won't lose any of those expensive registrations. You can also use it to broadcast messages—one event and just shout it out into the wild, adding listeners time after time.
00:03:44.000
You may run something like a news feed, which is currently only used by a few sites in Germany, but as you become more successful, other people will also want to use it. You don't have to do anything; you just shout out events, and you tell your clients, or consumers, to listen to those messages.
00:04:06.440
You can even use the same mechanism to replicate your data to another region in the world.
00:04:14.060
For example, you might have your RabbitMQ broker instance here in Germany and another one in Asia.
00:04:19.419
You can replicate the data there for faster access on both sides. So, I don't have time to go through all of that. I used this presentation before, and I have too much content on the slides.
00:04:31.030
On the left side, you see the publishers. On the right side, these are the consumers. In the center, we have basically RabbitMQ—the broker itself.
00:04:39.490
As you can see, the publishers publish to exchanges; they do not publish to queues. It always goes to an exchange, and the exchange routes the messages to a queue. I will explain how that happens.
00:04:54.400
Then, the queues connect to the consumers. In our case, queues will push messages to the consumers. This reminder leads me to explain 'ack'.
00:05:05.830
'Ack' is short for acknowledgment. Right now, we're just throwing all the messages down the pipeline.
00:05:12.520
At some point, you might want to have feedback: was this message received? Was it processed? Was there any issue? For that, we talk about acknowledgments.
00:05:25.690
You can have a positive acknowledgment, confirming receipt of a message, and you can also have a negative acknowledgment, which basically means to reject a message. Let’s look at some examples of how to handle that.
00:05:39.830
Starting with the left side, publishing messages: it's all about the trade-offs between speed and security. RabbitMQ in particular gives you lots of different options on how to handle sending messages and consuming messages.
00:05:55.060
What you do in between messages is all about trade-offs. Do you want to ensure that your message was received and handled, or do you not care so much? Because maybe you have locks, or you just want to push messages through the system as fast as possible.
00:06:08.200
If speed is more critical than reliability—if you don’t mind if a few messages are dropped—then you can go fast, knowing that there are no guarantees.
00:06:22.990
Messages are just sent from A to B, and you hope the consumer handles it. If a couple of messages drop, you don’t mind as much; it’s more important that the system doesn’t slow down.
00:06:33.220
However, there's an issue: you might overload the consumers and drop a lot of messages. In RabbitMQ, this is often handled through publisher confirmations.
00:06:45.430
The publisher confirmation happens on the left side of this diagram. When the broker is able to route a message from an exchange to a queue, and the message is sitting on that queue, a confirmation is sent to the publisher, signaling that all is well.
00:06:57.110
So, while the message is not consumed yet, the publisher knows it will be handled.
00:07:06.800
You can't really lose your messages this way; you can ensure that messages are usually routable in RabbitMQ. However, you need a lot of memory if your system is fast enough.
00:07:18.960
If the memory load is high, this means your queues should empty themselves really fast, which is fine. You can also persist them to disk if you’re afraid that your RabbitMQ instance could go down. It shouldn’t, but you know it can happen.
00:07:35.330
RabbitMQ doesn't use Redis or similar tools; it has a built-in database optimized for writing, while most other databases are optimized for speed.
00:07:46.620
That’s because it only writes a message to the database. It never reads it, and it deletes it when handling it. It may only read messages if something goes wrong.
00:08:00.060
You can create a cluster—you usually work with two types of nodes: read nodes and write nodes. It’s often used not for performance but for increased redundancy.
00:08:12.860
This means having one master that replicates data to other nodes. If the master goes down, replicas can take over.
00:08:23.820
All this configuration can happen automatically across the board.
00:08:30.600
We call these queues high availability queues, and that summarizes the fast left side.
00:08:36.720
Now let's move to the right side of consuming messages, keeping in mind the same trade-offs of speed and security.
00:08:46.250
When listening to messages, if it's too fast, we can overload the consumers, so that's something you may want to prevent.
00:08:55.070
In the fastest mode, there are no guarantees. There are no acknowledgments—nothing. Just give me all the messages you have, and I'll attempt to handle them.
00:09:14.190
You can set a quality of service level, which means you set a prefetch count—for example, ten messages, fifty or a hundred, whatever you choose.
00:09:32.350
After consuming that number of messages, the consumer would send an acknowledgment that it received those messages.
00:09:41.780
Both sides—the queue side (the broker) and the consumer—account for messages. The broker sends ID to confirm if messages are received.
00:09:58.200
If not all have been received, it will send only the messages that were not confirmed again so you never lose messages, and it's quite fast without too much overhead.
00:10:12.920
You can even be more granular. You can send acknowledgment after each message manually. An automatic acknowledgment would be sent when a message was received by the consumer.
00:10:28.220
The consumer can just send an acknowledgment, saying "I got it." The issue is that your consumer might crash or fail to process this message, and you would lose it.
00:10:46.580
So you usually want to send manual acknowledgment and do it after processing the message. This makes the whole process a bit slower, but it's a much safer way to do things.
00:10:57.990
This all depends on how important the information is that you're transferring.
00:11:12.920
When I'm talking about acknowledgment, I should mention non-acknowledgment or rejection. The consumer may try to handle a message that doesn't work; for instance, the data wasn’t formatted properly, or the external system it needed to communicate with is unavailable.
00:11:26.470
What happens when I reject a message? Theoretically, you could send the message all the way back to the publisher, but that's usually a bad idea. The publisher just gets a message that it sent like minutes ago or even seconds or an hour ago.
00:11:41.820
This creates excessive logic in the publisher. The consumer should handle it itself and determine how to manage messages that fail.
00:11:57.320
Typically, we reject messages and then provide instructions to another system about how to handle those rejected messages.
00:12:10.340
So, what are coherent nodes? I don’t have time for examples. We talked about the left and right sides: the publishers and the consumers. Now let’s come to the center: the broker.
00:12:29.470
The broker and exchanges route messages. To connect publishers and consumers, we have at least one exchange.
00:12:44.880
This exchange will bind queues. Each exchange will bind to at least one queue, and typically many of them.
00:12:58.080
The binding queues are handled in various ways, as I will explain. The broker routes messages from exchanges to queues.
00:13:09.140
There are different types of exchanges for different use cases. In a direct exchange, routing matches exactly with a binding key. If the string matches, then you receive this message.
00:13:27.160
With a fan-out exchange, every queue that binds to a fan-out exchange will receive all messages sent to that exchange. It's like broadcasting.
00:13:42.470
The most interesting one is the topic exchange, where routing keys use dot separators and queues find a binding key that matches the pattern.
00:13:59.560
For instance, you can match a word with asterisks, and hashtags can match a word followed by dots.
00:14:15.470
The whole string can emulate a direct exchange with the same string on both sides, or emulate a fan-out exchange with a hash symbol to receive all messages.
00:14:28.000
There are more types of exchanges, such as header exchanges that are part of message properties. It's basically a hash with key-value pairs you can use for routing.
00:14:39.250
RabbitMQ allows you to add plugins for more unusual types of exchanges you might need. You can also use exchanges to handle message failures.
00:14:56.060
We have alternate exchanges. Whenever routing doesn’t work, and a publisher sends a message to an exchange that cannot route it due to no queue defined with that key, it can happen.
00:15:11.320
In such cases, you can use alternate exchanges to handle those unrouteable messages.
00:15:25.130
For instance, the message was received by the exchange, but if it doesn't work, it can be sent to a different exchange, an alternate exchange, and you can monitor what happens with messages in that exchange.
00:15:39.000
For consumers, if a consumer receives a message but can't handle it, it rejects that message. We talked about what to do with rejected messages—it's a good way to deal with them using a dead letter exchange.
00:15:50.690
A dead letter exchange can have two exact same queues created from a normal exchange. It captures all the rejected messages. What happens next really depends on what you want to do.
00:16:06.890
You could implement a retry mechanism—maybe attempt to process them in an hour or a day—or you could handle them manually. You could go in, see what didn't work, and correct issues like broken source code.
00:16:28.840
Alternatively, you could have an automated process that routes those messages to an error logging or monitoring system.
00:16:39.720
So, you might end up deleting the queues if they didn’t work, but at least you collected the routing information.
00:16:57.030
Keeping in mind the previous examples of routing, I want to concentrate on traffic exchanges here.
00:17:09.280
Imagine we have an existing schema utilizing cities, topics, and categories for a service connecting tech communities around the world.
00:17:25.020
The schema could look like this: five cities, corresponding to topics and categories; this gives us 125 possibilities to create routing keys.
00:17:41.110
For example, Barcelona can serve as either an event or a job. If we have job postings, they could relate specifically to test jobs in Ruby.
00:17:58.100
These keys would start with ´Barcelona´ as I'm currently in Barcelona.
00:18:12.310
So, we can have various routing keys like 'Barcelona.event' or 'Barcelona.job' to capture particular events or job postings.
00:18:29.570
We can bind to queues with these keys, utilizing pattern matching. Thus, a binding key could be 'Barcelona.#', which captures all related events.
00:18:40.950
If we were to take it a step further, it might be like routing keys like 'registration.user.new' for registrations or updates.
00:18:57.000
That’s the central part: the exchange decides which queue receives which messages.
00:19:09.960
One message can go to multiple queues. It doesn’t need to go to just one. Every queue with a matching binding key will receive the message.
00:19:27.430
A queue can have multiple consumers, allowing a load balancing effect, ensuring tasks are handled more efficiently.
00:19:39.510
This load balancing is typically done in a round-robin manner. You can also have several consumers listening to different queues to receive more messages.
00:19:55.250
You have great flexibility in how you structure your system and how you extend it.
00:20:13.270
So, we’ve talked about publishers, the broker with exchanges and queues, and consumers, but we haven't discussed the individual messages.
00:20:32.980
In this example, the messages are indicated by arrows passing through. We have the method frame, not mentioned so far, which includes routing information like the routing key.
00:20:50.200
We also have a content header frame, which contains message properties. For instance, looking at the 'use by' column, we find information relevant for you, the user.
00:21:05.920
This isn’t used by RabbitMQ, allowing you the freedom to utilize it as you prefer. Important properties may include content type and content encoding.
00:21:23.730
The publisher should specify the content type—for example, application/json—and the encoding, such as UTF-8, which informs how the message should be read.
00:21:40.330
Usually, the consumer should deserialize the message properly. Instead of passing raw JSON to the application, you may want to handle it as a Ruby object or at least a hash.
00:21:56.960
The message also encompasses optional fields such as message ID, which you can create yourself, and timestamps.
00:22:15.300
You can define an expiration time, where the message would automatically be deleted or rejected if it's too old.
00:22:33.950
You can set a delivery mode, correlation ID, and priority properties. We haven’t discussed the payload part yet.
00:22:49.500
The payload is part of the body frame, which constitutes what we're sending—our actual message. This may be as simple as a user ID or a more complex object that depends on your system's setup.
00:23:06.760
It may contain more information if you plan to migrate to a different system later on, as other systems may not possess all the meta-information RabbitMQ offers.
00:23:22.469
The important thing to keep in mind is that the payload is the body of the actual message.
00:23:37.340
I realize I may have skipped some content; I have some coding examples which are written in Ruby and utilize the Bunny gem.
00:23:50.540
This is the most popular solution you'll find nowadays, so please bear with me. In the coding example, you can see my Ruby RabbitMQ system.
00:24:06.020
I initiate my Bunny object, establishing a new connection and creating a channel for the connection.
00:24:24.210
This process occurs once for the publisher and once for each consumer. Keep in mind, the consumer and publisher can be part of the same application.
00:24:41.450
You might encounter a monolith, but you still need background processing in between. So, you can publish a message and consume it later.
00:24:57.040
This setup is the same for declaring an exchange. The queues are initially connected, allowing us to bind them together.
00:25:12.700
In the last line, we see that the queues are created and the exchange bind themselves, utilizing the binding key.
00:25:31.360
In this case, I'm not extensively using pattern matching, but if you do utilize a topic exchange, you can also publish messages.
00:25:49.610
I’ll publish messages with the relevant information, including the message ID and any data properties that come into play.
00:26:04.440
This message publishing acts as a pairing between the sender and messages being correctly routed.
00:26:24.950
Now, when consuming messages, you'll create another channel connected to the same queues to subscribe consumers.
00:26:40.380
The consumer is defined by which channel and queue it belongs to. The consumer tag will be an ID generated by RabbitMQ, or you can set your own IDs.
00:26:56.930
There are various options to set here, including the automation of acknowledgment procedures.
00:27:09.970
After subscribing to the queue, you're creating a callback for message delivery, securing that you receive the proper properties and payload.
00:27:22.870
This example indicates that various methods of acquiring and managing messages are highlighted. The payload can have various other properties.
00:27:35.830
Lastly, you can cancel subscribing when needed—maybe during deployments or for maintenance.
00:27:49.420
When you take the consumer offline, all messages will then wait in the queue for you.
00:28:00.970
One system integrates all messages while keeping redundancy and allowing you to monitor the state of your queues.
00:28:13.300
In an ideal world, your queue lengths would always be zero, indicating that all messages arrive and get consumed immediately.
00:28:31.120
You may also want to monitor different processes, such as long-running jobs that may take longer than expected to finish.
00:28:50.230
Setting some thresholds is ideal. If the queues are filling up significantly, you might consider investigating the underlying processes.
00:29:02.610
When it comes to clusters, I won't delve into all setup practices. Some best practices will be marked, and I’ll send you a link after this.
00:29:11.990
It's worth examining the software that makes using RabbitMQ easier, as well as client libraries.
00:29:28.170
At this Ruby conference, I, of course, have to mention the Bunny gem. But you don't have to rely solely on the gem; frameworks also exist.
00:29:47.180
The variety of client applications in different programming languages increases, so as highlighted, RabbitMQ has been stable for over ten years.
00:30:00.680
Coming from banking and finance, it’s well-managed and documented. The top link is a great reference to the documentation.
00:30:15.580
Download it; you can try it easily. Thank you very much for your time, and I hope you enjoyed learning about RabbitMQ. You earned yourself a piece of cake and coffee. I’ll see you back in this room in 45 minutes; next up will be Anna talking.