00:00:20.320
I'm Ross Andrews, and this talk is a quick introduction to ZeroMQ.
00:00:32.559
So first off, what is ZeroMQ? There are a lot of misconceptions about it. Not many people know it's used in Mongrel too; it got really popular really fast because of Zed Shaw using it there. But not a lot of people actually know about it. It's a messaging bus, which right away sounds like a weird enterprise thing; we're all Ruby developers, and we don't really like enterprise stuff.
00:00:51.680
Right away, people start to distrust things like that. Really, it's an enterprise idea that has been made very Rails-like and hacker-friendly. It has zero broker servers, so you don't need to set up any special machines to run ZeroMQ other than the ones that are talking to each other. You don't need to set up anything else.
00:01:16.479
The way I like to put it is this: In the SQLite manual, it says SQLite is not supposed to replace Oracle; it's meant to replace fopen. ZeroMQ is not supposed to replace AMQ or any other enterprise messaging bus; it's supposed to replace TCP. It is essentially sockets, but it works in the way programmers think sockets should work. It offers a more stable and reliable way to send messages back and forth between your machines.
00:01:53.200
With ZeroMQ, you get things like ensuring that a message is going to arrive in the order it was sent, that it will arrive complete and correct, and that it won't be corrupted in flight. Within reason, you can actually ensure that it's going to arrive. Obviously, if the network goes down for a really long time, then things aren't going to happen, but it can recover from minor network blips and issues.
00:02:32.160
The next question is, okay, I already have a TCP library; it's called TCP. What does ZeroMQ give me? Why do I want to use it? The answer is scalability. Rails developers are super obsessed with scalability, but there's another kind of scalability we often forget.
00:03:02.800
When we talk about scalability, we're usually talking about performance. Scalability means serving more requests faster. So, your app uses less memory, can handle more users, or whatever. That's one kind of scalability, but the reality is that you're hardly ever just growing in performance requirements. You hire more people, you add more features, and you have more special cases in your data. Therefore, you also need to deal with complexity scaling.
00:03:36.799
ZeroMQ handles that kind of scalability as your application code gets larger. ZeroMQ gives you a really sharp knife to cut your code apart and make it easier to follow and understand. It allows you to handle not just your user base getting bigger and your traffic getting bigger, but also your code getting more complex. This is kind of why we use Ruby to begin with; it handles large codebases better than Java. In fact, with Java, your codebase gets more complicated faster than it would with Ruby.
00:04:31.040
With a normal Rails architecture, your codebase gets more complicated faster than it does with a ZeroMQ-style architecture. So, if you're going to use ZeroMQ and get the most out of it, you might want to start thinking in terms of a certain model called the actor model, which is similar to service-oriented architecture.
00:04:57.280
The idea behind the actor model is to have little tiny self-contained pieces that only pass messages around. They have no state, so if one of them dies, you just spawn another one. If you have more traffic, you spawn more instances. You're supposed to use access to everything and use ZeroMQ to facilitate the communication between these actors.
00:05:38.720
In places where you typically have background jobs in Rails, you can break those off into their own processes running independently. Instead of adding a task to a database table and having a background job process it after a certain interval, you send a message through ZeroMQ. That background job can respond immediately, handling its process whenever necessary.
00:06:07.520
The model of small pieces loosely joined is how you should be writing your code. This approach is also applicable on a larger scale, facilitating cleaner and more encapsulated processes. ZeroMQ provides a set of tools and patterns to help you visualize and organize the architecture of your application, promoting this loose coupling.
00:06:42.560
To design your application with ZeroMQ successfully, gather everyone in a room with a giant whiteboard and map out the entire architecture, detailing how all the actors will communicate and what messages they will send. If done correctly, each component becomes easy to mock up.
00:07:00.720
Now, I want to spend a couple of minutes discussing what ZeroMQ is not. A common misconception is that people want to write HTTP servers with ZeroMQ. The truth is, you can't. While ZeroMQ is opinionated software in one way, it is also very flexible in another. It doesn't care what type of data you send in the messages, but it is strict about networking—ZeroMQ sockets can only talk to other ZeroMQ sockets.
00:07:36.880
So you can't have a ZeroMQ socket listening on port 80 for HTTP requests. Instead, you can implement a service that listens for HTTP requests on port 80 and translates those requests into ZeroMQ messages before sending them on a different socket. This is exactly how Mongrel 2 works. An actor receives messages from the outside world, translates them, and forwards them to the relevant parts of your application.
00:08:17.920
Let's move on to the rest of the talk. First, I will cover how to install ZeroMQ, which has a Ruby gem available. ZeroMQ itself is a C library that comes with bindings for numerous languages, meaning virtually every language you might need has a ZeroMQ library. The API is consistent across languages and consists of only about six functions.
00:09:17.520
If you're on a Mac, you can simply use Homebrew to install the C library by running 'brew install zmq' and then install the Ruby gem. The only exception to this is on Windows, where installation might be more complicated, as Windows does not fully support the way Unix-based systems work, leading to some features being unavailable.
00:09:50.360
While important features do work across transport layers, you should prefer using TCP for most scenarios, as it's quite fast and won't noticeably impact performance. Note that in-process communication (IPC) behaves a bit differently on Windows. In fact, IPC doesn't work without additional configurations due to the differences in models.
00:10:20.480
Now, let's start with the first program demonstration. ZeroMQ has different types of sockets that represent typical network architectures. The most common one is the request-reply pattern: you send a request and receive one reply back. This pattern resembles how HTTP works, and ZeroMQ enforces this structure.
00:10:54.640
In this case, the client sends request packets, making requests and receiving responses. I must mention that this demo will not be in Ruby, as I couldn’t get the ZeroMQ gem to build on my laptop. Instead, I will demonstrate using a different language.
00:11:21.120
The first step is to load the ZeroMQ library with 'zmq.init', create a new request socket, and connect it to localhost on port 4568. The client will then send a 'ping' message followed by a number ten times, pausing for one second in between each send. The server side starts the same way; it initializes ZeroMQ, opens a socket, binds it to the port, receives messages, and sends responses back.
00:12:01.360
If I now start the server and then start the client, you will see it works as expected. The client sends messages, and the server processes them, printing out replies. An interesting aspect to note is that if the server is stopped and the client runs, the connection will block until the server is available again. Unlike typical TCP connections, ZeroMQ doesn’t crash; it waits for a response.
00:13:37.440
One crucial point to note is that if the server dies while sending a request, the same blocking behavior occurs. This state allows for resilience in handling connection issues, where the client only requires a response before proceeding.
00:14:45.679
So, this reliability of ZeroMQ means that if one half of the connection dies, the other half remains operational, continuing to block until it successfully receives a message.
00:15:10.560
The next example involves the publish-subscribe socket, another common pattern where one entity pushes data, and multiple subscribers receive it. It’s like Twitter or RSS. The publisher doesn't care who receives the data, while subscribers can filter which messages they are interested in.
00:15:48.720
In this demonstration, we set up a publisher that generates random ZIP codes and temperatures. The subscriber will only receive messages specifically for the 78701 Austin ZIP code. So, we create a ZeroMQ context, open a socket, and set the subscription criteria to filter messages based on specified bytes.
00:16:31.600
The publisher can generate random zip codes and temperatures, and then publish these strings, which the subscriber processes upon receipt. This example illustrates the effectiveness of ZeroMQ in handling message flows, allowing for significant performance while maintaining light overhead.
00:17:35.680
When we run them, we're getting messages back that align with expected output based on the filter for 78701. Nan’s about one out of every hundred thousand messages is processed, meaning that ZeroMQ manages to provide quite a robust throughput, showcasing its performance capabilities.
00:18:07.440
If the publisher shuts down temporarily, the subscriber simply stops receiving messages until it resumes—further demonstrating the resilience and durability of this pattern. The efficient data delivery mechanism of ZeroMQ enables seamless message exchanges while minimizing resource load.
00:18:56.080
Another important point about publish-subscribe sockets is that they are stateless, which makes them more durable in comparison to request-reply sockets. With publish-subscribe, sockets do not depend on maintaining state to send or receive messages.
00:19:32.960
The question regarding drop messages came up: subscribers may miss messages if they lose connection while it’s offline. However, if you implement certain strategies, such as caching messages until the subscriber reconnects, you can effectively avoid losing important data.
00:20:01.520
When considering a broader application architecture and how to leverage ZeroMQ within it, a common modern approach is to employ the Push-Pull pattern in conjunction with load balancing. This configuration allows you to distribute workloads across multiple subscribers, enhancing throughput and efficiency.
00:20:41.040
In summary, depending on your business needs and use cases, transitioning towards ZeroMQ for messaging could provide a robust alternative to traditional models like Redis. ZeroMQ shines particularly in cross-language, cross-platform contexts where seamless integration becomes crucial.
00:21:08.320
Along with providing varied socket types like Push, Pull, and Pair, ZeroMQ equips developers with tools optimized for communications in distributed systems. The flexibility in its design means that it can adapt to unique architectural needs, aligning with advanced methodologies like CQRS and event sourcing.
00:21:50.400
In conclusion, if you have two units that need to communicate, definitely consider using ZeroMQ. If you have one unit with two tasks, breaking it into components while using ZeroMQ can enhance scalability and reduce complexity, offering a smooth pathway to building high-performing asynchronous systems.
00:22:37.840
So, are there any questions?
00:23:02.400
It's important that when messages are sent over a socket, they are always received in the order they were sent, which is a key advantage over straight TCP.
00:23:50.960
In conclusion, remember that while ZeroMQ can enhance the performance of message brokering in a distributed system, it doesn't handle message persistence out of the box. You will need to implement recovery mechanisms to ensure that critical messages aren’t dropped.
00:24:29.320
Further discussions revolved around comparing ZeroMQ with other libraries like Redis, emphasizing the significance of understanding specific use cases and variations depending on context.
00:25:18.320
Many attendees expressed keen interest in learning how ZeroMQ could practically fit into their asynchronous and event-driven architectures, highlighting the diverse applicability rather than being limited by programming language boundaries.
00:26:12.800
Through engaging dialogue, participants were able to explore how hooking into ZeroMQ might enhance application scalability, debugging processes, and foster better code maintenance practices.
00:26:40.400
As the conversation continued, several insights emerged around the ongoing challenges developers face when integrating multiple technologies and managing message queues effectively.
00:27:15.920
The critical balance in messaging architecture is to provide adequate responsiveness while handling varying loads, and ZeroMQ appears to meet those requirements without overwhelming the system.
00:28:00.720
Thus, adopting ZeroMQ can be a forward-thinking strategy that could lead to more decoupled services and an overarching design pattern based on message-oriented middleware.
00:28:50.320
Reflecting on the dynamic nature of modern software architectures, notably in multi-threaded applications, the audience saw compelling advantages ZeroMQ introduces, potentially revitalizing existing systems.
00:29:38.880
To conclude the session, an emphasis was placed on ongoing education about messaging systems while staying open to adopting newer approaches that satisfy evolving software demands.
00:30:25.520
Thank you all for your time. I'm looking forward to hearing your feedback and continuing our conversation over the integration of ZeroMQ in our upcoming projects.
