Getting started with ZeroMQ

LoneStarRuby Conf 2011

Ross Andrews

1 talk

Getting started with ZeroMQ

by Ross Andrews

In the video titled 'Getting Started with ZeroMQ', Ross Andrews provides an introduction to the messaging library ZeroMQ and its application in architecting distributed applications, particularly for Ruby developers. He addresses common misconceptions surrounding ZeroMQ, emphasizing its role as a messaging bus that operates without broker servers, differentiating it from traditional enterprise messaging solutions. The talk underscores key benefits of ZeroMQ, including scalability, reliability, and simplicity in handling message communications.

Key Points Discussed:
- Definition and Purpose of ZeroMQ:

- ZeroMQ is aimed at replacing TCP, functioning like sockets but designed for programmer efficiency. It's not intended to replace fully-fledged enterprise solutions like AMQ.

- It ensures message order, completeness, and correctness, with capabilities to recover from minor network issues.

Scalability Considerations:
- ZeroMQ's architecture allows developers to address both performance and complexity scaling, accommodating growing user bases and feature sets without compromising code clarity.
- The actor model is introduced, promoting self-contained components that communicate via messages, simplifying code structure.
Architecture Design Recommendations:
- Planning an application’s architecture on a whiteboard is advisable, detailing how each actor communicates using ZeroMQ for effective organization and visualization.
Misconceptions about ZeroMQ:
- Clarifies that ZeroMQ can't be directly used for HTTP servers but can interface with them by translating requests into ZeroMQ messages.
Installation and Compatibility:
- Installation of ZeroMQ via Ruby gems and C library is discussed, with a note on compatibility issues, particularly on Windows.
Demonstration of Socket Types:
- Demonstrates the request-reply pattern using a simple client-server interaction, showcasing ZeroMQ's blocking behavior and resilience.
- The publish-subscribe pattern is illustrated with an example generating random temperature data for specific ZIP codes, highlighting statelessness and message filtering capabilities.
- Emphasizes the issues of message drops in publish-subscribe models and suggests caching strategies to mitigate data loss.
Advanced Architectural Strategies:
- Discusses combining the Push-Pull pattern for workloads and load balancing, comparing this with other messaging solutions like Redis.

Conclusions and Takeaways:

- ZeroMQ can enhance messaging performance in distributed systems, especially in asynchronous architectures.

- Developers are encouraged to consider ZeroMQ for scalable and resilient communication, leading to improved system design and better decoupling of services.

- Continuous education on messaging systems is vital for adapting to evolving software requirements.

- The session concludes with an invitation for questions and discussion on ZeroMQ's integration into future projects.

00:00:20.320 I'm Ross Andrews, and this talk is a quick introduction to ZeroMQ.

00:00:32.559 So first off, what is ZeroMQ? There are a lot of misconceptions about it. Not many people know it's used in Mongrel too; it got really popular really fast because of Zed Shaw using it there. But not a lot of people actually know about it. It's a messaging bus, which right away sounds like a weird enterprise thing; we're all Ruby developers, and we don't really like enterprise stuff.

00:00:51.680 Right away, people start to distrust things like that. Really, it's an enterprise idea that has been made very Rails-like and hacker-friendly. It has zero broker servers, so you don't need to set up any special machines to run ZeroMQ other than the ones that are talking to each other. You don't need to set up anything else.

00:01:16.479 The way I like to put it is this: In the SQLite manual, it says SQLite is not supposed to replace Oracle; it's meant to replace fopen. ZeroMQ is not supposed to replace AMQ or any other enterprise messaging bus; it's supposed to replace TCP. It is essentially sockets, but it works in the way programmers think sockets should work. It offers a more stable and reliable way to send messages back and forth between your machines.

00:01:53.200 With ZeroMQ, you get things like ensuring that a message is going to arrive in the order it was sent, that it will arrive complete and correct, and that it won't be corrupted in flight. Within reason, you can actually ensure that it's going to arrive. Obviously, if the network goes down for a really long time, then things aren't going to happen, but it can recover from minor network blips and issues.

00:02:32.160 The next question is, okay, I already have a TCP library; it's called TCP. What does ZeroMQ give me? Why do I want to use it? The answer is scalability. Rails developers are super obsessed with scalability, but there's another kind of scalability we often forget.

00:03:02.800 When we talk about scalability, we're usually talking about performance. Scalability means serving more requests faster. So, your app uses less memory, can handle more users, or whatever. That's one kind of scalability, but the reality is that you're hardly ever just growing in performance requirements. You hire more people, you add more features, and you have more special cases in your data. Therefore, you also need to deal with complexity scaling.

00:03:36.799 ZeroMQ handles that kind of scalability as your application code gets larger. ZeroMQ gives you a really sharp knife to cut your code apart and make it easier to follow and understand. It allows you to handle not just your user base getting bigger and your traffic getting bigger, but also your code getting more complex. This is kind of why we use Ruby to begin with; it handles large codebases better than Java. In fact, with Java, your codebase gets more complicated faster than it would with Ruby.

00:04:31.040 With a normal Rails architecture, your codebase gets more complicated faster than it does with a ZeroMQ-style architecture. So, if you're going to use ZeroMQ and get the most out of it, you might want to start thinking in terms of a certain model called the actor model, which is similar to service-oriented architecture.

00:04:57.280 The idea behind the actor model is to have little tiny self-contained pieces that only pass messages around. They have no state, so if one of them dies, you just spawn another one. If you have more traffic, you spawn more instances. You're supposed to use access to everything and use ZeroMQ to facilitate the communication between these actors.

00:05:38.720 In places where you typically have background jobs in Rails, you can break those off into their own processes running independently. Instead of adding a task to a database table and having a background job process it after a certain interval, you send a message through ZeroMQ. That background job can respond immediately, handling its process whenever necessary.

00:06:07.520 The model of small pieces loosely joined is how you should be writing your code. This approach is also applicable on a larger scale, facilitating cleaner and more encapsulated processes. ZeroMQ provides a set of tools and patterns to help you visualize and organize the architecture of your application, promoting this loose coupling.

00:06:42.560 To design your application with ZeroMQ successfully, gather everyone in a room with a giant whiteboard and map out the entire architecture, detailing how all the actors will communicate and what messages they will send. If done correctly, each component becomes easy to mock up.

00:07:00.720 Now, I want to spend a couple of minutes discussing what ZeroMQ is not. A common misconception is that people want to write HTTP servers with ZeroMQ. The truth is, you can't. While ZeroMQ is opinionated software in one way, it is also very flexible in another. It doesn't care what type of data you send in the messages, but it is strict about networking—ZeroMQ sockets can only talk to other ZeroMQ sockets.

00:07:36.880 So you can't have a ZeroMQ socket listening on port 80 for HTTP requests. Instead, you can implement a service that listens for HTTP requests on port 80 and translates those requests into ZeroMQ messages before sending them on a different socket. This is exactly how Mongrel 2 works. An actor receives messages from the outside world, translates them, and forwards them to the relevant parts of your application.

00:08:17.920 Let's move on to the rest of the talk. First, I will cover how to install ZeroMQ, which has a Ruby gem available. ZeroMQ itself is a C library that comes with bindings for numerous languages, meaning virtually every language you might need has a ZeroMQ library. The API is consistent across languages and consists of only about six functions.

00:09:17.520 If you're on a Mac, you can simply use Homebrew to install the C library by running 'brew install zmq' and then install the Ruby gem. The only exception to this is on Windows, where installation might be more complicated, as Windows does not fully support the way Unix-based systems work, leading to some features being unavailable.

00:09:50.360 While important features do work across transport layers, you should prefer using TCP for most scenarios, as it's quite fast and won't noticeably impact performance. Note that in-process communication (IPC) behaves a bit differently on Windows. In fact, IPC doesn't work without additional configurations due to the differences in models.

00:10:20.480 Now, let's start with the first program demonstration. ZeroMQ has different types of sockets that represent typical network architectures. The most common one is the request-reply pattern: you send a request and receive one reply back. This pattern resembles how HTTP works, and ZeroMQ enforces this structure.

00:10:54.640 In this case, the client sends request packets, making requests and receiving responses. I must mention that this demo will not be in Ruby, as I couldn’t get the ZeroMQ gem to build on my laptop. Instead, I will demonstrate using a different language.

00:11:21.120 The first step is to load the ZeroMQ library with 'zmq.init', create a new request socket, and connect it to localhost on port 4568. The client will then send a 'ping' message followed by a number ten times, pausing for one second in between each send. The server side starts the same way; it initializes ZeroMQ, opens a socket, binds it to the port, receives messages, and sends responses back.

00:12:01.360 If I now start the server and then start the client, you will see it works as expected. The client sends messages, and the server processes them, printing out replies. An interesting aspect to note is that if the server is stopped and the client runs, the connection will block until the server is available again. Unlike typical TCP connections, ZeroMQ doesn’t crash; it waits for a response.

00:13:37.440 One crucial point to note is that if the server dies while sending a request, the same blocking behavior occurs. This state allows for resilience in handling connection issues, where the client only requires a response before proceeding.

00:14:45.679 So, this reliability of ZeroMQ means that if one half of the connection dies, the other half remains operational, continuing to block until it successfully receives a message.

00:15:10.560 The next example involves the publish-subscribe socket, another common pattern where one entity pushes data, and multiple subscribers receive it. It’s like Twitter or RSS. The publisher doesn't care who receives the data, while subscribers can filter which messages they are interested in.

00:15:48.720 In this demonstration, we set up a publisher that generates random ZIP codes and temperatures. The subscriber will only receive messages specifically for the 78701 Austin ZIP code. So, we create a ZeroMQ context, open a socket, and set the subscription criteria to filter messages based on specified bytes.

00:16:31.600 The publisher can generate random zip codes and temperatures, and then publish these strings, which the subscriber processes upon receipt. This example illustrates the effectiveness of ZeroMQ in handling message flows, allowing for significant performance while maintaining light overhead.

00:17:35.680 When we run them, we're getting messages back that align with expected output based on the filter for 78701. Nan’s about one out of every hundred thousand messages is processed, meaning that ZeroMQ manages to provide quite a robust throughput, showcasing its performance capabilities.

00:18:07.440 If the publisher shuts down temporarily, the subscriber simply stops receiving messages until it resumes—further demonstrating the resilience and durability of this pattern. The efficient data delivery mechanism of ZeroMQ enables seamless message exchanges while minimizing resource load.

00:18:56.080 Another important point about publish-subscribe sockets is that they are stateless, which makes them more durable in comparison to request-reply sockets. With publish-subscribe, sockets do not depend on maintaining state to send or receive messages.

00:19:32.960 The question regarding drop messages came up: subscribers may miss messages if they lose connection while it’s offline. However, if you implement certain strategies, such as caching messages until the subscriber reconnects, you can effectively avoid losing important data.

00:20:01.520 When considering a broader application architecture and how to leverage ZeroMQ within it, a common modern approach is to employ the Push-Pull pattern in conjunction with load balancing. This configuration allows you to distribute workloads across multiple subscribers, enhancing throughput and efficiency.

00:20:41.040 In summary, depending on your business needs and use cases, transitioning towards ZeroMQ for messaging could provide a robust alternative to traditional models like Redis. ZeroMQ shines particularly in cross-language, cross-platform contexts where seamless integration becomes crucial.

00:21:08.320 Along with providing varied socket types like Push, Pull, and Pair, ZeroMQ equips developers with tools optimized for communications in distributed systems. The flexibility in its design means that it can adapt to unique architectural needs, aligning with advanced methodologies like CQRS and event sourcing.

00:21:50.400 In conclusion, if you have two units that need to communicate, definitely consider using ZeroMQ. If you have one unit with two tasks, breaking it into components while using ZeroMQ can enhance scalability and reduce complexity, offering a smooth pathway to building high-performing asynchronous systems.

00:22:37.840 So, are there any questions?

00:23:02.400 It's important that when messages are sent over a socket, they are always received in the order they were sent, which is a key advantage over straight TCP.

00:23:50.960 In conclusion, remember that while ZeroMQ can enhance the performance of message brokering in a distributed system, it doesn't handle message persistence out of the box. You will need to implement recovery mechanisms to ensure that critical messages aren’t dropped.

00:24:29.320 Further discussions revolved around comparing ZeroMQ with other libraries like Redis, emphasizing the significance of understanding specific use cases and variations depending on context.

00:25:18.320 Many attendees expressed keen interest in learning how ZeroMQ could practically fit into their asynchronous and event-driven architectures, highlighting the diverse applicability rather than being limited by programming language boundaries.

00:26:12.800 Through engaging dialogue, participants were able to explore how hooking into ZeroMQ might enhance application scalability, debugging processes, and foster better code maintenance practices.

00:26:40.400 As the conversation continued, several insights emerged around the ongoing challenges developers face when integrating multiple technologies and managing message queues effectively.

00:27:15.920 The critical balance in messaging architecture is to provide adequate responsiveness while handling varying loads, and ZeroMQ appears to meet those requirements without overwhelming the system.

00:28:00.720 Thus, adopting ZeroMQ can be a forward-thinking strategy that could lead to more decoupled services and an overarching design pattern based on message-oriented middleware.

00:28:50.320 Reflecting on the dynamic nature of modern software architectures, notably in multi-threaded applications, the audience saw compelling advantages ZeroMQ introduces, potentially revitalizing existing systems.

00:29:38.880 To conclude the session, an emphasis was placed on ongoing education about messaging systems while staying open to adopting newer approaches that satisfy evolving software demands.

00:30:25.520 Thank you all for your time. I'm looking forward to hearing your feedback and continuing our conversation over the integration of ZeroMQ in our upcoming projects.

LoneStarRuby Conf 2011