Distributed Patterns in Ruby

by Eric Redmond

In the video "Distributed Patterns in Ruby" presented by Eric Redmond at Ancient City Ruby 2013, the focus is on the principles of scalability in distributed systems, emphasizing data distribution and message passing. Redmond explains that scalability is not determined by programming languages or web architectures but by how effectively distributed systems manage data and messages. The key concepts discussed include:

Definition of Distributed Systems: A distributed system requires autonomous nodes that communicate, unlike a mere array of servers sharing information.
Scalability: Distributed systems allow for horizontal scaling, increasing availability and reliability even if individual servers fail.
CAP Theorem: This theorem states that in a distributed system, one can only achieve two of the following three goals: consistency, availability, and partition tolerance.
Distributed Patterns: Key patterns introduced include distributed hash tables, messaging patterns, vector clocks, Merkle trees, and MapReduce, all of which are easily implemented in Ruby and underpin many distributed systems.

The presentation features several practical examples and coding demonstrations to illustrate distributed patterns:

Distributed Hash Table (DHT): Redmond describes a consistent hashing approach to distribute data evenly across nodes, minimizing disruption when nodes are added or removed.
Messaging Patterns: The focus is on the request-reply pattern using ZeroMQ to enable efficient communication between nodes. Redmond explains the creation of a robust messaging network for querying nodes.
Replication and Availability: Discussion on how to implement replication strategies to ensure data integrity and availability, especially during node failures.
Conflict Resolution: Vector clocks are introduced to manage concurrent updates while maintaining data consistency across nodes.
Merkle Trees: These are used for structured validation and accessing distributed data effectively as the system scales.
MapReduce Framework: This allows for efficient processing of large datasets through parallel computing and is critical for complex queries in distributed databases.

The main conclusion emphasizes that understanding distributed patterns and effectively leveraging Ruby's capabilities can lead to the development of robust distributed systems. The session wraps up with a focus on designing systems that address challenges in reliability, availability, and performance while utilizing best practices in Ruby for scalable architecture.

00:00:00.030 All right, so I'm from Portland, the land of pasty skin and moisture. I would first love to appreciate you Floridians for bringing in the clouds and the rain. I feel like it was for me, and I'm very comfortable, so thanks!

00:00:07.259 What I'm going to talk about today is distributed patterns in Ruby. I feel obliged to mention again that I work at Basho. I also have a couple of books out: 'Databases in Seven Weeks' and, more recently, a little react book that's free to download from my GitHub account. I want to post all of these online, so you don't need to feverishly write anything down.

00:00:20.550 My GitHub account is github.com/ericredmond/little-react-book. I tell people that if you buy the book and don't like it, I won't offer a refund. But what you can do is buy five more copies and burn them; that will teach me a valuable lesson.

00:00:40.140 Now, there is this artist, Richard Avedon, who I'm kind of a fan of. He attempted to create a sketch with his left hand, and this is what he ended up with. You can tell that he had a hard time holding the pen, as his hand isn't strong in that way. This isn't anything he's practiced, but you can tell this is not the work of an amateur.

00:01:04.680 You can tell the character is female; the face isn't bad. The girl on the far side has the classic anime sweat drop. He clearly knows what he's doing. The skill isn't in his hand; it's in his head. He knows what to look for, even though he is not adept. The reason I wanted to point that out is that you can glue together toolkits, but it's not going to fix an ignorance problem.

00:01:25.170 But there's a flip side to that. Just like with the artist, if you know what you're doing, you don't necessarily need toolkits to stand on; you can do a pretty good job on your own. You should understand what it is and then pick the tool to solve the problem. You shouldn't just pick tools and copy what others have done, assuming your system will resemble Google just because you copied their approach.

00:01:58.110 What I'm going to talk about is distributed systems in particular. This is one of my favorite definitions of a distributed system. I like this definition because it specifically notes the emphasized words, as they're all important. If you take any of these bolded terms away, you don't really have a distributed system anymore.

00:02:34.079 Rails users should take special note of this definition because if you have a bunch of autonomous nodes that are servers, and they're all taking requests but not communicating with each other, you don't really have a distributed system; you just have an array of servers sharing information.

00:03:06.799 Distributed systems are great. I hope we all understand the reasoning why you'd want a distributed system. It's a great way of scaling horizontally. There's only so much you can scale vertically, and having one server grow bigger has limits.

00:03:25.439 It's also really great for availability. If one of those individual servers goes down, your entire system can still be up and running. The more servers you add, the likelihood of any single server going down increases exponentially. For instance, if you have a 0.01 chance of any server going down and you have a thousand of them, the chance of at least one going down is about 60%. Something is going to go down.

00:04:03.900 However, there is a dark side to this: the CAP theorem, which is generally understood as a database problem but is actually just a problem with distributed systems in general. Who here is familiar with the CAP theorem, by the way? Okay, I'm going to explain it.

00:04:30.630 So, what the CAP theorem says is that if you have a distributed system, you must be partition tolerant; partition tolerance is a necessary condition. You can only have two of these three properties: consistency, availability, and partition tolerance. A distributed system must be partition tolerant, but you can either have it be consistent, where the entire system returns the same answer to any request, or it can be available, which means that every time you request something from the system, you get a result.

00:05:03.240 Let’s take these three people in the front row as an example. Let's say that you guys are nodes in a distributed system. As a client, I come to you and say the word of the day is 'jump.' Now you all know the word of the day is jump. Then you get up to go to the restroom, and while you're gone, I change the word of the day to 'leap.' Now, if someone stops you in the hall and asks, 'what's the word of the day?' you have a choice.

00:05:43.740 You can either remain available and say, 'the last thing I knew, the answer was jump,' not knowing I changed it to leap. Now you create an inconsistent system because if I ask you versus the others, I get different answers. Alternatively, you can choose not to answer at all and say, 'I don’t know,' which makes you unavailable, but you haven't destroyed the integrity of the system in terms of consistency, as you're not giving me a different answer.

00:06:26.600 So that’s the CAP theorem in a nutshell: when it’s distributed, you can either choose consistency or availability, but you can’t have both. It’s a physics problem. Just as a side note, I’m telling you with some authority that anybody who claims they’ve beaten the CAP theorem, or that it doesn't matter, likely doesn't understand the implications of what they’re saying.

00:06:43.990 Now we're going to talk about distributed patterns. These are some of the distributed patterns we’re going to cover: distributed hash table, message patterns, vector clocks, Merkle trees, and MapReduce. I picked these for a couple of reasons.

00:06:51.960 First, they're really easy to implement in Ruby, and we're going to write some examples now. Second, many of these distributed patterns underlie various distributed systems. For instance, the company Basho, where I work, makes the Riak database, which utilizes all of these patterns. And we are not the only ones. Cassandra does too; nearly all NoSQL databases implement some form of MapReduce.

00:07:39.080 Let's start with the distributed hash table, also called consistent hashing, which distributes data evenly. The most crucial part is that, unlike a classic hash table that you're all familiar with, there is minimal disruption when nodes are added or removed.

00:08:02.520 I made a naive hash implementation where I store the number of nodes and give a block of buckets in my array, sending requests to those nodes. My hash uses a SHA-1 algorithm to determine where the data is stored. The important part is how I define my array. For instance, if I have four nodes and want each one to handle a thousand values, I would create an array of four thousand.

00:08:52.200 Here’s the problem with standard hash implementation. Say I start with three nodes. The number I get has to be much larger than the range of the array if you’re using something like SHA. So, to fit a number within the right bucket, I perform a modulo operation on it. For instance, if the number is converted to ‘4’ and I do modulo ‘3,’ it puts it in bucket position one. This is great initially, but if I add a new node, all the data in the original location may now need to be rehashed.

00:09:57.900 For example, if I add node D, the same object that previously fell into '4' might now fall into '0,' moving data between nodes. This is an expensive operation in terms of resources. So, consistent hashing addresses this by placing nodes on a circular structure and minimizing disruption when nodes join or leave.

00:10:42.790 Using a consistent hash, when I remove a node 'C' and add 'D,' only the values stale are in the range specifically between ‘C’ and ‘D.’ This is a simple consistent hashing implementation. While I keep the nodes tracked in a hash, what matters is minimizing the thrashing when adding and removing nodes from our network.

00:11:29.600 Now, I want to balance the ring so that keys are evenly distributed. When adding nodes, it may lead to uneven distribution of keys. A balanced approach assigns ranges of keys to each node, allowing for smooth operations during scaling while minimizing the chance of redundancy.

00:12:12.500 Later on, I tripled the nodes and ran the consistency hash on my dataset; instead of a 90% miss rate like my naive implementation, it dropped to 7%. We were clearly on the right track. I should point out that adding more nodes can still result in imbalances, but adjustments can be made through practice.

00:12:50.940 With the proper balance in mind, we introduced the preference list concept to handle availability. The preference list contains node partitions that you wish to replicate actions across. This ensures that if one node goes down, enough copies of any given data exist elsewhere.

00:13:51.170 Interacting with the nodes involves simple serialization and deserialization of the objects being shared to maintain coherency during operations. This method is essential when adding in our messaging network so that servers can communicate effectively.

00:14:40.260 When we talk about messaging patterns, we have three key types: request-reply, publish-subscribe, and pipeline. For today, we're using request-reply to query nodes. Each node will essentially run on a separate server, and when you query a node, they will forward requests if the value requested is beyond their reach.

00:15:38.520 With this implementation, I’m using ZeroMQ to facilitate communication between nodes. If you’re unfamiliar with it, I highly recommend you take a look. ZeroMQ is transport-agnostic, making interactions simple and efficient. It handles buffering and framing for you, enabling a straightforward message-oriented protocol.

00:16:38.940 During operation, I created two threads for a server and a client connection. The server listens on a defined port for messages containing the 'put' command. Upon receiving a command, it will take action accordingly, reflecting changes made from the client.

00:17:34.530 As the nodes began to communicate, values were retrieved and stored efficiently. Even though the implementation may have its drawbacks, being able to query nodes dynamically reflects positive scalability considering potential network issues.

00:18:22.540 However, opportunities for improvement remain. Though I can retrieve data from individual nodes, we haven’t implemented our replication strategy yet. A system that lacks redundancy could lead to data loss if a node crashes.

00:19:06.430 To mitigate this issue, we're introducing a leader election process for management of ring changes when new nodes join or leave the network. Although this can create potential bottlenecks or points of failure, ensuring reliability is paramount.

00:19:46.230 When a node joins, it will inform the designated leader node, and the command will propagate through the network. We employ ZeroMQ to immediately update any changes across the system, keeping all nodes informed of necessary changes.

00:20:40.930 Next comes the challenge of maintaining availability when node failures occur. Introducing replication as a standard procedure allows for error reduction, ensuring that even if a node does go down, the data remains intact across other nodes.

00:21:31.570 We implement a replication strategy by configuring the nodes for simultaneous writing while managing read requests. This way, even if a crash happens, the most current data can still be accessed promptly.

00:22:29.730 However, synchronizing data can present conflicts when multiple updates occur concurrently across different nodes. To handle this, we utilize vector clocks to keep track of changes logically without needing physical synchronization. This allows for a reliable means of conflict resolution.

00:23:00.110 As the demo progresses, I run through various scenarios displaying how managing data copy consistency plays out in practice. There are various methods employed: picking the latest update, labeling the changes that came about, and merging those results efficiently.

00:23:50.600 While handling system corrections, we recognize the importance of Merkle trees for structured validation processes. As additional nodes participate in the network, accessing and verifying distributed data becomes manageable and adheres to best practices.

00:24:45.470 To circle back towards the end, the final topic we discussed was the MapReduce framework, which permits extensive data processing through distributed parallel computing. This represents a compelling structure for efficiently managing complex queries within a key-value store, enabling synchronized aggregation of results.

00:25:45.280 To conclude, what we’ve done in Ruby provides a powerful foundation for a distributed database solution capable of robust operations. Having implemented conflict resolution, replication, and self-healing mechanisms holds promises for a reliable database.

00:26:36.490 In summary, distributed systems need to be effectively designed to address challenges while ensuring reliability, availability, and performance. Ultimately, the principles we've engaged today illuminate how best to leverage tools and practices in Ruby for fostering scalable architectures.

00:27:32.450 Thank you all for your attention! Are there any questions?

00:27:52.930 That's an excellent question. The preference list for replicated values should provide higher availability. We will implement this very soon. It’s perhaps one of the first things that should cross your mind with regard to ensuring data integrity.

00:28:34.780 Indeed, replication strategies need to catch up with dynamic cluster scaling. But I will show how to address the situation strategically. We aim to ensure that every message is received, leading to timely adjustments—variable dependability is crucial in distributed systems.

00:29:18.540 We introduce a pub/sub method for contacting all nodes in the cluster, which is essential for communication during updates or errors. This method can help elect leaders and redistribute information over the network.

00:30:04.160 Once nodes are informed, they conduct their corresponding actions as directed by the leader, making the ongoing collaboration feasible. A well-orchestrated solution ensures that higher-level directives can promptly manage joins and failures.

00:30:34.540 Finally, let’s see how SLT nodes handle replication across partitions. As you can appreciate, the overhead is generally manageable when balanced out against the operational advantages.

00:31:00.410 So, we can regard our summary of effective data sharing, fault tolerance, and distributed pattern implementation as a way to overcome conventional barriers. This foundation in Ruby can cater to dynamic systems, paving the way for significant collaborative processes.

00:31:45.680 Thank you very much!