Evented Autonomous Services in Ruby

GORUCO 2018: Evented Autonomous Services in Ruby by Scott Bellware

GoRuCo 2018

00:00:15.680 My name is Scott Bellware, as Luke embarrassingly pointed out. I really hope that this presentation lives up to the introduction, so thank you for the extra nerves, Luke. I'll get you back.

00:00:22.619 I'd like to talk today about evented autonomous services in Ruby. But first, let's discuss why we would want to do such a thing. I want to set a little context; when I'm talking about services today, I'm referring to services in the framework of service-oriented architecture, which inevitably means microservices. Microservices are a derivative of service-oriented architecture, so we're not going to discuss service objects, which are those interactor-type things that you find in a folder called 'services' under your app folder in your Rails app. We're also not talking about operating system processes, like you might say that your MySQL process is the database service, whereas your web server process is the web server service. We're not talking about apps because apps aren't services, and we're not talking about APIs because APIs aren't services.

00:00:46.140 So, why services? It's easier. Sure, there are benefits we always hear about, like scalability and performance, but ultimately, for me, it’s about productivity. Why is it easier? Because services are focused on a very limited number of things; they’re highly cohesive and respect all the design principles that guarantee and ensure productivity happens. That's ultimately all about decoupling. Decoupling, as much as it is a structural design principle, is also a user experience and cognitive optimization that allows you to think about only the thing that’s under your nose in your editor, without having to think about secondary effects. So, it’s ultimately easier to see mistakes.

00:01:35.729 As we talk about why we would want to do this, we should really focus here on why we would want to do this in Ruby. Ultimately, it's because you have skills in Ruby; you've invested time and energy in mastering the language and the runtime, along with the libraries. You have a code investment in your organization, in building core and business logic that’s ultimately an asset you don’t want to throw away. Sometimes, we reach a point in our lives — as Luke pointed out — where we want to leave and do something different; we need a lifestyle change. That’s a separate issue, but if you are a Ruby developer in a Ruby shop, I think a better question is: why wouldn’t you build services in Ruby?

00:02:54.060 There’s also a career component here. If you have a career in Ruby, you inevitably have a career in Rails. As you move through your career in Rails from junior developer to senior developer, you may find out that Rails has its own glass ceiling. The good news for beginners is that everybody moving along this line tends to gather at the top, and ultimately many burn out or move on, leading to a cascading effect that demotivates junior developers who see senior developers leaving or burning out. I would like to propose an alternative career path where the upper level of achievement is not limited to monolithic Rails development work, which we find so familiar. A junior developer can progress to a senior Rails developer, but there’s also an entirely different career path available that doesn’t require a dramatic change unless that’s what you truly want.

00:04:47.690 When discussing microservices, there’s a lot of ambiguity around what they are and the differences among them. Martin Fowler has a great talk from the GOTO Conference in 2014 where he mentions the notion of ‘smart pipes and dumb endpoints.’ In microservices, we can think of smart pipes as things like enterprise service buses, brokers, and service orchestration, which are message transports that move messages around. In the early days, we had elaborate tools where we would put intelligence into them, deploying these transports to production. We envisioned letting senior-level developers configure the tools while junior-level developers focused on application logic, completely ignorant of the messaging happening behind the scenes. However, that approach did not work well.

00:07:03.890 These tools often had state, databases, and configuration, and moving that stuff around to production and updating it was not as efficient as we thought. When we were debugging those systems in production, it was a challenge due to the lack of debuggable intelligence. What is a message? A message is essentially an object with attributes that can be serialized and is sent over a transport. For example, if we have two services, an instruction is sent from one service to another. We instantiate a message, give it some data, and send it over the wire to be processed, after which it's removed. This raises the challenge of guaranteed message delivery — there’s no assurance that a message will ever reach its destination.

00:09:02.930 If an acknowledgment isn’t received, the message remains in the queue and may be resent, resulting in the risk of reprocessing a command, such as a direct deposit for salaries. If a processing error causes everyone’s paycheck to be processed twice, the results could be disastrous! To illustrate the difference between queues and streams, one of the authors of a special-purpose database for message storage called EventStore said: 'What is a queue but a degenerate form of a stream?' Streams can accomplish everything that queues can, but the reverse is not true. In event streams, services write messages to an endless log, while another service reads messages by incrementing the message number.

00:10:45.510 The main distinction here is that messages aren’t moving anywhere; they stay the same, remaining unacknowledged as their offsets are read. When we deal with event sourcing, we’ll have a series of commands related to, say, bank account transactions. The results of processing the commands lead to several events recorded in a stream. For example, the initial withdrawal command has an ID, amount, and a timestamp. The corresponding withdrawal event will have those attributes plus an additional timestamp indicating when it was processed. In this model, we are effectively copying data from the command to the event and recording it, along with a timestamp.

00:13:35.800 The process involves validating a command, as you would with an HTTP form, retrieving an account entity, and applying logic to determine whether to accept or reject the command. For instance, if you want to withdraw $100 but only have $10, you reject the command with an 'insufficient funds' event, and write that event accordingly. Using a toolkit called Eventide, which is designed for building evented autonomous services in Ruby, you implement the retrieval and validation logic. The account object must determine whether to accept or reject the command, and recording that decision will create an event.

00:16:47.560 Next, we delve into event sourcing. In an event stream like our bank example, various account events are stored, such as deposits and withdrawals. An account entity's state is built from events, unlike a typical Rails model that would talk directly to the database. Event sourcing involves applying these events in sequential order to generate the current state of the account, reflecting the balance by processing each event one at a time. The events provide key data points; for instance, when a deposit event occurs, the balance updates according to the transaction. Each event incrementally builds the entity's state until it reaches the latest record.

00:19:21.700 While processing commands and events, we must ensure the application logic is transparent. This may seem complex and daunting, but your event sourcing code can remain clean and concise as long as you adhere to proper design practices. That's where domain-driven design comes in handy. Conceptually, services are designed to operate autonomously, meaning any service can be taken offline without impacting the others. Autonomy is crucial; it prevents cascading failures in your architecture, an essential design principle for successful microservices.

00:22:43.990 In our journey to build autonomous services, it's important to understand the vocabularies. A service should not have HTTP GET methods and should not represent data; it must function by executing tasks. Creators often confuse services as databases or tightly coupled applications. However, services should remove dependencies on specific state management, allowing them to function independently. New developers often feel obligated to switch languages for building services, but the truth is that language choice is irrelevant to architecture—as long as the target language can support the service design.

00:27:52.950 We used to think that containers were a requirement in service architectures, but they hold no inherent relation to functionality. A major misunderstanding is associating services directly with containers, as deployment concerns distinct from service responsibility. When working with services, the communication methods defined by their messaging and commands promote effective interactions. Finally, the common myth that systems are evolutionary steps towards microservices needs re-evaluation; they often lead to creating distributed monoliths rather than true service-oriented solutions. This will be more evident as you begin applying these principles in your work. For additional details or to explore tools for building evented autonomous services in Ruby, please visit the provided links. Thank you!