00:00:14.780
Thanks, everybody! I'm really excited to be here. We're going to talk about microservices. Who here has worked with microservices? Almost everybody! Who has at least heard of the term before? Okay, that looks like everyone. Awesome! First of all, that's no moon; that's a Venn diagram. Microservices is kind of a big universe, and we're focusing on just a small piece of it today: the UI implications of microservices. It might feel a bit overwhelming at first—like a firehose of information—so I apologize if some context is missing. We'll try to clarify things.
00:01:02.539
Who am I? As Adam said, my name is Ethan. I live in the western United States. In the spirit of full disclosure, I actually make my living programming in Node.js and not Ruby. However, I got my start in web development using a popular Ruby framework for creating web applications, though I won't name it since this is a Ruby conference. That being said, I’m currently writing a book for the Pragmatic Bookshelf about microservices, and I would love to discuss that with anyone interested.
00:01:45.500
I want to clarify that this is not necessarily an answers talk. There's a French philosopher named Claude Lévi-Strauss who said that a scientist is not someone who provides the right answers, but someone who asks the right questions. Some talks will equip you with all the knowledge you need to start coding right away, possibly teaching you about new libraries or frameworks. Unfortunately, that's not the goal of this talk. It's about asking questions and ideas, and how you can reason about the systems you're building. Because I believe that being able to ask the right questions is far more valuable than knowing specific lines of code.
00:02:31.459
So, what do I mean by microservices? There seem to be as many definitions of that term as there are cheeses in France. In the spirit of being on the same page, or having a ubiquitous language—as domain-driven design might suggest—let's start with what a monolith is. A monolith is not just a deployment strategy or a way to put all of your code into one repository; it is a data model.
00:03:30.730
To illustrate, I mentioned that I began in web development with a Ruby library for building web applications, and this library pointed me toward a specific architecture. A big part of it was the user's table. Who here has ever worked on a project involving users? Yes, I see some hands! That table is not unlike the ones I've seen in my early projects. At first, it was amazing—fifteen minutes from zero to a blog—that's incredible! But over time, it became challenging. I kept running into this wall of productivity, not just on projects I've initiated but on every project under this framework.
00:03:53.650
If you look at this graph and squint a bit, you might start to wonder if these pieces of data really belong together. When I look at this, I see at least four different domains. For example, an email and a password hash are useful for identifying a user. That’s how people typically log in. But what do those attributes have to do with tracking which features the user has interacted with in the system? These pieces of data are unrelated; the only thing they share is a user ID. And that's a prime example of the problem with monolithic structures.
00:04:43.700
When you try to fit all these distinct domains into a single representation, you end up with a monolithic data model. If you build an application on top of a monolithic data model, you have a monolithic application. Here’s an example: I've called out a user’s domain and a product’s domain, with a database in the middle. The greenish box indicates that this is all running in one program.
00:05:12.420
When two different functions are running in the same program and they communicate with each other, what is that mechanism called? What do you use to have one piece of code call another? Yes, those are method calls, or function calls. Exactly! So, when you hit that productivity wall, where it becomes difficult to work, you seek help from the blogosphere. We all love the blogosphere, and it often gives you advice like, "Just extract microservices." So, you pull out your user service, and it now runs on its own server, with its own process and database. Surely, you have services, right?
00:06:06.000
But, hmm... what are those connections now that they are all on different servers? That's right—HTTP! Everything else looks exactly the same. My access patterns are all the same; the only difference is that they're on different servers. All else being equal, who would prefer to deal with HTTP calls rather than function calls? It introduces greater operational concerns and a myriad of things that can go wrong. Thus, we encounter statements like the one from Aaron Patterson, a smart guy who has contributed significantly to the Ruby community, who amusingly remarked that microservices are great for transforming method calls into distributed computing problems. That's not what microservices are about.
00:06:38.660
So, let's engage the audience here: Distributed + Monolith equals... drumroll... distributed monolith! If you've worked with microservices that resembled this structure, you probably had a miserable experience. My first encounter with a service architecture was with one of these monstrosities, and I hated it. I ended up leaving that company because I spent months suggesting we should just combine everything back together because it was so difficult to work on. But it's crucial to recognize that this isn't what true services are; so if you had a bad experience with this, it wasn't a reflection of services themselves.
00:07:52.130
What do I actually mean by services? There are many factors that define what constitutes a service, but the most significant is autonomy. What do I mean by autonomy? A service does not request anything from other services. Services don’t have 'GET' APIs; they don’t represent things. We have components in our architecture that we obtain information from; we call those databases, and the process of querying them is called a query. If you take a database and put it behind HTTP, you still just have a database. In fact, some databases can respond to HTTP out of the box, so they remain databases. Furthermore, a service doesn’t depend on anyone else. For instance, if I have to ask you for something to do my job, you now become a dependency of mine, and I’m no longer autonomous. Dependency is the opposite of autonomy.
00:08:48.200
A great feature of this setup is that a service is completely oblivious to the existence of anything else in the system. Therefore, if one of the services goes down, it doesn’t cause the entire system to fail. Sure, the system won’t function as efficiently, but it won’t break entirely.
00:09:32.360
So, how do services communicate? This occurs through asynchronous messaging, with two types being events and commands. Events can be represented as JSON objects but will include a type, such as 'pen dropped.' This example represents a service that drops and picks up pins, but as the system designer, you determine those types. They're just strings, with associated data payloads recorded in the past tense. This is key because events signify actions that have already occurred and cannot be disputed. If you don't like a state indicated by an event, you can issue a command. For instance, 'drop pen.' If I were holding a pen and you wanted it on the ground, you would send a command to drop the pen, which I may accept or reject.
00:10:06.290
Commands are in the imperative mood; they instruct you to take action and can be invalid or rejected. That's how these services communicate. What’s entertaining about event sourcing is that these events ultimately define the system's state. In the traditional MVC CRUD world, you might just reflect a row in a database by stating that pen number one is currently in a picked-up state. However, the reality is that it was first dropped and then picked up. When you design a system like this, what are the major components involved?
00:10:35.160
Let’s consider a few illustrations. First, you have services, represented by lightning bolts because they perform actions. If you've ever been hit by lightning, you understand what I mean, although I hope that hasn’t happened to you! Next, there’s the application layer. This doesn’t imply there’s only one application, but this is where projects based on that Ruby web framework would reside. In the Node.js world, this would be your Express server; in Python, it would be Django. The application takes in requests from satisfied users, as we are professionals and our users appreciate us.
00:11:06.290
The application then writes commands, and sometimes events, to a transport mechanism often referred to as a message store. Services will observe these commands, write new ones, and generate new events in response to them. You also have what I term read models—materialized views or data views—which are crucial because lists of events are not typically useful for rendering a user interface unless you're an auditor. Most people are not auditors, so we require different data shapes.
00:11:30.200
Then, there are aggregators that take the events—a linear list of events—and shape them into formats that make more sense for displaying to users. None of this necessarily supposes that we’re discussing a distributed system, because services don’t equate to distributed systems. You can have distribution for various reasons, but they operate independently of one another.
00:12:14.900
What is a message store, then? A message store is a specialized database, optimized for storing and retrieving events. You might have heard of the Event Store project. Greg Young frequently discusses microservices and events. He has facilitated the creation of event-type projects, and Nathan, who spoke here last year, is one of its co-maintainers now, along with Scott Bell. There’s also the Rails Event Store maintained by Andre and his colleagues. These examples serve as message stores, meaning that’s the firehose introduction to the world we're discussing.
00:12:32.930
Hopefully, that made sense! If not, we’ll be around for the next couple of days. Please feel free to engage with me because I love this topic. What I didn’t mention is that the messages getting published follow a pub-sub style. If I’m publishing a message, I don’t know who will consume it, and if I react to any message, I don’t know who sent it. Everything operates asynchronously, which can result in potential odd situations.
00:13:06.910
For example, in a user interface, when a user does something, they expect a response. But how do I provide that response when the required information is not yet available? That's the focus of the remainder of our discussion. First, let’s discuss a common scenario. Who here has ever worked with 3D graphics or a game engine like Unity? A few people! Who’s ever played a video game? More hands! You see those objects in a 3D video game, and the way they get created is by defining the geometry of that object. Here’s a case in point: a zebra.
00:13:41.890
Using what’s known as a texture map, someone links the distinct points on the zebra to points on a 2D texture that an artist would generate. Remember, these artists are very talented! If you were to take the texture for a human head and map it to the geometry of a zebra, you would end up with a disastrous outcome. The point I’m trying to make is that if you're working within web development, you're likely coming from the MVC CRUD style of architecture. Consider that as the zebra—messy and unruly.
00:14:39.040
The texture, in this case, represents the ideas inherent in the MVC CRUD architecture. The model, in turn, could be your mental model. When you’re introduced to fresh ideas from service architecture, if you try to adapt those ideas and fit them into the outdated MVC CRUD model, you’re going to become frustrated and confused. You might even go so far as to write articles like "The Majestic Monolith." So, if you’ve started building a microservices system, and you reach a point where you simply have an HTTP form over a row in a database, and now you’re just considering the implications of an eventually consistent pub/sub model, you are well past the ideal point of rethinking and reevaluating your strategy.
00:15:15.850
You need to take a step back and reassess. Start by sidelining all the technology. Let’s focus on the experience you wish to deliver to your users. Once you have a concrete idea of that—obviously revealed through discussions with your business team and your UX experts—then you can start determining the appropriate technology. Technology is the easy part; figuring out what to show your users is the difficult part.
00:15:49.330
Let’s get into specific use cases. The first one I'd like to discuss is when sometimes 'nothing' is enough. There are cases where it’s entirely valid to ignore the asynchronous nature of your system. Consider analytics data tracking, for example. Suppose you have a website tracking video views, much like YouTube. Do your users really expect that data in real-time? If you watch a video on YouTube and reach the end, do you anticipate the view counter to increment immediately? Personally, I don’t. I’ve even observed that some videos, when they reach a specific count, cease to increase for a while because they start investigating if those views are legitimate.
00:16:40.140
In this case, there's no expectation for real-time feedback when tracking analytical data, and it’s perfectly acceptable for there to be some lag. The flow of data through the system might be: a user watches a video, and a request is sent to the application. The application then writes an event to the message store, stating, 'The video was viewed,' and at that point, the application can render a response to the user.
00:17:35.250
Here’s an illustration of this in action. This amazing website called Video Tutorials has a homepage, but if you can’t tell, there are no actual videos on it. I can click a button to simulate watching one of the videos, and you'll see right now, it indicates that only one video has been watched. I’ll refresh the page to confirm this, then click the button to simulate a view.
00:18:11.190
This is exciting; it's going to send a request to the server indicating that a video has been watched. The server responds, 'It did it,' but you'll notice the count still shows just one video. Why does it still say one? The request is sent to the server, and an event is recorded, but it’s the aggregator's responsibility to update the database indicating how many videos have been viewed. That process hasn’t been completed yet; therefore, the user doesn’t see real-time updates, and that’s okay because real-time feedback wasn’t anticipated.
00:18:58.300
If I refresh the page now, lo and behold—the count has updated! Incredible! We saw it here first. Sometimes, it’s perfectly acceptable to not worry about immediate data updates. As Elsa says in the movie Frozen, 'Let it flow, let it flow,' because that’s what you should do—allow the data to unfold as needed!
00:19:38.140
The next point I like to term 'stalling for time.' Who here has registered for a website before? Nice! When you registered, did you end up on a thank-you page that said, 'Thanks for registering! We'll send you an email, so please click it to confirm your email address'? This practice happens frequently, and I like to think of it as a way to stall for time. There are certainly valid reasons to verify an email address, but it also provides a perfect cover for masking the fact that when you registered, there wasn’t yet a formal record of your information. It just needs to flow through the event system.
00:20:25.210
You might see a similar strategy when you place an order on an e-commerce site, for example. When you submit an order, you may receive a page saying, 'Thank you for your order!' Without further information, you won't necessarily expect anything else because they'll inform you if a problem arises. In the case of bank transfers, those indeed don't happen instantaneously. You don’t always need to provide real-time information to users.
00:21:14.190
Think of this metaphor: this is a swan—so graceful and elegant. They appear magnificent above the water, but if you dive beneath the surface, you might find that their feet are thrashing about. This represents the stalling-for-time strategy effectively. In scenarios that necessitate background processing, you want to convey to the user that something is occurring. For a user registration process, you can have them submit their information, leading to a screen that instructs them to check their email for further instructions.
00:21:56.550
Upon submitting their registration, the application will indicate this action. In the background, the registration service writes a command picked up by an email service to send the confirmation email. This occurs in less than a second and fits neatly within the request-response cycle. However, your users remain blissfully unaware! Unless they literally have your website and their email open simultaneously and keep refreshing the page, they won’t realize any delay. You get to look like a composed swan while your users remain satisfied.
00:22:50.470
Both of these examples highlight that while coming from an MVC CRUD background, there's often a hyper-optimization for immediate responses. Of course, there are cases when immediate answers are necessary, but many scenarios don’t require instant feedback. The MVC CRUD training can convince you that every response must arrive immediately; otherwise, significant problems arise. This isn’t always the reality across various cases, so it’s important to ask: what experience do I need to present to users? If they don’t expect instant feedback, it's perfectly acceptable not to provide that.
00:23:36.530
As we near the need for faster responses, it's necessary to mention that an event-based system requires considerable upfront analysis. I don’t mean to suggest a waterfall approach where you spend nine months planning, but even a week or two spent figuring out your direction is still agile by my standards. If taking one or two weeks to sort out your plan can save you from issues arising during development with a well-known Ruby framework for web applications, that's time well spent.
00:24:38.000
Now, returning to the video tutorials example: suppose you’re allowing users to upload videos since you need that content for your audience. However, there are rules regulating valid names—you don't want blank names or names with inappropriate words. Those rules find their place within a service, but they enforce processes asynchronously. So, when users submit names to me, I won’t immediately know if they are acceptable. I could duplicate those rules in the application, but if something changes, I’d be forced to update in two locations.
00:25:31.500
A more effective solution rests in adopting the event-driven model. We don’t want to display invalid names to viewers. If I’m the content creator and all of you are here to enjoy the content, I wouldn’t want to show inappropriate titles to you. Nevertheless, it would be advantageous for me to see the invalid name I submitted in order to correct it.
00:26:11.450
Here’s a little narrative: Mr. Grumpy Pants wants to make the web a frustrating place for users, utilizing any crude tools at his disposal. He attempts to submit an offensive video name. If I’m still operating within an MVC CRUD mindset, alarm bells will ring because I must prevent that submission! But I note that this representation of a stop sign is inaccurate—it's not what stop signs look like in Europe or the States! Nevertheless, those alarms are going off because this particular row in the database represents the entirety of the video in the system. If anything goes awry here, I won’t have any recovery method available because I lack that history.
00:26:54.440
The benefit of using an evented model, however, is that it isn't a disaster. Imagine this scenario: the video has been uploaded, Mr. Grumpy Pants sends his request to submit a horrible title, and the application records that title. Although he’s momentarily happy because the name got accepted, an important aspect is that we recorded the submission of this inappropriate name as part of the video's history. This event can't be disputed; it occurred. But wait! We still have events and aggregators that can help us with this situation. Using CQRS, as Andre discussed earlier, I can take these events and reformulate them into a user-friendly view.
00:27:56.910
The happy viewers see a version of the video with an appropriate name, while Mr. Grumpy Pants sees a prompt that instructs him to correct his submission. In essence, you can derive different perspectives from the same data, elevating what would normally be validating data at the database level into something relevant to the core business model. You shouldn't compartmentalize validation within the database layer. The validations for your system are part of its core domain.
00:28:51.170
By removing them from the database layer, you can allow a service to control that part of your model. Rather than being an ancillary issue tied to persistence concerns, validation becomes a top concern within your architecture. Through this approach, you can manage potentially disastrous situations by displaying one version to users while offering insightful feedback to data submitters, helping improve the overall experience.
00:29:36.720
Now, let’s conclude. Once again, I want to emphasize my love for microservices architecture—not the wine, but the principles behind it! The separation of concerns, the delineation between write models and read models, is fantastic. It’s enjoyable to work with, and when you’re engaged with a distributed system, writing an event while your electronic minions act on your commands is always exhilarating!
00:30:01.720
However, if you approach microservices with an MVC CRUD mentality, it’s going to be nonsensical for you. Picture a zebra-shaped model with a human face—it’s unsettling! The takeaway here is to contemplate what you're aiming to provide to your users prior to writing any code. This is sound advice for any project, but it’s especially critical in a your architecture like this. Embrace the event model; it grants options previously unavailable, such as accepting bad inputs and managing them retroactively.
00:30:50.220
Continually engage with both your business team and your UX experts. We’ve all seen the interfaces developers create, and while there are a few exceptions, I doubt anyone wants to use a UI I’d design! Always ask yourself the right questions, and you’ll find technological solutions will emerge naturally. If you’re seeking further information, I highly recommend exploring the Event type project. It’s excellent reading, even if you don’t use it daily; it will broaden your perspective. Although I primarily work in Node.js, this approach has greatly improved my comprehension of microservices.
00:31:36.560
I maintain a mailing list if anyone’s interested, and while the slides from this presentation haven’t been posted yet due to issues sharing them through my phone, I will get that sorted out. I also have code related to the subjects we covered today and will share it—likely via Twitter. Again, I’ve included many valuable resources and great minds to follow in this field. Since this is a Ruby conference, pretend the last reference doesn't exist. Happy coding!
00:32:10.150
Any questions? Yes, what constitutes a processed command for you?
00:32:39.220
I consider it processed when the service has produced the event because the service is ultimately responsible for processing the command. A slight delay may occur before aggregators pick that up and share it with the rest of the world, but that moment of event production is when the command is considered processed. Thank you!
00:32:58.919
Next question?
00:33:12.709
I wonder, if we validate each piece of data this way and ask our data creator to adjust it, wouldn’t it tremendously increase the complexity of the system? We could have not only the video title that needs adjustment but also the description and potentially several other fields, necessitating new UI components.
00:33:40.960
That’s an excellent question! To clarify, you don't necessarily have to implement this process universally. The MVC CRUD paradigm will still work in specific cases. All I’m suggesting is that alternatives do exist. There are methodologies I didn’t discuss, such as task-driven user interfaces. Most systems I've encountered present users with massive forms showcasing every field in the database, expecting them to edit everything before submitting.
00:34:02.550
One disadvantage of this approach is that the necessary process isn't embedded in the software; users must grasp it independently. If I work on this software and subsequently leave the company, I hope I’ve sufficiently trained someone to understand the entire form submission process. By employing task-driven interfaces, your forms might become more concise initially. You're less likely to submit a massive form festooned with items that require fixing overall.
00:34:51.769
It’s likely true that adopting this method would yield more screens to design, but my experience suggests that I don't perceive it as adding complexity—each screen features less complexity than larger forms would have within an MVC CRUD model. So you manage a number of smaller elements that are easier to handle than a single, more complex element.
00:35:36.750
Have you ever had one of those piggy banks where when you drop a coin inside, it rolls down and falls through the intended slot? That’s akin to how an event-sourced system operates—the events flow smoothly and hit their targets, functioning as expected, while with a CRUD framework, you have to juggle a host of validation processes. So typically, you would have more elements, but they’re more manageable in size and complexity. Adopting smaller elements helps to streamline the workflow.
00:36:36.910
Thank you for that question!