Ruby Video
Talks
Designing Hypermedia APIs
Steve Klabnik
@steveklabnik
#rest-api
#ruby-on-rails
#api-design
#decoupling
Summarized using AI

Designing Hypermedia APIs

by Steve Klabnik

In the talk titled "Designing Hypermedia APIs" at RailsConf 2012, Steve Klabnik explores how to create effective APIs that truly leverage the power of the web and HTTP, moving beyond traditional RESTful designs. He begins by addressing the misconceptions around REST in Rails, illustrating that many developers feel dissatisfied with its implementation. Steve asserts that while Rails has popularized REST, it does not fully embody the original principles of REST advocated by Roy Fielding in his dissertation. Here are the key points discussed:

  • Rest vs. Hypermedia: Klabnik emphasizes a shift from typical REST APIs to Hypermedia APIs, which enhance client-server interactions through links and guiding processes rather than relying solely on URL endpoints.
  • Importance of Learning: Steve shares his personal journey of understanding REST and hypermedia, indicating the significance of continuous learning in software development.
  • Pros and Cons: Hypermedia APIs allow for better scalability and change management but may introduce latency issues, making them unsuitable for applications requiring immediate responses.
  • Design Steps: He outlines five critical steps for building a hypermedia API: evaluating business processes, creating a state machine based on those processes, determining media type needs, designing a suitable media type, and implementing the API.
  • Case Study: Klabnik discusses a practical example involving an API that wraps the W3C HTML validator, employing a structured format to improve usability and flexibility in processing requests.
  • Media Type Design: The importance of building or leveraging existing media types to drive hypermedia functionality in JSON is also discussed.

In conclusion, Steve Klabnik advocates for a future where APIs evolve to be more adaptable, guiding clients through interactions rather than simply serving data. This results in a more effective and user-friendly approach to building web applications, emphasizing the long-term benefits of thoughtful API design over short-term efficiency. Through hypermedia, developers can foster robust, scalable applications that are more responsive to user needs and business changes.

00:00:24.640 If you want to follow along on GitHub, I have this presentation. It's a show-off presentation, so the slides are also up there. If you want to see the latest slide stuff, I can pretend you're not tweeting and not paying attention to me while you follow my slides. So that's the link to the show-off if you want to follow along at home.
00:00:50.170 Alright, this is a talk called 'Designing Hypermedia APIs.' First of all, hello everybody! I'm Steve. Thank you for coming to this awesome RailsConf first thing in the morning. It's bright and early, and there was a pre-party last night, so I appreciate you waking up at 10:30 AM, which is always difficult.
00:01:08.360 If you don’t know me and what I do primarily, I hack on Hackety Hack, which was originally created by Why the Lucky Stiff to teach programming to kids and other people via Ruby. I try very hard to not make it just for kids but to make it accessible for anyone learning how to program. I also teach classes with Jumpstart Lab, so I’m all teaching all the time. If your team needs to get better at Ruby and/or Rails, give us a call. We run the best training classes on Earth.
00:01:21.650 We're doing the Hunger Academy program with LivingSocial right now, which is pretty awesome—taking people who have never been developers before and turning them into professional developers. So, it's a pretty cool experience.
00:01:41.360 Anyway, this talk is sort of about a journey that I went on in the spirit of a Native American spirit trip or whatever our conception of that is—not a real one, but the idea of one. I sort of went wandering into the desert to find myself, and when I came back, totally disgusting and not having showered, I said, 'Guys, guys, I saw the answer out there in the desert!' You were probably thinking, 'Yeah, sure Steve, whatever.' But I have this continuous idea of a personal ethic of always learning.
00:02:15.970 I always like to be learning new things, whether they're programming-related or not. Whenever I hear someone I respect say something I don't understand or I think they're wrong, I make a mental note. For instance, Kota Hale is not stupid, but he thinks that Rails is bad—someday I should figure out why he thinks that because I respect Kota's opinion. This happened with REST one time; I heard a lot of knowledgeable people say that Rails' idea of REST is not actually REST. I thought that was kind of weird, so I wondered why they thought that.
00:02:53.860 Eventually, when I'm bored, I look into things I don’t know about yet to see why they might hold that opinion. One day, I decided to do that for REST and realized that Rails' REST is not actually real REST. I'm a really big fan of Lean methodologies for startups and projects, so I decided to essentially lean startup a book. I announced that I learned why Rails is not actually REST, and I planned to write a book about it. It’s not written yet, but here's an email form where you can leave your name, and I'll email you when it's ready.
00:03:30.480 Initially, I did this to see if my laptop would fall asleep and to gauge whether or not anyone else cared about the topic. It turns out that many of you did care, so I spent the next eight months of my life researching to ensure I knew my stuff. The last thing you want is to tell everyone you're an expert and not be one. I read everything relating to REST, and this presentation is a synthesis of what I’ve learned. The book I’m writing will be a bigger synthesis of what I've learned, but we'll discuss that later.
00:04:17.680 The first thing I want to assert is that REST is over. If you've seen this Portlandia sketch, it actually has a much deeper meaning if you think about what happens in the sketch. Basically, if you're trying to convince someone to change their opinion, the last thing you want to do is tell them they're stupid. If you act aggressively towards people, they're not going to change their opinion; they’re going to dig in and fight back.
00:04:49.590 If I say, 'Hey, you guys don’t actually know what REST is because you’re stupid,' your first reaction would be to say, 'You know Steve, you suck; you don't know what you're talking about,' which isn't a productive way to change people’s opinions. Secondly, REST has kind of bad branding. Does anyone know why it's called Representational State Transfer? It totally is meaningless. This is not really Fielding's fault— whose dissertation defined what it means to be restful. When you write academic papers, you're not concerned about marketability.
00:05:21.680 You're just focused on getting your work done, and so it’s not Fielding’s fault that this buzzword has become essentially meaningless. Since no one knows what it actually means and everyone uses it wrong, rather than trying to fight over the term REST and arguing over what is the right REST, it’s much more productive to invent another buzzword. Those of us who care about what REST means have decided that fighting about it isn’t worthwhile.
00:06:04.360 In fact, Rails does REST perfectly, so in the rest of this talk when I say REST, I will be referring to Rails' idea of REST—not the actual idea of REST. To move forward and have a productive discussion, I want to clarify that this doesn’t necessarily mean it's bad. You heard what DHH said earlier today, right? People said that Rails' REST was useless and now we consider it the default.
00:06:47.880 I would much rather work with a Rails-ish REST API than SOAP or any of the other really terrible APIs we had before. It is fundamentally a good thing, but I believe we can do better. So, in the spirit of not resting on our laurels, if we really want to move forward with large applications and awesome APIs, we can do better than what we are currently doing. This talk is about how to move from the standard APIs we know and use today to something better.
00:07:30.220 Instead of REST APIs, we're going to talk about hypermedia APIs. This communicates much more effectively what I'm actually talking about if you know what hypermedia is. Basically, hypermedia is the idea of having media or text in an older context, and you add links to it—that's the 'hyper' part. It's about connecting things together. Hypermedia APIs are fundamentally about transferring text documents or other media types that contain links to connect them to the rest of the world.
00:08:25.570 There is a reason we talk about why this matters and why it's important. It's also a good name because it points out the fundamental differences from the Rails idea. How many of you return links in your JSON representations to other resources in your API? You don’t. Some of you might, but generally, we don’t. This is a fundamental difference between what I'm talking about and what we do today: it revolves around these links.
00:09:02.360 So why should you bother with hypermedia APIs? Before discussing the details, let's cover why you should care. Hypermedia designs scale better, are more easily changed, and they promote decoupling and encapsulation, all bringing the benefits of good software engineering. I liked what DHH said earlier, even though I don’t completely agree with everything.
00:09:28.560 He alluded to how we developers love to say things like, 'I love change, I love decoupling, I love encapsulation,' but then we create directly coupled accessory classes and make huge classes that go on for miles. If you ask anyone if they write tightly coupled code, they’ll say, 'Of course not!' But Roy Fielding pointed out that APIs today often have an X-rated level of coupling.
00:10:06.950 We have incredibly tight coupling between clients and server, and we feel that pain subconsciously without realizing that we can do better. This approach is about radically decoupling things. The negatives of this—like any architectural choice—come down to latency. If you need something to execute in microseconds, then hypermedia APIs are not for you. Caches are one way these APIs can scale.
00:10:54.990 As the joke goes, there are two hard problems in computer science: naming things and cache invalidation. Cache invalidation is notoriously difficult to get right. Also, since these APIs generally transmit text, they won't be as efficient on an individual level as binary protocols. For example, if you compare JSON to BSON, BSON is much smaller because it represents data in a custom binary format that is more compact than text.
00:11:37.970 Yet, text has its own advantages, so typically, this increased size for data transfer doesn't matter, but sometimes it does. Roy Fielding has this great quote that I enjoy: 'REST is software design on the scale of decades. Every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency.' Unfortunately, people are usually better at short-term design than long-term design.
00:12:30.620 It’s natural for people to think short-term. For instance, I wear Vibrams but only go to the gym once or twice a month, knowing that running would benefit my long-term health. People often make short-term decisions that come back to bite them later. In software, we tend to have deadlines and release with the mindset of 'screw if the software is good or not'—the release is going out on Friday because we have to meet that deadline.
00:13:04.890 So we often make short-term decisions that lead to future pain, like writing code without tests. The two big themes here are longevity and independent evolution, which are the true benefits we fail at today with our APIs. When you release a new version of your API, all the old stuff breaks, or you force people to upgrade. We face continual churn, and things don’t last long enough.
00:13:51.170 We could be building software on the scale of decades, as HTML has survived for about 30 years now. The World Wide Web is the largest information system we’ve built as humanity, which is crazy! And yet, we all know that our software is flawed. The scalability of the web is not just a meme; it is truly a challenge. The web, as an incomprehensible large entity, works.
00:14:38.130 To handle a decentralized web composed of countless entities that must cooperate, we need design choices that accommodate this complexity. There are two rules that define a hypermedia API. One of the problems with REST is its proponents often do not clearly explain what it means and why you should care. I made it simple: use HTTP properly (with an asterisk) and use hypermedia to guide clients through your business process.
00:15:31.920 The asterisk means technically, you don’t have to use HTTP; you can use any protocol you want. However, in reality, you should use HTTP. This is because the constraints that define REST in Roy Fielding's dissertation can be met simply by following web standards. If you use HTTP correctly, you check off most of the boxes, except for that sub-constraint on the uniform interface.
00:16:02.560 The second rule is to use hypermedia to guide clients through the business process. We will spend the rest of the talk discussing this last bullet point because it is the hard one. It's something you have to handle yourself, and it's not something that can be standardized easily. This is an application-level concern, not a protocol-level concern.
00:16:47.700 So, all it takes is using links and using HTTP correctly. Just to clarify, this does not mean using HTTP correctly. While it is a valid HTTP request, it may not be semantically valid. We can say that something should function in this manner under standards, but just because it executes correctly doesn't mean it adheres to the standard's intent.
00:17:37.760 So, when I discuss this, I'm talking about using HTTP properly. Now let’s discuss five steps to actually build an API in this manner. These are rough guidelines focused primarily on server-side stuff because this is RailsConf and we are building servers, not clients. I'll comment on client-side matters at the end.
00:18:04.190 The five steps involve evaluating your business processes, creating a state machine to represent them, using that state machine to evaluate media type needs, creating or using existing media types, and finally, implementing it. Notice there's nothing about URLs, objects, or routes, which are totally irrelevant.
00:18:29.420 Let’s look at an example. It's crucial to think about a concrete application when discussing abstract web standards. There's a project called Mendicant University, which helps intermediate Ruby developers become great Ruby developers. One of their students built a service called W3C Love, which wraps the W3C HTML validator. If you’ve ever used that before, you know it only allows you to input one webpage at a time, which is inconvenient.
00:19:06.490 To mitigate this issue, we wrote a gem called W3C Love that lets you submit a website and it crawls through your site, validating all the pages and giving you a report on whether they’ve passed validation. It’s super cool, and I have plans for integration tests to ensure page validity before deploying.
00:19:38.470 Now, if Jaime came to me and asked how to enhance this API, I would say it should look like a REST API with a GET request to a specific URL with a parameter containing the URI of the site you want to check. This would return a JSON response detailing the success of validation, the number of pages scanned, the errors found, etc. There's also a service to check individual pages, granting the user direct access to check specific URL functionality.
00:20:17.760 Consideration must be taken when evaluating the business processes involved with this web pages API. For effective API design, one should analyze current software practices. I think the great thing about test-driven development is that it forces me to confront the software I am building rather than experimenting arbitrarily.
00:20:57.280 For W3C Love, we now have two fundamental processes to handle: the web pages API and the site maps API. Individual pages versus whole sites. For this, we need to build a state machine that fits those two processes. I love using the dot language for creating state machines because it allows writing text files that turn into beautiful graphs.
00:22:58.670 This state machine needs to encompass the request process to retrieve data. The process will involve inputting the URL into a form, followed by processing that URL. The request form would allow users to input this URL with the endpoint processing the validation task. The goal is to represent this state machine accurately.
00:23:47.490 Even though this is a very simple process, we are neural relationships to maintain consistency going forward. We need to evaluate our media type needs, but unfortunately, basic JSON cannot drive a hypermedia API. JSON is great; however, it lacks hypermedia semantics. Both JSON and XML provide minimal context, merely demonstrating how to serialize some data.
00:24:38.390 So, what’s the standardized way to create a link in JSON? There is no binding agreement on that, so we can build older formats on top of JSON. While you aren't restricted to using JSON to drive your API, you should construct additional semantics for media that do not lose meaning.
00:25:28.320 There’s a tension between using types that already exist and crafting your own. Ideally, you should use an existing type, and there are fantastic hypermedia JSON types available, like Collection+JSON and HAL. If everyone makes their own type, we haven't gained anything. However, if we all start using one shared format, it would enhance our comprehensive understanding without sacrificing semantic meaning.
00:26:24.950 We're going to create a custom media type for demonstration purposes. Media type design is an art as well as a science; it's complex, and there’s no definitive right or wrong way to do it. It’s crucial for designers to consider how to structure the data being transferred and the controls that facilitate hypermedia's linking elements.
00:27:08.510 We would build a type called W3C Love Validation+JSON, with its media type designation looking like this: applications/vnd.w3c.love.validation+json. If you’re unfamiliar with media types, they break into two parts; the first is the application category, like HTML or application/json. The “+json” part implies that if the semantics are not understood, this still processes correctly as JSON, creating a robust standard.
00:27:54.950 This would lead us to define attributes for our JSON, specifying the scraping address, success flag, timestamps, or error counts. The media type’s definition must encompass the necessary details and provide optimal functionality when implemented. Building an accurate response emphasizes the need to illustrate hypermedia controls, such as links and forms that blend representations. A response can have links that guide users through each process.
00:29:24.660 The response must possess clear forms that facilitate user input. For example, when retrieving a sitemap link, the provided information must maintain relevance and context for the user. This can clarify how to interact with the offered services while remaining streamlined and coherent.
00:30:12.340 Let's follow through the logic of how an interface might interact with a web service. The API response indicates different actions. For example, we utilize the first response to prepare for why users might interact with a particular service. Another characteristic we want our clients to monitor is what the server responds with and how those interactions occur.
00:31:08.200 Now, when illustrating server responses, the client will need to extract a valid user input form. This form needs the action attribute that tells it where to send the request. After submitting and getting a new response, the new order of provided data must populate accordingly. This ensures that the client can reflect real-time changes in the interface once the links and forms are processed accurately.
00:32:01.920 To recap, on the server side of processes, evaluate your current business processes, create a clear and distinct state machine for your current work processes, and construct a solid media type that meets those needs. This understanding is essential for defining how your service functions, allowing for further enhancements over API advancements.
00:32:56.760 This talk is intended to familiarize you with hypermedia principles and how to apply these semantics effectively within API designs. This is a broad, intricate topic; I dedicated a significant amount of time delving into literature and case studies surrounding hypermedia APIs. It's a multifaceted and revealing subject.
00:33:29.930 I'm writing a book—well, a website called 'Designing Hypermedia APIs' where I share regular updates and insights. It's designed to be a resource for those curious to learn more about hypermedia principles on a deeper level. I aim to deliver consistent information rather than allow the writing process to take forever.
00:33:57.830 I’m going to conclude the slides here, but I still have time to engage in a discussion on client aspects and showcase some code for the client side. My handle on Twitter connects as well as my personal website. I encourage everyone to explore resources from Jumpstart Lab, as a lot of our curriculum is freely accessible online.
00:34:35.900 We maintain our instructional content through an open-source platform, inviting contributions and updates. If you want to engage, feel free to submit your edits or enhancements, and I will gladly merge those into our repository. It reflects the collaborative spirit we value in learning.
00:34:53.370 That's everything from me today. Thank you for your attendance and enjoy the rest of your RailsConf!
Edit
Explore all talks recorded at RailsConf 2012
+61