Designing Hypermedia APIs

Talks

Steve Klabnik

#software-architecture

#web-development

#decoupling

Designing Hypermedia APIs

by Steve Klabnik

In the video titled "Designing Hypermedia APIs," Steve Klabnik discusses the design principles of hypermedia APIs, contrasting them with traditional RESTful APIs, particularly as implemented in Ruby on Rails. This talk, recorded at the wroc_love.rb 2012 event, addresses the conceptual mismatches between applications, APIs, and clients, advocating for a design approach that fully leverages hypermedia capabilities. Key points from his presentation include:

Understanding REST vs. Hypermedia: Klabnik clarifies that while Rails facilitates a form of REST, true REST should be viewed as hypermedia APIs, which provide more extensive decoupling and adaptation possibilities for clients over time.
Principles of Hypermedia APIs: He articulates that hypermedia APIs are flexible, allowing for server-side changes without client-side updates. This flexibility remarkably mirrors the operation of the World Wide Web.
Scalability Considerations: Hypermedia APIs can scale effectively, but they may not always be optimal for scenarios that demand low latency. Klabnik emphasizes the challenges associated with caching in these environments.
Key Design Steps: He introduces a five-step process for designing hypermedia APIs, which includes evaluating business processes, creating state machines, identifying media types appropriate for the APIs, and proper implementation of those media types to facilitate effective client-server interactions.
Media Types and Hypermedia: The presentation emphasizes the importance of media types, which form the crucial contract between server and client. Klabnik advocates the addition of hypermedia semantics to formats like JSON to enhance their utility in APIs.
Versioning Views: He critiques traditional versioning practices, arguing they often introduce unnecessary coupling and complexity and presents an alternative approach through flexible hypermedia APIs that maintain backwards compatibility.
Case Studies: Klabnik references large-scale systems like Google and the operational principles of the World Wide Web as foundational models for understanding the potential of hypermedia approach.

In conclusion, Klabnik's talk highlights the transformative potential of embracing hypermedia APIs for web applications and invites the audience to engage with his ongoing work and a public mailing list dedicated to discussions on hypermedia. This approach not only enhances API design but also promotes the evolution of client applications without disruption.

00:00:12.639 Hi everybody, I'm Steve. Thanks for having me here! I've always wanted to come to Poland, so this is really exciting. I’m mostly Slovak ethnically, and I have been meaning to visit this part of the world for a really long time. I'm really happy to be here.

00:00:29.679 So I’m here to talk to you about APIs today, specifically how to design hypermedia APIs. If you're not familiar with the work that I do, I hack on Hacky Hack, which is the easiest way to learn programming; it teaches programming through Ruby. I also teach classes with Jumpstart Labs, where we offer the best Ruby and Rails classes in the world. If anyone needs any training, feel free to talk to me about that.

00:00:41.360 This talk is about the journey I've been on over the last eight months of my life. There's a sort of traditional tale of American Indians where someone wanders off into the desert on a spirit journey and comes back all crazed, claiming to have seen the light. I tend to do that when I want to learn new things; I keep a list of topics I want to explore. Whenever I feel bored or ready to learn something new, I pull something from that list and investigate.

00:01:21.119 This often happens when I hear people mention that 'Rails doesn’t do real REST.' I found this interesting, so I filed it away to look into later. Eventually, I decided to dig deeper, and I discovered that if you read people’s doctoral dissertations as opposed to random blog posts, you sometimes learn that we are, in fact, doing things the wrong way. So, this is my opportunity to share with you what I've learned about REST and how to build RESTful APIs.

00:02:05.399 You may have seen my recent blog post on this topic. The title of the talk is 'Hypermedia APIs,' but I mentioned REST. If you’ve ever seen Portlandia, you might understand why I might never move to Portland because I’ll end up being one of those people. REST, as it’s commonly understood, seems a bit outdated. I’m not here to say that you're doing REST wrong; rather, let’s leave that notion aside.

00:02:29.879 For various reasons, some people in the REST community have started referring to truly RESTful APIs as hypermedia APIs. When I say REST during this talk, I'm referring to Rails REST, and when I mention hypermedia APIs, I'm talking about real REST. The terminology can be a bit strange at first, but as this concept gains traction, it will become clearer.

00:02:54.120 For example, O'Reilly has published a book titled 'Building Hypermedia APIs with HTML5 and Node,' and I think it’s fantastic. If O'Reilly publishes a book with a term in the title, I feel comfortable adopting that term. So let's move forward with the idea of real REST as hypermedia APIs.

00:03:21.640 Now, let’s discuss a bit about the background of what a hypermedia API actually is. Simplistically, hypermedia APIs are designed to scale better. They can be changed more easily, and they promote decoupling and encapsulation with all the regular benefits that these concepts bring. This type of design is the ultimate general decoupled API.

00:03:42.320 One of the primary metrics of software quality is your ability to withstand change over time. If you build your APIs this way, you can make modifications to how your API works on the server without needing to push new versions of the client. This might sound crazy, but consider how the World Wide Web operates—it’s mostly descriptive rather than prescriptive.

00:04:11.720 Think about when was the last time you had to upgrade your browser because Google deployed something new? You don’t. This is because a browser is a general API client that can work across multiple APIs, which are websites. If we can build our APIs similarly, we could enact significant changes without requiring a client-side update.

00:04:37.760 Take Twitter, for example: both 'new Twitter' and 'new new Twitter' made significant changes to how Twitter works. You didn’t need to upgrade your browser to access those changes; your interface simply adapted. This decoupling leads to substantial benefits, allowing APIs to evolve without breaking client applications.

00:05:01.679 However, hypermedia APIs are not perfect. They’re not always the best choice when it comes to latency. If you require sub-10 millisecond response times, this kind of API might not be for you. Also, caching is one primary way these APIs scale, and managing caching is notoriously difficult.

00:05:27.960 There’s a saying in computer science: there are two hard things: naming things and caching. Caching and validation can be incredibly challenging, which is essential to master when constructing an API this way. Additionally, because hypermedia messages are text-based, they will never be as efficient on a per-message basis as a binary protocol.

00:06:13.159 However, most small messages can fit within a single TCP packet, so the efficiency can be a perceived issue based on context. Comparing binary protocols like BSON or MongoDB to HTTP can reveal stronger compression and efficiency, but this may not apply universally. Thus, while hypermedia APIs offer numerous advantages, they may not suit every situation.

00:06:38.559 After all, the largest computer system in the world operates under these principles, and there is a kind of reverence in tech circles for the effectiveness of the World Wide Web. This praise is justified; large systems, such as Google's, can teach us valuable lessons about scaling.

00:06:50.320 Google, for instance, accounted for 6% of the world’s web traffic in 2010. This means that the systems built on the principles of the Internet are two orders of magnitude larger than Google's architecture, presenting substantial scaling opportunities and advantages. When discussing API scaling, considering the largest information system— the World Wide Web—is crucial before criticizing its structure.

00:08:05.840 I define hypermedia APIs as any APIs that fulfill two key criteria. The first is to use HTTP properly, with a little asterisk to denote that this is somewhat of a compromise; you’re not limited to HTTP but realistically, no one uses anything else. The second criterion is to effectively use hypermedia to guide clients through business processes. That’s the tricky part.

00:08:31.040 The reason Rails doesn’t conform to this by default is that while it does a good job using HTTP, hypermedia is not widely understood or applied by those creating APIs. There is also an ironic situation where, if everyone used HTTP when building these types of APIs, great potential would be unlocked. Roy Fielding’s doctoral thesis outlines specific constraints that must be adhered to in order to properly employ HTTP.

00:09:13.600 These elements are what your API must follow on a protocol level to be effective. The details may be a bit out of scope for now. Using HTTP correctly grants you nearly all the advantages needed in hypermedia APIs, with the exception of one critical constraint: 'hypermedia as the engine of application state.' You may have heard this referred to as HATEOAS.

00:09:42.040 This is why the focus on hypermedia APIs is essential; you need to utilize hypermedia for maximum flexibility and decoupling. In the past, people may have done things like sending a POST request to delete a photo, which was clearly incorrect. This approach demonstrates the importance of utilizing the correct HTTP methods.

00:10:05.360 Yesterday, there was a notion discussed about how the web is shifting toward thick JavaScript clients. I jokingly thought to myself that perhaps we should just eliminate servers altogether and move everything onto the client! However, we must remember that although we are on the cutting edge of our craft, many programmers out there do not pursue continuous learning or best practices.

00:10:41.560 There is a well-known incident involving Google developing a web accelerator. If everyone adhered properly to HTTP principles, their tool would have been incredibly useful. Essentially, when a user visits a web page, the accelerator could pre-fetch all the links on that page to facilitate instant browsing. However, due to poor implementations, people encountered issues such as accidentally deleting their Flickr account photos.

00:11:22.080 This example emphasizes the potential benefits of maintaining good HTTP practices. However, I acknowledge that discussions around hypermedia APIs may seem academic, influenced by doctoral theses. Despite that, the real-world benefits cannot be ignored.

00:11:54.600 Now, let's look at the main point of my talk: how to build an API in this fashion. It involves five steps that differ from traditional API design. The first step is to evaluate your business processes. The second is to create a state machine representing those processes. The third is to evaluate the needs for your media type.

00:12:49.040 The fourth step is to create or identify a media type that already exists and addresses those needs. Finally, the fifth step is the implementation process. Media types play a pivotal role in this design methodology.

00:13:15.960 When you conceptualize a website, think of a user browsing it. This interaction mimics a state machine where each link from the homepage represents a transition, and every new page signifies a new state. You can represent your entire website as a state machine comprising these links.

00:13:57.679 A significant aspect of user experience design involves minimizing steps in that state machine. For example, if your homepage requires a user to click a link to start interacting, you’ve effectively added an additional step. Research conducted by Amazon and Google illustrates how reducing the number of steps in a signup process can significantly lessen user drop-off.

00:14:55.480 That's why state machines are crucial, as REST represents a state transfer model. The 'state' terminology is embedded in the name itself. Just as important are the media types; they form the contract between server and client. One reason you don’t need to update your web browser when new versions of a website are released is due to the clear contract established by the protocol—HTTP—and the relevant media types.

00:15:33.920 You can create clients that work similarly by ensuring that your media types are adequately designed. For instance, a friend of mine, Jamie Anasta, created a website called W3C Love, which validates HTML. Jamie initially built an API in the Rails style and asked for my insights on improving it. As a result, we are in the process of transitioning his API to better align with hypermedia principles.

00:16:09.840 Currently, when you send a GET request to validate a website, you would receive a JSON response containing all relevant data. However, this design introduces coupling; if the URL changes in any way, existing clients break. Clients must know about specific URLs on the server, which creates tight coupling.

00:16:45.440 This problem persists across different functionalities within the API. Consider the process for checking a single web page versus an entire site, which often relies on a structured dependence on the URLs. Therefore, moving towards a hypermedia API model addresses this by allowing flexibility in client-server interactions.

00:17:16.720 So, what we need to do is provide the ability to check both web pages and entire sites. This means creating a state machine to encompass these business processes. The state machine starts at the root, where you request a form for the web page API. After processing the form, you receive all necessary information with the option to link back to the root.

00:17:56.760 For the site maps, you would similarly request a form to begin. This process can be expanded and modified based on requirements. Each step represents distinct operations, reinforcing the concept that hypermedia APIs should encapsulate these state machine ideas.

00:18:38.360 The form steps are crucial; they allow you to tell the API what website to check, thereby facilitating clear templating links. Building these templates is an essential aspect of decoupling the API effectively.

00:19:01.360 Delving deeper into media types, it’s essential to note that JSON cannot solely drive your API without hypermedia semantics. While you can use JSON as a serialization format, the real strength lies in adding hypermedia principles to it. Unfortunately, JSON doesn’t provide a canonical way to embed links, which is essential in hypermedia.

00:19:45.720 Although many preferred XML for hypermedia applications, JSON can also be valid. You can add hypermedia semantics, like collection+json, to enhance JSON's capabilities for your APIs. It's unnecessary to create a new format; you can modify existing structures to fit your needs.

00:20:23.920 So, I aimed to design a media type called 'w3c love.validation+json'. This media type includes a vendor prefix, allowing for transitional deployment before standardization. The 'plus json' indicates that while the media type is specific, it still adheres to a valid JSON format.

00:21:12.960 I specified the data elements to be included, which forms a standard structure while also defining hypermedia controls such as links and forms. These elements are crucial as they generate connections among various representations, facilitating user navigation.

00:21:54.840 Furthermore, I included a 'rel' attribute in the media type to describe the purpose of each link. This setup would allow users to obtain forms linking to the relevant resources. The actual implementation of the media type ultimately integrates seamlessly with both servers and clients.

00:22:34.720 In conclusion, with the proper implementation, clients would interact using this media type and receive responses dictated by the client-server contract. While the details of this process deserve comprehensive exploration, my aim was to present a broad overview of these approaches.

00:23:09.839 Summarizing the five steps again, the first is to evaluate your business processes, the second is to create a state machine that accurately reflects those processes, the third step involves identifying your hypermedia needs, and the fourth step requires either creating a new media type or using an existing one. Finally, implementation is key to completing this process.

00:24:04.800 Currently, I am writing a book on this topic. After deciding to tackle this in a lean fashion, I gathered feedback on whether there was interest, and the response was overwhelming! Therefore, I've spent time reading extensively on the topic, collecting insights from various sources, and talking to others.

00:24:52.120 I just released the alpha version of my book titled 'Designing Hypermedia APIs.' It offers insights into structuring hypermedia effectively and currently contains fourteen articles that serve as a starting point. The intention is to expand this considerably, aiming for three to four times the current length.

00:25:22.079 If you're interested in learning more, feel free to approach me at any time, or send me emails. Additionally, I’ve created a public mailing list at hypermedia-liberal-list.com, where discussions about hypermedia can take place outside the nuance of this book.

00:25:50.720 To wrap up, I'm on social media as @SteveKlabnik and my website is steveklabnik.com. For more information on my book, visit designinghypermediaapis.com. I also wanted to highlight the open-source work from Jumpstart, where we provide free resources for learning Ruby and Rails. Tutorials can be found at tutorials.jumpstart.com, with open-source documentation readily available on GitHub.

00:26:19.440 Thank you for listening. We have time for maybe one or two questions regarding the discussion on versioning, which some literature suggests placing in URLs. I wanted to touch on that aspect briefly.

00:26:45.759 My previous stance on versioning has changed dramatically. I initially believed it to be a good practice, but I now view it as an anti-pattern. Often, when encountering a need for managing changes in applications, the response is to version everything. However, this can create unnecessary coupling and complexity in the API.

00:27:38.560 Managing change can be handled flexibly through hypermedia APIs, which allow for extensibility. By adopting this approach, clients can effectively adapt and ignore deprecated fields when upgrading. Each media type should be designed to maintain that flexibility.

00:28:00.960 Using the example of HTML5, you can see how versioning has not played a significant role; for more than a decade, it has been about iterating enhancements instead of solely versioning. The views on versioning, especially in the context of hypermedia APIs, will be something I delve into further in my writing. But that’s the essential overview.

00:29:03.520 When discussing backwards compatibility and allowing clients to fallback to earlier media types, flexibility is crucial. Clients can still reference deprecated attributes while implementing new ones, ensuring both old and new clients can function simultaneously. We could talk in detail about that for quite some time.

00:29:37.360 Thank you very much for your attention today! I'm open to continuing this dialogue, should any further questions arise.

Steve Klabnik

@steveklabnik

wroclove.rb 2012