Principles of Awesome APIs and How to Build Them

Ruby

Keavy McMinn

Principles of Awesome APIs and How to Build Them

by Keavy McMinn

In her talk titled Principles of Awesome APIs and How to Build Them, presented at RubyConf 2019, Keavy McMinn emphasizes the importance of creating consistent, stable, and well-documented APIs. Through an engaging narrative that parallels her experiences training for Ironman triathlons, she illustrates how embracing constraints leads to better API development. The following key points were discussed during the presentation:

Understanding API Challenges: Developers often face difficulties in maintaining APIs due to the nature of their use by both humans and machines, necessitating a conservative approach to changes.
Embracing Constraints: API developers can either fight against constraints or work with them. Accepting the reality of API constraints allows for smoother development and increased user trust.
Consistency: It is critical to maintain uniformity in how data is represented across different endpoints. McMinn advocates for employing tools like JSON Schema as a source of truth to validate and document API responses, ensuring consistency and preventing discrepancies.
Stability: Writing clear documentation and defining the purpose and structure of APIs is essential before diving into coding. Stability requires thinking about how changes affect users and minimizing negative impacts through transitions and versioning strategies.
Documentation: Comprehensive documentation, including a change log and deprecation policy, is vital for user confidence. Informing users about changes and providing guidelines ensures that they are prepared for any updates.
Ownership and Governance: Clear ownership of API components and documentation of responsibilities are necessary to maintain quality and manage changes efficiently.

McMinn concludes by stressing that successful API development should not be hard; it simply needs to be consistent. Her insights focus on the philosophy that understanding and planning for constraints creates a pathway for building exceptional APIs, encouraging developers to utilize these principles practically and thoughtfully in their work.

00:00:13.880 Several years ago, when I was racing Ironman triathlons, I spent some time training on the island of Lanzarote. It's a place many triathletes love to hit; you can ride for hours without seeing a single car. However, the weather and landscape can be brutal.

00:00:21.529 When I first rode there, I had to stop a few minutes into the ride because I couldn't believe how strong the wind was. I thought I was going to get blown clean off my bike, but I'm an idiot, and I decided to train for the Ironman race there. I'm kind of small, and on a triathlon bike, the wind can really push me around.

00:00:32.689 It was grueling battling the wind. Some days, the headwind was so strong that I had to work really hard just to inch forward at all, and I finished fast ascents in such a state of shock just to still be upright that my legs trembled for minutes. I remember at the end of my first year of training, I was tired, teary, and told my friends that this course and its weather were too tough for me. I said, 'I don't think I can do it.' A more experienced friend offered me some great advice; she said, 'Try to work with the wind. Don't fight it all the time, otherwise you'll just exhaust yourself.' Initially, I wasn't sure how to physically do that when I had to go in a certain direction.

00:01:04.049 But through practice, I got better at sensing what the wind was doing, assessing the risk, and learning how to work with my bike and body weight. I gradually learned how to judge the crosswinds, riding at a 30-degree angle, literally leaning into the wind. I shifted my mental attitude; I stopped trying to ignore or fight it, and I developed a respect for the wind. I learned that accepting and embracing the wind made all the difference.

00:01:58.290 A good API needs to be consistent, stable, and well-documented. We know this as consumers ourselves; these are the qualities we want from APIs that we have to use. We know how annoying a poor API experience can be. We might be totally dependent on an API for our product, but yet we don't have any control over it. We have to react to it, perhaps sometimes scrambling to get our tooling updated just in time with producers' changes. That's when we even know about the change in advance. We might use a handful of APIs to help with smaller tasks. Maybe someone set up a cron job that runs a script, connects to an API, and produces a monthly report from the data it fetches.

00:02:42.390 We're not actively monitoring it because it's just a little script somewhere, and it works just fine like that for years. Then, the department that reads that monthly report starts to notice that the numbers look a bit off. So, you dig and eventually discover that the API is not returning numbers in the same way. Now, half the data in that report is just wrong or missing. If an API change is handled poorly, it can be really painful for us as consumers. From our experience as producers, there's a certain type of developer who doesn't like things to be wrong. You may know this type or be one yourself; the type of developer who writes scripts to catch inconsistencies in your codebase, refactors the whole test suite to ensure every scenario is covered, and gives the most thoughtful and thorough code reviews.

00:03:42.870 This is not me, by the way, but this is someone you want on your API team. However, the struggle for them is that sometimes, when things are wrong with an API, they have to stay wrong. Sometimes you can't fix things because your users are accustomed to the wrong behavior, so fixing it would actually break it, and that can be really hard for some developers to accept. The ability to change the API or not, as is often the case, is probably the biggest pain point we feel as API developers. I want to pause for a moment to remind ourselves what an API is and why that ability to change is so hard for many use cases.

00:04:52.080 An API is the interface between our product and its users. It's interacted with via code. When we're developing software for humans to use, if we change how it looks or works slightly, the humans might get confused, but they'll eventually figure it out. When it's computers on the other side, however, that’s not going to fly; code can't just figure it out. Our users rely on our APIs to interact in real time with their code. By the nature of that territory, it needs to be a very conservative space for development.

00:06:03.630 When we’re developing APIs, we're bound by many constraints centered around the communication and expectations between both sides of that interface. That's the nature of developing APIs, similar to riding a bike in the wind; we're developing in a landscape of constant constraints. If we want to produce an awesome API, which we know means being consistent, stable, and well-documented, we need to work within those constraints. Trying to fight the reality of APIs ensures the nature of that interface is doomed to failure for the product and misery for us as developers.

00:07:08.160 From other areas of development, we might be used to imposing our will on the product, so it might be frustrating for us to not be able to make changes when we want. But that's the nature of building APIs. We can either be miserable fighting those constraints or we can embrace them. One thing that I think is useful for us to remember is that these constraints are good for us too. If you're writing an API for yourself, for a hobby project where you're both the producer and the consumer, you can do whatever you want whenever you want. But this talk is aimed at those of us building APIs for other people to use.

00:08:03.530 You want people to use your product, and the API might be the foundation of your entire service. It might enable your product to grow and evolve through the creativity and power of all the tools that build on top of it. The ability to build an ecosystem around your product through the API is hugely powerful for a business; it increases the value exponentially. You're brokering relationships between parties that might otherwise not connect, but each time you break the API, that’s one less tool that can work with your service.

00:08:57.510 So these constraints, when respected, offer protection for both sides. They enable our users to lend their trust to build their software on top of ours, and they also enable us to grow and maintain a wider user base, which in turn enables our success. Here are some ways that I've found useful for embracing and working with those constraints to help achieve those main principles of being consistent, stable, and well-documented. First, to be consistent, we need to be rigorously consistent in how we represent our data.

00:09:45.630 For example, if a user has a property of 'admin' that is returned as a boolean in one endpoint, then it shouldn't be returned as a string or an integer in another. You might think, well, of course we wouldn't do that, and no one would intentionally, but as an API codebase or team grows, these inconsistencies can slip through to the public interface quite easily. This is especially true when there are multiple apps that make up a public API. To help avoid these mistakes, we can use tooling to help maintain the consistency of our API data. There are a lot of choices for this; these are just a few.

00:10:45.680 These specifications provide a specific way to describe what objects exist in your API, what they look like, and how people can work with them. My preferred option is JSON Schema because it's relatively simple, yet flexible and powerful. For example, you can use it to validate user input as well as your output. In this conservative nature of API development, it's usually best to approach any shiny new thing with caution, so it's reassuring to know that JSON Schema has been around for years and is already tested in production by several APIs that many of us know and use, like Heroku, GitHub, and soon Fastly.

00:11:40.690 A schema can be the one source of truth for your API. Knowing your source for data is accurate means you can confidently use it to generate documentation examples, test the representation of objects in your requests and responses, and validate user input. I'm going to talk specifically about JSON Schema, but you might prefer to choose a different tool. The important thing is that you use something as your one source of truth, selecting the tool that best suits your context, your organization, and your codebase.

00:12:15.350 There are some particularly handy tools for working with JSON Schema and Ruby. The Committee gem enables you to perform validation of user requests against your schema, which makes it really easy to provide users with immediate feedback when they supply a parameter that's the wrong type or not quite formatted correctly. You can perform that validation centrally before a request even hits the specific endpoint code.

00:12:44.450 The PRM DJM enables you to combine and verify multiple schema files, which is super useful once you're beyond needing to describe one or two objects. It's much easier to manage those in smaller chunks of JSON at a time, so you might have one schema file for users and another for teams, for example. You can find schemas for some APIs online. For instance, Heroku has an endpoint to read the schema for their API, which is a super useful way to learn through real-life examples of seeing the constraints that someone else applies to their data as a complement to the examples you'd find in the docs.

00:13:36.550 If you're thinking this all sounds fantastic, let's go back to work and make a schema to enforce consistency. It'll be amazing! But here's a word of warning: building a schema file like this is ideal if you have a greenfield project or just have the luxury of a fresh start. If you have an existing API, you probably can't apply a schema in all those ways immediately because you likely have a whole bunch of inconsistencies in your API that you may not be aware of yet. But if it were left up to humans at this point, they probably do exist, and that's okay; everyone has skeletons in their API closet.

00:14:49.800 Where you don't have a schema, a safer starting point would be to first get a measure of inconsistencies. You can take a close look at your documented responses, running comparisons between response examples in your docs and the responses you get when you actually make a call to the API. It might take a bit of time to set up a test account with sample data, and you might need to parse the examples you give in your documentation. But then, you can write Ruby scripts to make real requests to your APIs for each of those and compare the results against the documentation.

00:15:48.680 You could note where and how the responses differ—are there any additional or missing keys in the hashes? Are any of the attributes classified differently from what you specified? Are there any boolean values that are actually strings? I did this recently, and even the exercise of reading your docs and forming all those requests is beneficial. It's a great way to familiarize yourself with the public interface and see things from the consumer's perspective, building up that empathy. You could create a schema for a small portion of the API; say, in your app, you have a user object and some endpoints for users. You could try writing a schema just for those.

00:16:52.360 Then, run your tests for the user endpoints against that schema and see how many of your tests fail and why they fail. You can even do that in production and log where the mismatches are. If the output is what you really want, then you can adjust the schema to reflect that. But if the schema was correct and the output doesn't match it, then you can't change that output immediately because that's the output your users are used to. Breaking it to fix it can be really challenging.

00:17:53.360 It can be interesting and pretty effective to get a better understanding of the inconsistencies by taking a deep look into the code—using code to analyze code. I've used the white court parser for this job, which is a Ruby gem that allows you to parse your code and form abstract syntax trees (ASTs). As the tree name suggests, it gives you an object that you can then crawl around to find the branches and leaves that you're most interested in.

00:18:34.030 This doesn't need to be pretty or polished; my own scripts that do this are definitely not. This type of tooling isn't going into production, so it's okay for it to be dirty. The goal is to help you explore the code and to answer the things that you're curious about. For example, you could use a parser to peer into endpoint code and see what's being called, like monitoring and measuring the calls around authentication. You can note what methods are being used, what arguments are passed, and what classes the arguments are.

00:19:25.860 Parsing the code with code is an immensely powerful technique. Exploring it at all can yield really useful insights, helping you make decisions about inconsistencies. Maybe you've done a thing three different ways, but now you can pick one way that you want to deliberately set as the standard for the future. I've recently done this for the Fastly public API, where the API is formed across multiple apps, providing us insights and measures of a ton of useful information that would otherwise be pretty difficult to gather.

00:20:19.920 As a bonus, looking for and measuring these things can also set you up nicely to have a custom linter that helps you monitor your code in the future. Getting to consistent data might be a methodical process that requires painstaking work, but it will be worth it. The second big principle of a good API is stability.

00:21:12.870 If you do one thing, I recommend it simply to think deeply and write down what the API is for and what each endpoint is for. This seems so simple that many people don’t do it, preferring instead to dig into the more technical work of writing code or specs. But even with API-specific specs, I think you can get distracted by the syntax and the opinions of the tooling, and all that can come later. As a starting point, I believe it's way more beneficial to write down just in plain language what the API is for.

00:22:15.760 That seemingly simple task is actually really hard to do, but it's faintly important. In a high-level design for an API, I recommend including usage examples for any new endpoints. What would users do with it? What variations might they try? What workflows would it need to support? What does your business need from it? What are the potential performance issues? What would happen if there were 10,000 of the objects that could potentially be returned? What does the path look like? Are there any parameters in the path? What do they look like, and what will you call it?

00:23:11.520 Keeping in mind that if you can think of a neat title for an endpoint, that’s probably a sign that your design is a bit off. Remember, there's a cost to adding new endpoints—more code to write, tests to maintain, documentation to draft, and code reviews to conduct. So you want to consider, do we really need this addition? Do you really want to walk that dog every morning when you're tired and there's a blizzard outside? Write those high-level design docs before you write any code, before you've invested in a particular approach. While the cost of change is low, the more thought and care you put into the API design upfront, the more likely you will ship the right thing the first time and reduce the risk of needing to make changes later.

00:24:08.840 We need the API to be a calm, predictable, and reliable space, but that interface is surrounded by things that are not calm. Everything is changing: our product, the technology choices we make to build and document our API, our users' needs and workflows, our assumptions about those needs, our legal obligations, security vulnerabilities, bugs in the code, and the priorities of our teammates and company leadership. The API is at the center of a constant state of flux, and on top of that, our API also has to change in order to stay relevant to that wider context.

00:25:13.440 So, while the stable center has to change with everything else, that change has to be very controlled, if indeed it's even possible. We experience constant friction between the need to change and our constraints as developers or shepherds of the API. It's good for us to be aware of that friction and the demands for change, but we should not react to all of those demands all the time.

00:26:00.750 We need to optimize for walking the API along a very stable, thoughtful, and compassionate path through all of that. Sometimes, that will mean pushing back against some of those forces that might disturb the calm center; it might involve asking a lot of questions, helping to find creative ways to absorb the problem on your side rather than imposing it on the consumer, or sometimes just saying no or not yet. Shepherding APIs often doesn't make you the most popular person in the engineering organization.

00:27:02.030 Some literature on API development will tell you that change is bad, and therefore you should simply never make breaking changes to an API. I don't buy that approach; to me, it's not working within the constraints, it's more like punting on them. Don't get me wrong; breaking changes should be very rare and a last resort, but in reality, the need will arise. We need to be able to evolve our API along with our product in response to situations like major availability or security problems.

00:28:07.130 If you really think about it, change itself is not necessarily bad; it’s the negative impact from that change that is bad. So the goal is to minimize the negative impact. A great way to help with that is to provide transitions: a transition workflow or a way for users to adapt to the new thing. This can take different forms. For example, the GitHub API releases new or changed endpoints under a preview, which is a specific accept header that users can opt-in to use.

00:28:51.260 In versioning, for example, the Stripe API offers very granular versions and commits to supporting all previous versions, which must be a huge undertaking behind the scenes. With either of those approaches, you just want to ensure that users can use multiple versions—say one on staging as they try it out and prepare for the new behavior—while still maintaining a different version in production. Another simple way to provide a transition: for example, if you really need to change the name of an attribute, you could offer both the old and the new names for a transition period before you remove the old one.

00:29:51.700 Regardless of the form you choose—or combination thereof—the key is to provide a way for users to transition to the new thing gracefully. But accept that the reality is that we may not be able to reach everyone, and some users might not take any action. It's highly unlikely to achieve zero negative impact; the only way to get to zero would be if you had zero users, in which case, cool! The goal is to minimize the negative impact as much as you can reasonably do.

00:30:49.500 Lastly, an awesome API needs to be well-documented. Provide clear information to users on how you intend to change or break the API through things like documenting your deprecation policy. I recommend providing generous, conservative time frames for expectations around any kind of preview or beta periods, for which I suggest short timeframes; otherwise, people forget it's not really the final version, and it’s not really suitable for production.

00:31:55.620 Also, provide a change log with specific information on all the changes that developers can adopt to use. Being explicit and transparent with these things gives users some confidence in the process and builds trust, which is a crucial ingredient to sustaining an awesome API. Keep in mind that people don’t read, so even if you've given all the notices and warnings, news of upcoming changes won’t reach all your users. Even with the best will in the world, even with great monitoring and usage metrics, some users might not be reachable—perhaps the people who wrote the code using your API are no longer the ones maintaining it.

00:32:48.610 The first time some people might notice a change is when the things that depended on it stop working. To help compensate, you can implement a practice where, after all the regular, timely notices, you schedule a deploy with the change temporarily, maybe just for an hour. This allows enough time for people to prepare for the upcoming change on their side before you make the switch for real.

00:33:30.580 There’s so much detail to get right on an endpoint-by-endpoint basis that it’s easy to overlook the overall picture. However, providing a way for some people to see a holistic view of the API can be highly beneficial—especially within your company. Consider implementing scripts that capture key data points, which could be recorded in a spreadsheet to provide your API or product team with useful insights. This type of work doesn't have to be pretty or polished; it simply enables you to take a holistic view of that data whenever needed.

00:34:25.030 For example, you might capture notes on things like which endpoints are undocumented, which are in some state of non-public accessibility, or anything earmarked for deprecation and when it was due to be removed altogether. To collectively build and maintain consistency and stability, you need to govern choices and decisions. Writing down internal guidelines on how your company builds APIs will make it easier for current and future teammates to create a consistent, stable API.

00:35:05.000 Ensure you have a clear set of guidelines on how to make changes; any ambiguity regarding how to make changes can be a significant source of stress and frustration for engineering teams. Set clear expectations and share guidance internally on what's okay or not okay, and how to go about making changes. An internal API style guide is extremely useful—if it's a RESTful API, for example, you might include what patterns for paths are acceptable or which URLs with a verb or adjective in them are considered a bad practice.

00:35:57.000 Of course, there will be exceptions to any rule. One example is from the GitHub API, which has an endpoint to fetch the latest release that clearly has an adjective in the path. This was a conscious design decision because users almost always want just the latest release; it makes for a better interface. So, while it's important to use common sense and best practices, some choices will be controversial where there is a fairly clear good design and a bad design. However, most of the specific choices might not matter as much as making a choice and applying it consistently.

00:36:46.400 Your internal guidelines will always be guidelines, and just decisions will need to be created by humans weighing up the trade-offs. Lastly, in your internal tech documentation, I highly recommend recording who is responsible for what, in order to maintain an awesome API. Your group needs to have a clear understanding of ownership and the decision-making process. When that's ambiguous, it creates frustration and just wastes time and energy.

00:37:36.700 It also encourages poor quality code, as endpoints become orphaned if no one specifically owns them; no one fixes the flaky tests or bugs that arise because, well, it’s not their code. I recommend carefully deciding and documenting things like who is responsible for building APIs for new features, who is responsible for the underlying concepts, and who handles the plumbing, documentation, and other aspects of that work.

00:38:31.390 Everything needs an owner—a responsible party—and that information should be easily discoverable. Also, outline who gets a say in the decision-making process. Many people feel uncomfortable establishing a decision-making process up front, perhaps preferring to attain some form of consensus. If that’s the case in your group, I suggest it’s worth asking what happens when we can't reach a consensus, and maybe it’s worth writing that down as a safeguard.

00:39:34.170 That island of Lanzarote ultimately became one of my favorite places to train. I spent weeks there, getting to know the roads like the back of my hand. It improved me as an athlete—not just with technical skills but with my mental toughness. And while I loved riding my bike on a calm, sunny day, I now know I can adjust my mindset and embrace the day when conditions get tough.

00:40:29.900 I love working with APIs—not even despite the constraints that come with that territory, but because of them. I cherish creating something that helps other developers build upon what I develop; that’s a special privilege I take seriously. I hope you find something useful in the ways I’ve shared to work within constraints to build awesome APIs. The essential takeaway is that you shouldn’t fight your constraints. Understand what they are, plan for them, and work with them. And I want to leave you with this quote: 'It doesn’t have to be hard; it just has to be consistent.' This might refer to training for Ironman triathlons, but I think it sums up API development pretty well. Thank you.

Keavy McMinn

@keavy

RubyConf 2019