Advanced API design: how an awesome API can attract friends, make you rich, and change the world

Web Services

Jonathan Dahl

1 talk

Advanced API design: how an awesome API can attract friends, make you rich, and change the world

by Jonathan Dahl

In this presentation titled "Advanced API Design: How an Awesome API Can Attract Friends, Make You Rich, and Change the World," Jonathan Dahl discusses the intricacies of designing effective APIs, emphasizing their growing importance in modern software development. He begins by explaining the basic definition of an API (Application Programming Interface) and distinguishes between local and web service APIs, noting that REST is the preferred structure over SOAP for API development today.

Dahl shares personal experience as a co-founder of the startup Zencoder, which offers a high-performance video encoding API, highlighting the significance of an API-centric business model versus traditional product-based models. He categorizes APIs into four types:

Secondary APIs: These extend existing applications, like Flickr.
Infrastructure as an API: Services like Amazon Web Services provide foundational functions via APIs.
API-specific technologies: Solutions such as Twilio simplify complex tasks through APIs.
Sci-fi APIs: Innovative APIs enabling cloud processing, exemplified by PyCloud.

The core focus of the talk revolves around tips for designing a great API, including:
- Versioning: Introduce versioning early to avoid breaking changes as the API evolves.
- REST conventions: Follow common REST API design principles.
- HTTP response codes: Use appropriate codes for various outcomes, enhancing clarity for users.
- Smart validations: Provide specific feedback on errors to improve user experience.
- Documentation: Ensure detailed documentation, as it’s crucial for user engagement.
- Library support: Provide code libraries for popular languages to ease integration.
- API speed: Optimize performance to meet user expectations for responsiveness.
- Error handling: Implement effective error messages to aid troubleshooting and improve API usability.

Dahl concludes by stating that an exceptional API not only enhances developer satisfaction but can also lead to substantial business success, community building, and the potential to change industries. He reiterates the transformative effect of well-designed APIs, which can foster innovation and open up new opportunities for startups. In summary, Dahl advocates for the significance of effective API design as a pathway to technical recognition and commercial success.

00:00:30.000 Hey everyone, I’m John, and I’m going to talk about advanced API design—how an awesome API can attract friends, make you rich, and change the world.

00:00:35.360 Depending on who you are, you might be thinking this sounds kind of interesting—APIs are interesting. Others of you might be thinking this sounds absolutely boring. What could be more boring than an API? So if you're in the latter camp, I apologize in advance for the next 40 minutes. But I do want to try to convince you that APIs are interesting for a lot of reasons.

00:00:58.399 I’m only going to spend a couple of minutes today on the basics of what an API is and how they work. Then I kind of want to get into some more interesting things. So let’s try to do this in about two minutes.

00:01:15.840 An API, or Application Programming Interface, is a way for two applications to talk to each other—two bits of software to communicate. It’s similar to a user interface where a user interacts with software, but in this case, it’s software talking to software.

00:01:33.439 The first interesting thing is that they are both interfaces. There are a couple of kinds of APIs. You can refer to a library or an SDK as a local API; it’s an API within a specific system of software, perhaps within the same memory space between two bits of code. For example, SFH and Ruby act as APIs to Ruby functionality.

00:01:50.799 However, what I’m going to talk about today is not this kind of API; it’s web service APIs. A web service API allows two external systems to communicate with each other. Think of it as a universal API: you don’t need to be running the same language, the same operating system, or even be located on the same server.

00:02:07.960 Earlier web services were done with the Simple Object Access Protocol (SOAP), which was ironically named. SOAP has recently given way to REST as an API structure. I’m not going to talk too much about SOAP, but REST has become the preferred method for almost every API being written today. REST is based on the realization that HTTP itself serves as an API protocol. In fact, the web is an API—a communication channel between your web browser and a website.

00:02:21.920 Your web browser makes API requests, like 'GET records' to read information or 'POST a record' to write information. When you’re setting up an API, you need some way to structure the data to send information back and forth. The typical formats are either XML or JSON. XML is somewhat entrenched in many systems, while JSON essentially does the same thing in a more elegant way.

00:02:40.480 Both formats might need to be supported depending on user requirements. So, that’s exactly two minutes, and I did okay this time; practice makes perfect.

00:03:05.519 Now let’s talk about things that are a little more interesting. I want to start by sharing my autobiographical experience with APIs and then move on to some other topics. I’m a co-founder of a startup called Zencoder. We essentially provide an API for high-performance video encoding in the cloud, meaning our whole product revolves around our API.

00:03:22.840 We don’t really have a physical product that you would be inclined to pay for; it’s primarily an API. While we have a user interface, it is not particularly compelling; you wouldn't pay us just to access the UI. Likewise, we have backend processes happening behind the scenes—a sort of black box—where the exact mechanics of our process aren’t visible. All that matters is whether we respond properly to API requests.

00:03:47.879 For anyone here who has tried SendPut before, you might think we just have a GUI in the office downloading files and encoding them with Handbrake, right? But the reality is you don’t really care about that, as long as we fulfill our API obligations.

00:04:12.360 Let’s look at some categories of APIs. Let me know if this is cutting out. Can you hear me? Okay, let’s try this other mic. How does this sound? Can you hear me now? Good.

00:04:30.759 So let’s look at different categories of APIs. The first category I want to discuss is what I’ll call secondary APIs. These are APIs where the application can function without the API; the API simply extends the application’s capabilities. For instance, you can use Flickr independently, but it opened up significant functionality through an API that allows for more powerful use cases.

00:05:07.440 The second category of APIs is Infrastructure as an API. This is where things get truly interesting. Amazon Web Services (AWS) is an API for various functions, like file storage, Linux computing, messaging, etc. The next category relates to recent trends in software, which are API-specific technologies.

00:05:54.400 While you could perform the same functions without an API—such as purchasing a box, a server, or software—doing it through an API saves substantial time and investment. Twilio serves as an API for telecommunications; while you can set everything up separately, delivering it through an API is highly efficient.

00:06:43.000 Finally, I have what I refer to as sci-fi APIs. Has anyone used PyCloud? That’s an API for Python processing in the cloud. You can integrate PyCloud within your Python applications, and your Python code is executed in the cloud, enabling you to solve massively parallel problems.

00:07:05.959 For example, if you have a computationally intensive task, you can write and run the code locally, and then execute it in the cloud, which is incredibly powerful. Another interesting case is Amazon Turk and C-Flow, which are APIs allowing you to interact with people directly.

00:07:32.440 Now that gives you an idea about where APIs are currently headed. I want to spend most of my time today discussing how to create a good API. We put a lot of effort into API design at Zencoder, and we receive positive feedback regarding our API.

00:07:49.520 However, I won’t overemphasize Zencoder in this talk, so I will redact its mention when referring to specific examples. I really love good APIs; they make my life as a developer significantly easier.

00:08:03.999 Does anyone resonate with that sentiment? APIs can often be painful and challenging to work with, so designing them well is vital. But how do you design a good API? Who here has ever worked on an API? That’s about a third of the room; that’s cool! A large part of the rest of you will likely work on one in the coming years.

00:08:49.841 I’m going to share about 10 to 12 tips based on our experience to highlight how to build a good API. We may not implement all of them, but they represent what we should strive for. Additionally, I would love to hear your ideas that contribute to good API design.

00:09:33.160 First, you should version your API. It’s best to do this early in the API's life because retroactively adding versioning can be quite challenging. The reason to version your API is simple: if you have an API that lets you read a record, and you decide to change the output for a color property from a string representation to a hex value, then any applications relying on the original API behavior may break.

00:09:56.959 If you version your API early on, you could do something like this: 'API v1' for the string representation, and 'API v2' for the hex value. It’s important to note that you shouldn’t update your version too frequently; aim to maintain backward compatibility as much as possible.

00:10:21.760 Second, you should follow REST conventions. REST is the way most APIs are structured today; I’m sure there are exceptions, but it's a widely-used API language. Additionally, if you’ve ever built a Rails app, you’ve automatically created a REST API.

00:10:43.639 The Rails controller structure inherently facilitates how you'd want your API to function. The idea is to combine nouns with verbs: when you perform an action, there is something enacted upon, hence the verbs and the nouns. REST's conventions are elegant and mirror our natural thought processes—it’s about reading, writing, or deleting information.

00:11:06.480 Next, use appropriate HTTP response codes. While you’ve probably seen codes like 200 and 404, there are many other HTTP response codes to utilize. For instance, if you're creating records via POST requests, responding with a 201 status code (Created) provides more information than just a 200 (OK).

00:11:26.320 You can use 400 for client errors and 500 for server errors. It’s essential to provide accurate responses: for example, when receiving invalid JSON, a response of 400 Bad Request makes sense. If someone tries to execute a forbidden action, use 401 Unauthorized. Incorporating a 422 status code indicates an unprocessable entity that failed validation should your system need it.

00:12:12.640 Next, focus on smart validations. Imagine a scenario where someone posts an API key that isn’t valid; how should you respond? Rather than returning a 500 Server Error—which implies the fault lies on the server's end—you can use a 401 Unauthorized response for an invalid API key.

00:12:53.920 Providing specific feedback also helps: if an API key is invalid, return an array of error strings clarifying the issue, just like web forms in Rails. You can go further by validating the structure of the API request; for example, if it's missing a necessary field or includes spaces where they shouldn't, highlight that specifically.

00:13:11.360 Next is compatibility. If you want your API to support everyone, consider compatibility with both JSON and XML. Some users may not be familiar with JSON; they might think it’s a security error. Conversely, others may prefer JSON since it offers a more intuitive experience. Supporting both doesn't require excessive effort, and frameworks like Rails can automatically manage the parsing.

00:13:30.920 Next, ensure that your API is well documented. Integrating with a poorly documented API is nearly impossible. Therefore, it’s important to invest time into creating comprehensive documentation. Users truly appreciate this, even if it isn’t the most thrilling part of the project.

00:14:02.080 At Zencoder, we implemented a DSL for documentation to manage numerous API settings effectively. Each setting provides a short description, a long description, valid data types, examples, and more, allowing for dynamic documentation creation. This helps users see precisely how to use settings in API requests.

00:14:40.439 Even if your API design is excellent, having libraries for popular programming languages, like Ruby wrappers or Python wrappers, can enhance user experience. A raw API is essential, allowing users to access functionality directly within their languages.

00:15:16.800 Support your API—especially if API usage is key to your business. APIs can intimidating to users since they often feel like black boxes that may break without warning.

00:15:52.100 Make your API fast. API requests don’t need to endure the full overhead of regular web requests, so consider optimizations. Writing code that hits an API frequently is easier than having a user reload a page multiple times a second.

00:16:22.440 Rate limiting is useful too. Restrict how many requests a user can make per second to protect your system's resources. Logging API requests can provide insight into what users are doing and assist them in troubleshooting problems.

00:16:43.840 Lastly, even though an API is meant for software, creating tools for users—a request builder or a user-friendly interface—can drastically improve the user’s understanding of the API. A simple request builder provides an easy way for users to interact with the API to create valid requests.

00:17:00.839 Consider how you can facilitate your users' understanding. For example, think about good naming conventions for ecosystem APIs; clear and intuitive naming helps users identify what actions correspond to specific API calls.

00:17:32.600 Implement error handling effectively. Rather than failing silently, ensure that invalid input returns clear error messages. Monitoring the types of errors encountered by users can help refine documentation and support communication.

00:18:00.720 Lastly, always think about how you would use the API yourself. Ask for peer input, and if possible, conduct user testing to gain insights into potential enhancements.

00:18:28.640 Ultimately, if you create a great API, you stand to gain a lot both in personal satisfaction and professional success. This aligns with the three benefits I mentioned at the beginning: making friends, creating wealth (whether with VC backing or self-sufficiency), and changing the world.

00:18:57.360 An excellent API can help you make connections. For example, we received an enthusiastic tweet recognizing our API's outstanding quality, even extending a humorous offer of free hugs to our developers.

00:19:28.560 This phenomenon illustrates the asymmetrical reward cur as enhancements in API quality are incrementally recognized in some circles. The takeaway is that when you elevate your API, its value and recognition increases significantly.

00:20:04.880 The second benefit of an amazing API is the potential for profitable business outcomes and improved job satisfaction. Technology infrastructure spending is predicted to more than double in the coming years; startups are constantly emerging in this space.

00:20:39.800 Riding this wave provides ample opportunities for innovative technologies delivered via APIs. For example, creating a 3D printing API could allow significant access to manufacturing capabilities.

00:21:10.640 Finally, you have the power to change the world through APIs. The earlier tech boom proved that innovative approaches can lead to substantial investment and development.

00:21:53.000 With progress in open-source technologies, you can access everything needed to create winning solutions without the hefty infrastructure costs from previous successes. This freedom enables us to bootstrap viable ideas without depending on external funding.

00:22:20.400 In summary, open-source revolutions of the past have dramatically changed how we approach software development and empowered teams to bring new ideas to life.

00:22:52.920 If you build an incredible API, you change the game. And with great API design, not only do you get recognition, you foster community, economy, and innovation.

00:23:39.840 Does anyone have questions or thoughts? Great, I'd love to hear them.

00:23:56.840 [Various audience questions and answers follow]

LA RubyConf 2011