Yes, You Should Provide a Client Library For Your API

HTTP

Daniel Azuma

@dazuma

#api

#developer-experience-dx

Yes, You Should Provide a Client Library For Your API

by Daniel Azuma

In the talk titled "Yes, You Should Provide a Client Library For Your API," presented by Daniel Azuma at RubyConf 2018, the importance of providing an API client library for RESTful JSON-based APIs is discussed. Azuma explains that while creating an API allows for access via HTTP requests, a dedicated client library can significantly enhance the user experience through improved safety, security, readability, and ease of use.

Key points from the presentation include:

Understanding API Client Libraries: An API client library serves as an interface, simplifying the interaction with an API by allowing users to call methods instead of constructing HTTP requests directly. This creates a more intuitive developer experience.
The Challenges of API Development: Effective API design involves complexities such as resource mapping, naming conventions, versioning, and error handling. Azuma shares experiences from his career, noting the time lost due to confusion in API usage between different teams, emphasizing that well-designed APIs are crucial for productivity.
Benefits of a Client Library: Client libraries abstract intricate details such as authentication, retries, and session management, allowing developers to focus on higher-level logic without wrestling with lower-level networking concerns.
Best Practices for Client Libraries:
- Leverage Ruby's unique features and types to create a familiar interface for Ruby developers.
- Implement robust error handling to manage common issues like quota errors and session expiration.
- Ensure security by incorporating input validation and encryption seamlessly into the library.
- Provide performance optimizations through caching and efficient data handling.
Automation in Client Library Generation: Azuma discusses utilizing Interface Description Languages (IDLs) like OpenAPI to automate the generation of client libraries, thereby maintaining consistency and reducing the maintenance burden. Google employs similar strategies with tools like protocol buffers and gRPC.

In conclusion, Daniel Azuma advocates for the creation of client libraries as they play a vital role in easing developer workflows and enhancing overall API usability. He stresses that even organizations with numerous APIs should prioritize user experience by investing in well-documented and automated client library generation.

00:00:20.650 So this hour, we're going to talk about APIs. Who here is an API producer? Who here has created and released an API? Okay, about a quarter to a third of you. Alright, now who here is a web developer? Who has released a website? That's about at least half, half to two-thirds. You two are API producers; you are writing code that's designed to be called by other code, like a web browser, so you also have produced an API. Now, who here has written a class—a Ruby class, Java class, C++ class? Most hands should be up. Okay, you are also API producers; you are writing code that is designed to be called by other code. Now, in this hour, we're going to focus on network APIs, not class APIs. Some of this will apply to classes and some parts of it won't, but some of the underlying principles will be the same.

00:02:15.490 Okay, last question: who here speaks the language—English, Japanese, Spanish, Hindi? Everyone’s hands should be up. Alright, you are not necessarily API producers, but you’re doing many of the same things; you’re communicating, you’re interacting with something else, some other person. You’re communicating. So, this is something that we all do, and it can be useful to think of APIs in terms of communication.

00:02:51.630 In this session, we're going to focus on APIs, in particular, on client libraries—API client libraries. Now, what is a client library? It's a Ruby gem, a Ruby library that provides a nice Ruby interface for calling an API—a network API. Suppose you've built an API that uses HTTP and JSON. Users can call your API using a standard HTTP client, for example. A simple example could use net HTTP, which is part of the Ruby standard library, to call an API. You would construct an HTTP request, marshal some JSON data, and parse a JSON response. That seems straightforward, but what if you could make that same API call by just invoking a method? This is clearly a lot shorter and easier than constructing that HTTP request.

00:03:50.360 We could say this provides a better user experience for your users—better yet, a happier developer experience. Of course, this means you have to write that client class, provide those methods, build it into a library, and maintain it. You have to keep it up to date and document it; all that baggage raises some good questions: How do you do all that? What are some techniques that can help you create a good client library? And is that engineering effort worth it? Those are some of the questions we will cover in this session.

00:05:06.820 We’ll think about why an API client library might be useful. We’ll talk about how it can provide a better and happier developer experience for your users. We’ll see some of the common features of client libraries, the techniques, and the practices that go into writing them. Then we'll discuss tools that might help you generate client libraries for your APIs. Before we get started, a bit about your speaker: My name is Daniel Azuma. I've been part of the Ruby community for a little while and have been developing in Ruby for about 13 years. I did Java and C++ and other things before that. Currently, I work at Google, where I serve as the Ruby engineering lead for Google Cloud. My job is to make the cloud better for Ruby developers like you. One of the major areas my team works on at Google is managing the Ruby client libraries for Google's APIs. Google is a large company with many products and several hundred public APIs, not to mention our internal services. So, we provide client libraries for all of those APIs.

00:06:59.040 That leads to a significant amount of code to maintain. When you reach this level, you start to wonder: is it worth it? Is all that engineering effort needed for maintaining a client library worth it when users could simply construct HTTP requests themselves? Most of you probably do not face the scale we deal with at Google, but it’s still a valid question and one that I faced early in my career, long before joining Google. Imagine me early in my career—perhaps not that early, but years ago. I was at a startup and we were a Ruby shop building a back-end API for a mobile app. My team consisted entirely of back-end developers who knew little about mobile languages and frameworks at the time. So, we hired an external engineering team, actually an offshore team, to handle our mobile development.

00:08:53.110 My back-end team deployed our API to staging and documented it carefully because we didn't know what tools or libraries the mobile team had. We didn't want to take the time to learn those mobile languages and frameworks ourselves. We created this spec and sent it to the mobile team. After a week, we started receiving emails from them expressing confusion. They claimed they couldn’t understand parts of it, primarily the authentication and request signing processes, and it turned into a back-and-forth situation where we had to hold late-night meetings to clarify these points. The mobile team was in a different time zone, and I found myself at the office until about 10 or 11 o'clock every night, and we went back and forth trying to iron out these issues.

00:10:09.390 Despite thinking we had documented this well, we ended up losing three to four weeks just getting our API to talk to the mobile team before they even started on the mobile interface. If you're in a startup, you understand that a few weeks can be an eternity when bootstrapping your business. Later, after we finally shipped, we discussed what we could have done differently. Should we have written the client mobile interface ourselves? I don’t know for sure—it's a tough question. However, I do know this experience opened my eyes to the fact that APIs are hard—they're really hard. They’re hard in terms of resource design, mapping your problem domain to RESTful APIs. Naming is hard, versioning is hard, future-proofing is hard, and using or calling an API can also be troublesome.

00:11:47.270 For instance, think about authentication—who here has implemented an authentication flow? Alright, who here has gotten it right the first time? Not many! Another area is retry mechanisms; when can you retry safely? How many times do you retry, and how long do you wait between retries? There is also exponential back-off, and what happens if you get quota errors or other complications? These are all intricate details. And let’s not forget about caching—knowing when to cache your requests can be tricky. Consider also error handling, long-running operations, and the overall complexity of real-world APIs, which often comprise much more than just basic HTTP requests.

00:12:52.290 Matzo spoke this morning about programming languages as a form of expression of ideas, which is a great description. It applies to APIs as well; they represent a form of expression of ideas. When I stand here and share ideas with you, I'm imparting content through words, but communication is much more than just words. There's side-channel information being passed back and forth—emphasis, tone of voice, and facial expressions are all crucial elements. Context matters too—the order in which we present things, the setting, the fact that we're at a Ruby conference, and my identity as a Ruby developer and Google engineer shapes how you perceive my message.

00:15:30.890 We all communicate and understand side-channel meanings. There is a protocol for human communication, and you're familiar with that protocol. That's what I see the role of a client library as being: it must know this protocol. Not just the HTTP or JSON protocol, but also the authentication, retries, sessions, caching, and all those side concerns that are not handled by your HTTP library. A client library manages all that protocol so users don’t have to deal with it. So, what does this look like in practice? Let’s take a look at a simple interaction with a client library, this time with Google's Translate API. You notice that once we create an object that represents the service, we can start making API calls by calling its methods. The API calls return mostly objects we can extract information from.

00:18:09.130 Notice what we don’t see: we don’t see URLs, JSON, or HTTP directly in our client usage. This might be an HTTP JSON API, but the implementation details are abstracted away, as that’s not how users want to interact with your API. We aim to hide complex, difficult issues like retries or caching; our goal is to create a good user experience. We want the developer experience to be enjoyable, as this is the core of why we create client libraries. It's a key component of being a Rubyist—focusing on happiness, productivity, and community.

00:20:32.890 Let’s discuss a few pro tips for effective client libraries: First, leverage Ruby abstractions; ensure your library feels familiar to Ruby developers. Use standard Ruby data types, like using Ruby's Time class for timestamps, and integrate with Ruby’s standard libraries when appropriate. For example, when working with logging, leverage Ruby's Logger class. Active Record provides an excellent example of effective abstractions in Ruby. It offers a method called find_each, which returns a Ruby enumerator, allowing developers to interact with large databases efficiently.

00:23:36.110 Secondly, handle errors in your client library effectively. Errors will happen; they are annoying, but you can mitigate the frustration by handling them within your library. This includes managing retries, quota errors, session expiration, and communicating error states back to users via exceptions. Proper error reporting translates to a better developer experience.

00:26:16.890 Let's talk about security. This is a meticulous concern; help your users by checking inputs and sanitizing inputs to prevent SQL injection attacks. If your API requires encryption or security protocols, build them into your client library. Don’t make your users struggle with the intricacies of request signing; instead, provide them with a seamless interface. You might hear concerns that client libraries add layers of abstraction that could harm performance. However, this is rarely the case. Network and database latency often dominate performance, and a well-designed client library can help improve performance by offering best practices like caching, batching, and handling asynchronous behavior effortlessly.

00:28:51.930 Finally, client libraries should provide instrumentation to gather usage information. Log API calls and document errors so that your users and you, as the API provider, can track what's happening within your application and make data-driven decisions. Although implementing a client library seemingly adds much work, organizations can utilize Interface Description Languages (IDLs) to automate aspects of client library generation. An IDL is a machine-readable specification detailing the API's methods, parameters, response types, and behaviors. These descriptions can be parsed to generate client libraries automatically.

00:31:30.980 Modern formats like OpenAPI allow you to create readable specs that interface well with various code generation tools. Google uses similar strategies with protocol buffers and gRPC, which can facilitate high-performance systems. Early in my tenure at Google, our client libraries used metaprogramming which had downsides like obscure code and poor documentation. We eventually transitioned to using code generators that produce static class definitions, enabling us to produce reliable code as well as documentation from the same specifications. This strategy fosters a consistent user experience, making it simpler for developers to engage with your APIs.

00:34:56.840 In summary, we discussed why you should consider providing an API client library, what functions it should serve, and its significance in enhancing user experience. We also detailed how organizations, even if they support numerous APIs across multiple languages, can successfully maintain and generate client libraries through automation. Client libraries play a pivotal role in ensuring a smooth developer experience, and yes, you should provide one for your API.

00:36:39.020 Thank you for coming. I’ve compiled a set of resources for getting started, which will be available at a URL I’m preparing. If you need to take a photo of your phone, that’s the URL to capture. The slides and video will be online, along with links and articles I’ve collected.

00:36:55.080 Are there any questions? Yes, back there.

00:37:12.350 The question is about handling API versioning and how to address breaking changes. At Google, we express versions clearly in our API clients, allowing users to specify which API version they want when creating their objects. Additionally, we’ve created higher-level interactions that tend to smooth out differences in API specifications for common operations.