Talks
Symmetric API Testing
Symmetric API Testing by Aditya Mukerjee
When implementing REST API servers and clients, testing is critical. Symmetric API testing is a design pattern that ensures compatibility between client and server with transparent tests. Testing APIs symmetrically reduces build times and code complexity, while simultaneously improving API stability.
Help us caption & translate this video!
GoRuCo 2016
00:00:15.730
Good morning, everyone! It's great to see so many people here, especially so early on a Saturday morning. My name is Aditya Mukerjee, and I am a risk engineer at Stripe.
00:00:21.050
Stripe is the payments backbone for many companies, including several of our sponsors here today, such as Lyft, Kickstarter, Slack, and UNICEF, among many others. On the risk team, I spend my days defending our users and customers—people like you—from online fraud.
00:00:39.440
If you're interested in hearing some battle stories or tales of what it's like on the dark web defending against fraudsters, I'm more than happy to talk with you about that later. But today, I'm here to talk about API design and testing.
00:00:52.430
Let's take a trip back to 2006. Ten years ago, Facebook had just opened itself up to high schoolers and college students, and people were freaking out about this. You may even have had a MySpace page; I would like to know how many people had a MySpace ten years ago.
00:01:10.580
Okay, good! I was afraid that only ten people would raise their hands, which would mean most people in this room were liars, because, let’s admit it, we all had MySpaces back then. Ten years ago, the best practices of web development dictated that you would do something like this: write a web server that would respond to HTTP requests by talking to your database, doing some computation, and then spinning back some HTML that the browser would render.
00:01:30.590
If you wanted to allow someone to write a native client for your application, the native client would use some raw format to communicate, likely XML at the time. It would still talk to the server, which would communicate with the database, perform some more computation, and then spit back data in that raw format. This system is fundamentally flawed; it is asymmetric.
00:01:56.659
Different clients are using different mediums to interact with the server to achieve fundamentally the same result. Fortunately, at some point, we realized that this was silly, and we consolidated our approaches. This is what the state looks like in 2016: we now understand that the browser is a special type of client, but it's still just a client. It communicates with the server using the exact same format and protocol as any other client, whether that's a desktop client or a mobile client.
00:02:33.170
Now, the server doesn’t even need to know what kind of client it’s communicating with, and this is beautiful. This is what we call symmetry in API design. When writing symmetric APIs, there are a few points you should keep in mind.
00:03:05.780
First, you want to design your API endpoints first. You don’t need to go overboard with this, but it’s crucial to make sure that your interfaces are clearly defined right from the start. Even if you're just prototyping and don't intend to make backward compatibility promises, you need to clearly outline how your client and server will communicate.
00:03:19.599
Designing your API first will force you to consider the key functional components of your application. If you've ever taken courses on UX, UI, or product design, you will have learned about user stories and storyboarding; this is essentially a lightweight way of achieving that. The interface defines how your users will interact with your product.
00:04:14.260
Finally, you always want to develop your API server and client together, even if someone else thinks they will provide the client for you. You should always write at least one client for every API you are building. If you do not, you won't fully understand your users' experiences, as you wouldn't have gone through the process yourself.
00:04:40.030
As for testing, there are several different ways that you might want to test APIs. I’ve polled many people, and no one can quite agree on the exact differences between integration tests and functional tests. I spoke to about ten people and got twenty different answers, but fortunately, for the purposes of this talk, it doesn't really matter too much.
00:05:04.479
We're just discussing degrees of granularity versus interconnection. For this talk, I’m going to use integration tests as a catch-all term to mean anything that tests more than a single application function, anything that examines how multiple application functions work together, or anything that relies on non-application state, such as a database call.
00:05:51.530
Integration tests provide several significant benefits. First of all, they give us the behavior that our users actually experience. It’s one thing to test all the ways our components should work together in isolation, which is what our unit tests provide, but we also want to test how our code actually functions in practice.
00:06:32.850
Integration tests can also provide information about changes that may occur in the external environment. For example, if you upgrade your database and it now uses a different format for communication, running only your unit tests might not indicate that your client or server suddenly broke. However, the same characteristics that make integration tests powerful can also render them fragile and unreliable.
00:07:21.090
They rely on external systems, which is beneficial because they can catch errors due to changes in those external systems. Still, it poses a challenge when tests fail, as it becomes unclear whether this is due to something outside our project scope or responsibility, or something we can control.
00:07:54.229
There's a great story I'd like to share about this. Some years back, NASA was building a satellite that needed to dock with the ISS. They conducted thorough testing at ground level to ensure that the probe would extend the correct distance from the body of the satellite. However, when they sent it into space, they forgot that space is a vacuum.
00:08:06.200
As a result, the force needed to project something a certain distance from the satellite body is slightly less in a vacuum than with atmospheric pressure, leading to a failure of the entire project. This illustrates the crucial need for thorough integration testing; although they performed unit tests at ground level, they neglected the integration tests that would account for environmental conditions.
00:08:54.199
Most projects will utilize a combination of unit tests, integration tests, and functional tests, which necessitates duplicating a lot of code. While this is beneficial for overall coverage, it ultimately increases the volume of code that must be maintained. More code means a greater chance of introducing bugs into our tests.
00:09:06.650
Let’s look at an example: we can consider Anaconda, the Twitter client library for Go. Anaconda was one of the first Go client libraries to return native structs, which you can think of as native objects. It was also the first Go library to handle rate limiting behind the scenes.
00:09:45.990
Most client libraries require you to handle errors related to rate limiting manually, informing you that you need to slow down. However, Anaconda manages this behind the scenes; it determines when you should retry and automatically replays the request for you.
00:10:37.969
I am using Anaconda as an example since it is open-source; you can explore it later if you’re interested. Importantly, for symmetric API testing, you want to test the client and server simultaneously. Even though I wrote the library, I did not write the Twitter API to go with it.
00:11:25.270
As I mentioned, the examples I use are in Go, but these design patterns are language-agnostic and can be applied in any programming language. A common approach to testing is to stub out responses. For instance, here’s an example JSON response from Twitter’s documentation for retrieving user information, alongside a corresponding struct that represents that response.
00:12:24.290
We can marshal the JSON into a struct to clearly illustrate the correspondence between the JSON response and the struct. For now, we will write this out manually, allowing us to mock these JSON responses in our tests using the native types. But what happens if our response structure changes?
00:13:26.900
Our tests will still refer to the old stub responses and won’t know about the new structure the server is providing. For example, Twitter used to return a string to indicate whether a tweet was censored in a specific country. At some point, they changed this to an array of strings, allowing tweets to be censored in multiple countries simultaneously.
00:14:26.570
This issue only occurs in production or when running tests against the live API. You may think this doesn't matter if you’re working in a dynamically typed language, but it can lead to errors. For instance, if you're writing JavaScript and use .length on a field you think is a string but is actually an array, the result will be semantically valid but not what you need.
00:15:19.890
Wouldn’t it be ideal to combine the reliability of unit tests with the completeness of integration tests without having to write additional code? Unfortunately, we can achieve this! Let’s focus on the client side of the tests.
00:16:08.070
We start by creating a local server. The first few lines of our testing code set it up, telling the client to query the local server instead of the actual API. We can construct the endpoint and hard-code the example response we want to return for our unit tests.
00:17:07.690
Instead of hard-coding responses for every controller, we can intercept live responses, record them, and then replay them. This process allows us to easily handle all HTTP requests dispatched through one source, which is crucial for simplicity.
00:17:54.720
What this essentially means is we don’t have to add response recording to every single endpoint or function separately. We modify it just once, drastically reducing the barrier to testing every API endpoint for structural integrity. There’s no reason to ship untested code when this process is so straightforward.
00:18:51.720
Thus, instead of creating separate handler functions for each endpoint, we can store the recorded responses in a directory dedicated to those JSON responses. For example, for the endpoint /user/lookup, we can save responses in a directory called JSON/user/lookup.json.
00:19:47.430
You may wonder why we named it JSON rather than responses. This choice is intentional, as we want to test symmetrically—the same data types that the client reads in the responses are also used by the server.
00:20:39.059
The production server processes a native type, computes, marshals it into JSON, and sends it over the wire. Why not share? Instead of manually writing that struct definition, we can use Go's capabilities to generate struct definitions from valid JSON.
00:21:34.650
For systems interacting with databases, even schema-less databases can have implicit schemas based on how you're interacting with your models. Therefore, it's beneficial to explicitly define your contracts.
00:22:45.000
To maintain consistency, we can use Go generate to update all struct definitions based on recorded JSON in our project directory. This ensures that our interface is preserved, and we know any changes will reflect in our code and tests.
00:23:35.859
This principle holds true across other programming languages as well. While Go generate caters specifically to Go, you can use various build systems to achieve similar effects in other environments. I want to briefly emphasis the importance of interfaces, as they are key to symmetric API testing.
00:24:20.220
An interface serves as a shared boundary that two components use to exchange information. In Go, any type can define an interface by implementing a specific method signature, allowing different types to satisfy the same interface.
00:25:16.970
This means you can write code against interfaces without knowing the specific types involved, which promotes flexibility and code that documents itself. Essentially, you want to ensure that your interfaces are designed thoughtfully as they facilitate symmetry in your application.
00:26:11.660
Interfacing guarantees that identical validity rules apply to both sides of the interface. You don't need interfaces built into the language to implement these patterns, and even in dynamically typed languages, you can infer interfaces through implementation.
00:27:12.480
It's important that you think about interfaces at the beginning of your coding and tooling processes, just as you would with testing and building.
00:27:58.470
We’re just scratching the surface with what interface types can achieve. If you’re interested in advanced uses of Go interfaces, I presented at Go for Khan in India and Dubai this past February, and I encourage you to check it out.
00:28:48.899
Until now, we've focused on testing just the client, but these same techniques can be applied to server logic. We record actual client requests and feed them directly to the server handlers. For example, the endpoint we’re testing could simply print a string based on a request’s input.
00:29:27.530
We are not making an actual HTTP request; instead, we create a request value representing what the server would see. This means we can test the handler directly without any network activity. When it comes to what we want to test on recorded responses, the obvious focus would be on state changes like database updates.
00:30:20.550
For instance, when testing a signup endpoint, we want to ensure a new user is created in the database, and also validate the server's response—ensuring the user object’s JSON representation is returned correctly.
00:31:16.700
Both the client and server interact with the same recordings for validation, ensuring consistency across the application. This calls for a symmetry layer that defines the contract for how these components communicate.
00:32:00.650
This principle of symmetry applies not only to JSON but also to various interchange formats, including Protocol Buffers or XML. It’s important to test the actual behavior of the client and server rather than an idealized model.
00:32:45.590
In summary, we have a few guiding principles for API testing. We have tests that compose and degrade gracefully, similar to how APIs should function. You should strive to write at least one client for every API and lower the barrier for writing tests to ensure the highest coverage.
00:33:36.589
Ultimately, remember that interfaces are powerful and are key to achieving symmetry within your applications. The strength of your API comes from the robust contracts you establish, either implicitly or explicitly.
