Foreign API Simulation with Sinatra

by Konstantin Tennhard

In this video titled 'Foreign API Simulation with Sinatra,' Konstantin Tennhard presents an innovative approach to testing HTTP connections when building API clients. Traditionally, developers have used libraries like VCR and WebMock to stub API requests, but Tennhard introduces a new method that utilizes Sinatra to create simulated foreign APIs directly within test suites.

Key points discussed in the talk include:

Introduction to Request Interceptor: Tennhard introduces his library, Request Interceptor, designed to facilitate the testing of HTTP requests without relying on external APIs.
Comparison with Existing Libraries: The speaker discusses the limitations of using VCR and WebMock, outlining how Request Interceptor provides a cleaner, more organized approach to defining API interactions.
How Request Interceptor Works: The library allows developers to intercept HTTP requests and reroute them to a Rack-compatible application using Sinatra. This provides greater flexibility in simulating different API behaviors.
Example Implementation: Tennhard shows a minimal example where a simple Rack app is created to respond to requests that match a specific hostname, showcasing the basic functionality of intercepting API calls.
Nesting and Customization: The talk covers advanced features such as nesting multiple interceptors for more complex testing scenarios and customizing the response for specific tests, allowing fine control over what each test needs to simulate.
Error Simulation: Another significant advantage of using Request Interceptor is its ability to simulate network errors, gracefully testing an application’s robustness against failures by raising exceptions in predefined scenarios.
Readable Code: Tennhard emphasizes the readability of the code created with Sinatra and Request Interceptor compared to traditional stubs, making it easier for developers to understand and maintain their test suites.
Future Plans: The talk concludes with Tennhard sharing his plans to enhance Request Interceptor by supporting traits for endpoint simulations and introducing different adapters for compatibility with various libraries.

Overall, Konstantin Tennhard's presentation highlights an efficient, organized, and flexible approach to testing API clients, advocating for the use of Sinatra in simulating foreign APIs within the testing frameworks of Ruby applications. This method enhances the clarity and maintainability of the code, ultimately leading to a better development experience.

00:00:09.230 Hi everyone, my name is Konstantin Tennhard.

00:00:12.570 I have to warn you, this is not going to be a funny talk—I'm German, we don't do that.

00:00:18.510 However, I don’t actually live in Germany anymore; last year I moved to Ottawa, Canada.

00:00:28.830 There are so many Ruby developers in Ottawa that I actually work for Shopify.

00:00:37.110 In fact, we have so many Ruby developers, we just don’t know where to place them.

00:00:43.620 We even sent some of them down to Kansas to speak at Redcar, and there's even two of us speaking later this afternoon about how we test and about Sprockets.

00:00:52.640 To continue with some shameless self-promotion, you can find me on Twitter and GitHub; my handle is @t60.

00:01:03.870 In my spare time, I maintain a couple of libraries that you might find interesting.

00:01:10.080 One of them is Action Widgets, which is a UI component micro framework for Ruby on Rails.

00:01:15.630 In fact, the slides you see here on the screen are powered by Action Widgets.

00:01:24.150 I also maintain Smart Properties, which is a supercharged Ruby attribute accessor, as well as a processing pipeline for Ruby on Rails to model complex business processes.

00:01:30.320 Today, I want to talk about Request Interceptor, which is my most recent library.

00:01:41.250 It allows you to simulate foreign APIs with Sinatra.

00:01:44.640 At its core, this talk is all about testing, and specifically one type of test: tests that involve HTTP connections.

00:01:52.920 Many of you might know the libraries VCR and WebMock, which are typically used to stub out individual requests or, in the case of VCR, replay requests that have previously been sent to a remote API.

00:02:00.750 I want to present a different approach today and talk about how we can use Sinatra to simulate a foreign API within our test suite.

00:02:08.789 This is essentially the core idea behind Request Interceptor.

00:02:13.470 I guess I had better show you how to use the library first, and then throughout the talk, we will dive deeper and deeper into how it actually works internally.

00:02:23.390 By the end, I'll explain some of the meta-programming techniques I used to hook into the Net::HTTP library to make all that magic happen.

00:02:30.320 So, as I mentioned, Request Interceptor does modify Net::HTTP, and just like WebMock and VCR, there is no clean way to interject yourself into what Net::HTTP does.

00:02:49.050 So, some trickery is required to make that work, but I will get back to that later.

00:02:56.700 The idea is that you can use any Rack-compatible app and use it as a request interceptor that intercepts an HTTP request sent out by your application.

00:03:06.180 It reroutes that request to your Rack app, which will then handle the request inline.

00:03:22.680 In fact, all you need to know is that Request Interceptor implements a `run` method that takes a Rack application as well as a host name pattern.

00:03:30.990 The hostname pattern is important; when Request Interceptor starts intercepting requests, it will actually look at the HTTP request and only redirect it to your Rack app if it matches the hostname.

00:03:44.190 Otherwise, the HTTP request will be made just as a regular remote request.

00:03:54.630 In the code example here, I define probably the most minimal Rack app you could potentially implement.

00:04:00.270 It simply returns an array with the status code, no headers, and a message 'Hello'.

00:04:08.400 Then, I use Request Interceptor to intercept all requests that match anything ending with 'example.com'.

00:04:11.610 I make my HTTP request and then assert the quality of the response, which should be 'Hello'.

00:04:16.920 The problem with bare metal Rack apps is that they are very inconvenient when trying to implement something more feature-complete.

00:04:35.570 You wouldn’t necessarily want to go with Rack directly; instead, you would want to pick something that offers more convenience.

00:04:47.420 For me, this convenience comes from Sinatra, which combines simplicity with a lot of flexibility on how to simulate these API endpoints.

00:05:04.600 For those of you who don’t know, Sinatra is a Ruby micro web framework based around a very simple idea.

00:05:09.270 You have a Sinatra application that provides you with the most important methods like GET, POST, PUT, and DELETE, which correspond to the HTTP methods.

00:05:24.730 These methods allow you to define request handlers in your Sinatra application.

00:05:33.510 A simple Sinatra app can look something like this: you don’t even need to wrap it in a class, as it provides some magic to make this work.

00:05:48.340 You require the library and define that your application is handling anything that comes into '/hello', returning 'Hello Sinatra'.

00:05:57.890 Given this conciseness and simplicity, Sinatra was an excellent choice to model APIs, and therefore makes a great pairing with Request Interceptor.

00:06:06.849 In fact, I went further because of this great combination; it is the combination I would suggest for you to use.

00:06:29.280 Instead of using Request Interceptor's run method with just any Rack app, I would recommend using Sinatra.

00:06:40.990 Request Interceptor gives you a defined method that allows you to create a new Sinatra application with some extra goodness.

00:06:53.040 The Request Interceptor allows you to define the hostname pattern, just as we have seen before, where we submit the host pattern and the application to the run method.

00:07:02.660 We now define it right on the application and define it as a regular Sinatra app with all the endpoints that we need.

00:07:15.500 The result of this define call is a class again, which is a Sinatra application with the added benefits.

00:07:27.370 One of those benefits is that this application provides you with an intercept method.

00:07:35.350 The intercept method is a convenient wrapper around the run method.

00:07:46.800 Instead of having to pass in the host name you want to match and which application to pass in, you can just call intercept on your interceptor, provide it with a block, and then fire off an HTTP request.

00:08:01.140 You can then assert that the correct message is returned.

00:08:23.210 More importantly, to test this, you likely want to track how many requests you made, which requests you actually made, and what the request and response data were.

00:08:39.170 To make this possible, the intercept method returns a transaction log, which is simply an array of Request Interceptor transactions.

00:08:56.110 These transactions are structs that give you access to the request and response that was made within the block.

00:09:08.120 These are instances of Rack::Mock::Request and Rack::Mock::Response, just like other libraries typically used for testing Rack applications.

00:09:24.580 I essentially use these to carry all the data for further inspection.

00:09:27.640 The example below shows you how you can assert on the path of the first transaction log entry.

00:09:37.620 In this case, I'm just asserting that my program called the path '/hello' of 'example.com'.

00:09:56.670 You can also nest them in case you want to communicate with multiple APIs.

00:10:15.880 At Shopify, I was on the team that implemented Uber and we treated Shopify like an API just as you would if you developed an app for Shopify.

00:10:27.910 We needed our application to communicate with both of these services, and it’s often necessary to know exactly which requests were sent where.

00:10:39.340 That’s why Request Interceptors support nesting, so both of these interceptors write a separate transaction log.

00:10:53.870 Yes, of course, the innermost interceptor takes precedence.

00:11:08.860 If you have two interceptors responding to the same domain, the innermost will win and intercept the request.

00:11:18.420 Another important feature is that you can customize an interceptor for an individual test.

00:11:32.170 The idea is that you typically outline your service that you are modeling in a single file and then customize it to fit the specific needs of your test.

00:11:46.730 For example, if you wanted to model a narrow response for one specific endpoint, you would take your interceptor, call the `customize` method on it, and override the previously defined endpoint.

00:12:02.400 Sinatra is smart enough to ensure that if you redefine an endpoint, the new endpoint takes precedence over the old one.

00:12:14.500 For this case, we are switching the hello endpoint message from 'Hi there' to 'Bonjour, monde!'

00:12:29.890 Now that you have a basic understanding of how they work, I want to talk a little about the advantages in comparison to VCR and WebMock that I think exist when using Request Interceptors.

00:12:56.180 For me, one of the biggest advantages is that the code is not cluttered throughout your test suite.

00:13:07.860 Instead, we have one definite file that defines a particular service.

00:13:09.930 In our case, Uber or Shopify, that implements all the endpoints we usually communicate with.

00:13:23.260 In this way, if you want to see in one place what your app is actually communicating with, you just open the file and look at the interceptor definition.

00:13:37.700 Another advantage is that interceptors provide greater power and flexibility.

00:13:45.920 Because we are talking about a Sinatra application, you can go as far as you want with it.

00:13:54.020 Theoretically, you could implement an in-memory database that keeps state if you want to simulate entire workflows, or you can keep it simple and return static responses from your endpoints.

00:14:08.440 Of course, since it's essentially just one file, you could also package it into a Ruby gem.

00:14:31.540 If you build a service that other developers use and you have a public API, you could provide them with a predefined interceptor they can use in their test suite.

00:14:41.650 This would help them avoid hitting your API with real requests from their test suite.

00:14:56.760 Finally, and this is personally very important to me, is that the code is just very readable, which is part of the nature of a Sinatra application.

00:15:06.380 I believe it's more readable than having stubbed WebMock mocks scattered throughout your test suite.

00:15:09.640 Instead, you have this one single application defining how your interceptor works.

00:15:20.240 Furthermore, there are features that I am not sure if you could simulate using WebMock or VCR. I want to discuss advanced concepts about using these interceptors.

00:15:39.730 One big concept for me is simulating network request failures.

00:15:56.680 Request Interceptors are set up in a way that propagates errors or exceptions raised in one of the endpoints.

00:16:10.960 I specifically disabled different scenarios functionality to handle exceptions and propagate them through the entire stack.

00:16:24.810 This allows you to simulate that a host is unreachable simply by raising the appropriate exception.

00:16:36.860 This makes it easy to test whether your application is robust enough to handle these error cases.

00:16:44.370 Sinatra gives you many tools that you can leverage to make interceptor definitions even more straightforward and code more readable.

00:17:00.630 One of the most important things is that being a standard Ruby class, you can define private helper methods that you can utilize throughout your interceptor and the customizations you use in your test suite.

00:17:16.420 You can apply standard object-oriented design principles to make your interceptors as readable and easy to use as possible.

00:17:34.520 There’s also the possibility of using Sinatra’s before and after callbacks, which run before or after a particular endpoint is hit.

00:17:42.470 For example, you could utilize an after callback to automatically encode data into JSON. When modeling a JSON API, you would avoid having to remember to call `.to_json` on whatever is being sent over the wire.

00:17:56.480 You can define this encoding once in a block, checking the type of the response.

00:18:09.140 If it’s an array or hash, you can encode it into JSON.

00:18:18.800 Of course, you can use Rack middleware, allowing you to model both Shopify and Uber interceptors as JSON APIs.

00:18:35.820 You will want to decode incoming JSON, making data manipulation in your interceptors far simpler.

00:18:49.290 Sinatra provides a method called `use` that allows you to inject Rack middleware that runs before your actual endpoint is hit.

00:19:03.400 Now that we have an understanding of how to use interceptors and why they provide a nice alternative to VCR or WebMock, I want to delve deeper into some of the internals.

00:19:20.160 I find it interesting to see some of the powerful features Ruby provides as a learning exercise.

00:19:36.370 In the beginning of the talk, I mentioned that a Request Interceptor's `run` method is sort of the core of the whole idea.

00:19:49.260 In fact, this is the concrete method implementation as it exists in the library.

00:19:55.040 There are essentially six steps, and I'll go over all of these steps to showcase how you can manipulate an existing Ruby library that doesn’t easily allow for this.

00:20:03.090 The first step is to clear the transaction log. It's very easy; I just clear the array that keeps all the transaction log entries from the previous run.

00:20:21.310 Next, I cache the original Net::HTTP methods to restore once the block finishes executing.

00:20:34.560 Then, I override the Net::HTTP methods with custom implementations, similar to what WebMock does.

00:20:46.250 I execute my test, and now my test will utilize these overridden Net::HTTP methods.

00:20:56.800 Finally, I collect my transactions and restore Net::HTTP to its original state.

00:21:01.930 The last part operates in an ensure block, guaranteeing that it always runs.

00:21:22.200 This ensures that your test suite never ends up in a state where Net::HTTP is not in its original state.

00:21:28.390 Again, it’s simple to clear the transaction log, so I want to explore caching the original methods.

00:21:43.740 The three methods you need to override for intercepting HTTP requests are `start`, `finish`, and `request`.

00:21:56.880 Now, caching the original methods means that you have a concrete request interceptor instance managing your test case.

00:22:02.600 You assign these methods to instance variables, which gives you access to unbound methods.

00:22:11.820 Essentially, you save the original method implementations and replace the three methods with your implementations.

00:22:27.000 The `start` and `finish` methods allow the application to believe that there’s an open TCP connection.

00:22:39.330 However, you don’t need one because of how the redirection to the Sinatra application works.

00:22:50.950 I won’t show the code for the request method as it’s more complex, but I will explain what’s happening in the Request Interceptor's `request` method.

00:23:06.490 I try to find an appropriate interceptor; I look at the HTTP request and the host name.

00:23:19.900 Then, I go through my list of host patterns and stored applications to see if one matches.

00:23:33.710 If I find one, I build a mock request; the initializer of `mock_request` takes a Rack application as its first argument.

00:23:47.070 Once initialized, that mock request can use the methods like `get`, `post`, `put`, and `delete` to simulate an actual HTTP transaction.

00:23:59.750 Once completed, I get back a `mock_response`, which I need to transform into a `Net::HTTP` response.

00:24:11.200 This step is crucial to make `Net::HTTP` believe it just talked to a remote service.

00:24:26.490 Finally, I log the transaction, taking the `mock_request` and `mock_response` and writing them into my transaction log for analysis in my tests.

00:24:39.660 What happens if no appropriate interceptor matches your hostname?

00:24:54.530 To implement an unobtrusive system, I didn’t want to block any HTTP communication.

00:25:05.880 What actually happens is that my current `Net::HTTP` instance—which may be in a modified state—must be restored.

00:25:18.530 The procedure involves temporarily saving the state and then enabling `Net::HTTP` to perform requests again.

00:25:34.130 The next slide explains restoring, which utilizes Ruby’s `define_method`, which can take both blocks and unbound methods.

00:25:47.170 This means that methods saved in instance variables can now be rebound, allowing restoration of `Net::HTTP` to its original methods.

00:26:00.290 This way, when a Request Interceptor does not find a matching application, it can restore and execute the request as if nothing happened.

00:26:12.470 So that was essentially how the request cycle works in Request Interceptor.

00:26:33.100 In comparison to WebMock, there are certainly similarities, but here you define a stub within your test.

00:26:50.390 For Request Interceptor, you instead redirect to the Sinatra application I just mentioned.

00:27:12.730 Moreover, there is error propagation to simulate network errors.

00:27:23.790 You can configure Sinatra to raise exceptions and not handle them via disabling 'show_exceptions' and enabling 'raise_errors', creating an aggressive mode.

00:27:41.540 While this setup wouldn’t be appropriate for production applications, it is practical for simulating network request errors.

00:27:52.720 I have further plans for Request Interceptor, such as implementing support for traits—similar to Factory Girl mechanisms—where you can define factories for your interceptors.

00:28:11.480 It accommodates cases where you want to simulate the same endpoint multiple times.

00:28:25.200 Additionally, I want to support different adapters, like Faraday, to expose multiple libraries.

00:28:34.490 That leads me to the end of my talk. To summarize, Request Interceptors provide a third alternative to VCR and WebMock.

00:28:46.960 What I like most is the concise service definition in one place instead of scattering definitions throughout the test suite.

00:28:57.930 They also provide an easy mechanism to customize them when specific needs arise.

00:29:08.440 Finally, Sinatra provides simplicity and flexibility, which ultimately leads to very readable code—a quality I greatly enjoy.

00:29:21.680 If you’re interested in reviewing the slides since I shared a lot of content, they are available online.

00:29:36.279 Thanks a lot for your attention!