00:00:09.230
Hi everyone, my name is Konstantin Tennhard.
00:00:12.570
I have to warn you, this is not going to be a funny talk—I'm German, we don't do that.
00:00:18.510
However, I don’t actually live in Germany anymore; last year I moved to Ottawa, Canada.
00:00:28.830
There are so many Ruby developers in Ottawa that I actually work for Shopify.
00:00:37.110
In fact, we have so many Ruby developers, we just don’t know where to place them.
00:00:43.620
We even sent some of them down to Kansas to speak at Redcar, and there's even two of us speaking later this afternoon about how we test and about Sprockets.
00:00:52.640
To continue with some shameless self-promotion, you can find me on Twitter and GitHub; my handle is @t60.
00:01:03.870
In my spare time, I maintain a couple of libraries that you might find interesting.
00:01:10.080
One of them is Action Widgets, which is a UI component micro framework for Ruby on Rails.
00:01:15.630
In fact, the slides you see here on the screen are powered by Action Widgets.
00:01:24.150
I also maintain Smart Properties, which is a supercharged Ruby attribute accessor, as well as a processing pipeline for Ruby on Rails to model complex business processes.
00:01:30.320
Today, I want to talk about Request Interceptor, which is my most recent library.
00:01:41.250
It allows you to simulate foreign APIs with Sinatra.
00:01:44.640
At its core, this talk is all about testing, and specifically one type of test: tests that involve HTTP connections.
00:01:52.920
Many of you might know the libraries VCR and WebMock, which are typically used to stub out individual requests or, in the case of VCR, replay requests that have previously been sent to a remote API.
00:02:00.750
I want to present a different approach today and talk about how we can use Sinatra to simulate a foreign API within our test suite.
00:02:08.789
This is essentially the core idea behind Request Interceptor.
00:02:13.470
I guess I had better show you how to use the library first, and then throughout the talk, we will dive deeper and deeper into how it actually works internally.
00:02:23.390
By the end, I'll explain some of the meta-programming techniques I used to hook into the Net::HTTP library to make all that magic happen.
00:02:30.320
So, as I mentioned, Request Interceptor does modify Net::HTTP, and just like WebMock and VCR, there is no clean way to interject yourself into what Net::HTTP does.
00:02:49.050
So, some trickery is required to make that work, but I will get back to that later.
00:02:56.700
The idea is that you can use any Rack-compatible app and use it as a request interceptor that intercepts an HTTP request sent out by your application.
00:03:06.180
It reroutes that request to your Rack app, which will then handle the request inline.
00:03:22.680
In fact, all you need to know is that Request Interceptor implements a `run` method that takes a Rack application as well as a host name pattern.
00:03:30.990
The hostname pattern is important; when Request Interceptor starts intercepting requests, it will actually look at the HTTP request and only redirect it to your Rack app if it matches the hostname.
00:03:44.190
Otherwise, the HTTP request will be made just as a regular remote request.
00:03:54.630
In the code example here, I define probably the most minimal Rack app you could potentially implement.
00:04:00.270
It simply returns an array with the status code, no headers, and a message 'Hello'.
00:04:08.400
Then, I use Request Interceptor to intercept all requests that match anything ending with 'example.com'.
00:04:11.610
I make my HTTP request and then assert the quality of the response, which should be 'Hello'.
00:04:16.920
The problem with bare metal Rack apps is that they are very inconvenient when trying to implement something more feature-complete.
00:04:35.570
You wouldn’t necessarily want to go with Rack directly; instead, you would want to pick something that offers more convenience.
00:04:47.420
For me, this convenience comes from Sinatra, which combines simplicity with a lot of flexibility on how to simulate these API endpoints.
00:05:04.600
For those of you who don’t know, Sinatra is a Ruby micro web framework based around a very simple idea.
00:05:09.270
You have a Sinatra application that provides you with the most important methods like GET, POST, PUT, and DELETE, which correspond to the HTTP methods.
00:05:24.730
These methods allow you to define request handlers in your Sinatra application.
00:05:33.510
A simple Sinatra app can look something like this: you don’t even need to wrap it in a class, as it provides some magic to make this work.
00:05:48.340
You require the library and define that your application is handling anything that comes into '/hello', returning 'Hello Sinatra'.
00:05:57.890
Given this conciseness and simplicity, Sinatra was an excellent choice to model APIs, and therefore makes a great pairing with Request Interceptor.
00:06:06.849
In fact, I went further because of this great combination; it is the combination I would suggest for you to use.
00:06:29.280
Instead of using Request Interceptor's run method with just any Rack app, I would recommend using Sinatra.
00:06:40.990
Request Interceptor gives you a defined method that allows you to create a new Sinatra application with some extra goodness.
00:06:53.040
The Request Interceptor allows you to define the hostname pattern, just as we have seen before, where we submit the host pattern and the application to the run method.
00:07:02.660
We now define it right on the application and define it as a regular Sinatra app with all the endpoints that we need.
00:07:15.500
The result of this define call is a class again, which is a Sinatra application with the added benefits.
00:07:27.370
One of those benefits is that this application provides you with an intercept method.
00:07:35.350
The intercept method is a convenient wrapper around the run method.
00:07:46.800
Instead of having to pass in the host name you want to match and which application to pass in, you can just call intercept on your interceptor, provide it with a block, and then fire off an HTTP request.
00:08:01.140
You can then assert that the correct message is returned.
00:08:23.210
More importantly, to test this, you likely want to track how many requests you made, which requests you actually made, and what the request and response data were.
00:08:39.170
To make this possible, the intercept method returns a transaction log, which is simply an array of Request Interceptor transactions.
00:08:56.110
These transactions are structs that give you access to the request and response that was made within the block.
00:09:08.120
These are instances of Rack::Mock::Request and Rack::Mock::Response, just like other libraries typically used for testing Rack applications.
00:09:24.580
I essentially use these to carry all the data for further inspection.
00:09:27.640
The example below shows you how you can assert on the path of the first transaction log entry.
00:09:37.620
In this case, I'm just asserting that my program called the path '/hello' of 'example.com'.
00:09:56.670
You can also nest them in case you want to communicate with multiple APIs.
00:10:15.880
At Shopify, I was on the team that implemented Uber and we treated Shopify like an API just as you would if you developed an app for Shopify.
00:10:27.910
We needed our application to communicate with both of these services, and it’s often necessary to know exactly which requests were sent where.
00:10:39.340
That’s why Request Interceptors support nesting, so both of these interceptors write a separate transaction log.
00:10:53.870
Yes, of course, the innermost interceptor takes precedence.
00:11:08.860
If you have two interceptors responding to the same domain, the innermost will win and intercept the request.
00:11:18.420
Another important feature is that you can customize an interceptor for an individual test.
00:11:32.170
The idea is that you typically outline your service that you are modeling in a single file and then customize it to fit the specific needs of your test.
00:11:46.730
For example, if you wanted to model a narrow response for one specific endpoint, you would take your interceptor, call the `customize` method on it, and override the previously defined endpoint.
00:12:02.400
Sinatra is smart enough to ensure that if you redefine an endpoint, the new endpoint takes precedence over the old one.
00:12:14.500
For this case, we are switching the hello endpoint message from 'Hi there' to 'Bonjour, monde!'
00:12:29.890
Now that you have a basic understanding of how they work, I want to talk a little about the advantages in comparison to VCR and WebMock that I think exist when using Request Interceptors.
00:12:56.180
For me, one of the biggest advantages is that the code is not cluttered throughout your test suite.
00:13:07.860
Instead, we have one definite file that defines a particular service.
00:13:09.930
In our case, Uber or Shopify, that implements all the endpoints we usually communicate with.
00:13:23.260
In this way, if you want to see in one place what your app is actually communicating with, you just open the file and look at the interceptor definition.
00:13:37.700
Another advantage is that interceptors provide greater power and flexibility.
00:13:45.920
Because we are talking about a Sinatra application, you can go as far as you want with it.
00:13:54.020
Theoretically, you could implement an in-memory database that keeps state if you want to simulate entire workflows, or you can keep it simple and return static responses from your endpoints.
00:14:08.440
Of course, since it's essentially just one file, you could also package it into a Ruby gem.
00:14:31.540
If you build a service that other developers use and you have a public API, you could provide them with a predefined interceptor they can use in their test suite.
00:14:41.650
This would help them avoid hitting your API with real requests from their test suite.
00:14:56.760
Finally, and this is personally very important to me, is that the code is just very readable, which is part of the nature of a Sinatra application.
00:15:06.380
I believe it's more readable than having stubbed WebMock mocks scattered throughout your test suite.
00:15:09.640
Instead, you have this one single application defining how your interceptor works.
00:15:20.240
Furthermore, there are features that I am not sure if you could simulate using WebMock or VCR. I want to discuss advanced concepts about using these interceptors.
00:15:39.730
One big concept for me is simulating network request failures.
00:15:56.680
Request Interceptors are set up in a way that propagates errors or exceptions raised in one of the endpoints.
00:16:10.960
I specifically disabled different scenarios functionality to handle exceptions and propagate them through the entire stack.
00:16:24.810
This allows you to simulate that a host is unreachable simply by raising the appropriate exception.
00:16:36.860
This makes it easy to test whether your application is robust enough to handle these error cases.
00:16:44.370
Sinatra gives you many tools that you can leverage to make interceptor definitions even more straightforward and code more readable.
00:17:00.630
One of the most important things is that being a standard Ruby class, you can define private helper methods that you can utilize throughout your interceptor and the customizations you use in your test suite.
00:17:16.420
You can apply standard object-oriented design principles to make your interceptors as readable and easy to use as possible.
00:17:34.520
There’s also the possibility of using Sinatra’s before and after callbacks, which run before or after a particular endpoint is hit.
00:17:42.470
For example, you could utilize an after callback to automatically encode data into JSON. When modeling a JSON API, you would avoid having to remember to call `.to_json` on whatever is being sent over the wire.
00:17:56.480
You can define this encoding once in a block, checking the type of the response.
00:18:09.140
If it’s an array or hash, you can encode it into JSON.
00:18:18.800
Of course, you can use Rack middleware, allowing you to model both Shopify and Uber interceptors as JSON APIs.
00:18:35.820
You will want to decode incoming JSON, making data manipulation in your interceptors far simpler.
00:18:49.290
Sinatra provides a method called `use` that allows you to inject Rack middleware that runs before your actual endpoint is hit.
00:19:03.400
Now that we have an understanding of how to use interceptors and why they provide a nice alternative to VCR or WebMock, I want to delve deeper into some of the internals.
00:19:20.160
I find it interesting to see some of the powerful features Ruby provides as a learning exercise.
00:19:36.370
In the beginning of the talk, I mentioned that a Request Interceptor's `run` method is sort of the core of the whole idea.
00:19:49.260
In fact, this is the concrete method implementation as it exists in the library.
00:19:55.040
There are essentially six steps, and I'll go over all of these steps to showcase how you can manipulate an existing Ruby library that doesn’t easily allow for this.
00:20:03.090
The first step is to clear the transaction log. It's very easy; I just clear the array that keeps all the transaction log entries from the previous run.
00:20:21.310
Next, I cache the original Net::HTTP methods to restore once the block finishes executing.
00:20:34.560
Then, I override the Net::HTTP methods with custom implementations, similar to what WebMock does.
00:20:46.250
I execute my test, and now my test will utilize these overridden Net::HTTP methods.
00:20:56.800
Finally, I collect my transactions and restore Net::HTTP to its original state.
00:21:01.930
The last part operates in an ensure block, guaranteeing that it always runs.
00:21:22.200
This ensures that your test suite never ends up in a state where Net::HTTP is not in its original state.
00:21:28.390
Again, it’s simple to clear the transaction log, so I want to explore caching the original methods.
00:21:43.740
The three methods you need to override for intercepting HTTP requests are `start`, `finish`, and `request`.
00:21:56.880
Now, caching the original methods means that you have a concrete request interceptor instance managing your test case.
00:22:02.600
You assign these methods to instance variables, which gives you access to unbound methods.
00:22:11.820
Essentially, you save the original method implementations and replace the three methods with your implementations.
00:22:27.000
The `start` and `finish` methods allow the application to believe that there’s an open TCP connection.
00:22:39.330
However, you don’t need one because of how the redirection to the Sinatra application works.
00:22:50.950
I won’t show the code for the request method as it’s more complex, but I will explain what’s happening in the Request Interceptor's `request` method.
00:23:06.490
I try to find an appropriate interceptor; I look at the HTTP request and the host name.
00:23:19.900
Then, I go through my list of host patterns and stored applications to see if one matches.
00:23:33.710
If I find one, I build a mock request; the initializer of `mock_request` takes a Rack application as its first argument.
00:23:47.070
Once initialized, that mock request can use the methods like `get`, `post`, `put`, and `delete` to simulate an actual HTTP transaction.
00:23:59.750
Once completed, I get back a `mock_response`, which I need to transform into a `Net::HTTP` response.
00:24:11.200
This step is crucial to make `Net::HTTP` believe it just talked to a remote service.
00:24:26.490
Finally, I log the transaction, taking the `mock_request` and `mock_response` and writing them into my transaction log for analysis in my tests.
00:24:39.660
What happens if no appropriate interceptor matches your hostname?
00:24:54.530
To implement an unobtrusive system, I didn’t want to block any HTTP communication.
00:25:05.880
What actually happens is that my current `Net::HTTP` instance—which may be in a modified state—must be restored.
00:25:18.530
The procedure involves temporarily saving the state and then enabling `Net::HTTP` to perform requests again.
00:25:34.130
The next slide explains restoring, which utilizes Ruby’s `define_method`, which can take both blocks and unbound methods.
00:25:47.170
This means that methods saved in instance variables can now be rebound, allowing restoration of `Net::HTTP` to its original methods.
00:26:00.290
This way, when a Request Interceptor does not find a matching application, it can restore and execute the request as if nothing happened.
00:26:12.470
So that was essentially how the request cycle works in Request Interceptor.
00:26:33.100
In comparison to WebMock, there are certainly similarities, but here you define a stub within your test.
00:26:50.390
For Request Interceptor, you instead redirect to the Sinatra application I just mentioned.
00:27:12.730
Moreover, there is error propagation to simulate network errors.
00:27:23.790
You can configure Sinatra to raise exceptions and not handle them via disabling 'show_exceptions' and enabling 'raise_errors', creating an aggressive mode.
00:27:41.540
While this setup wouldn’t be appropriate for production applications, it is practical for simulating network request errors.
00:27:52.720
I have further plans for Request Interceptor, such as implementing support for traits—similar to Factory Girl mechanisms—where you can define factories for your interceptors.
00:28:11.480
It accommodates cases where you want to simulate the same endpoint multiple times.
00:28:25.200
Additionally, I want to support different adapters, like Faraday, to expose multiple libraries.
00:28:34.490
That leads me to the end of my talk. To summarize, Request Interceptors provide a third alternative to VCR and WebMock.
00:28:46.960
What I like most is the concise service definition in one place instead of scattering definitions throughout the test suite.
00:28:57.930
They also provide an easy mechanism to customize them when specific needs arise.
00:29:08.440
Finally, Sinatra provides simplicity and flexibility, which ultimately leads to very readable code—a quality I greatly enjoy.
00:29:21.680
If you’re interested in reviewing the slides since I shared a lot of content, they are available online.
00:29:36.279
Thanks a lot for your attention!
