00:00:12.120
Today I'm going to talk to you about WebMock, a Ruby gem for mocking HTTP requests.
00:00:20.480
First, a little bit about me.
00:00:26.560
I've been developing software for over 20 years. Back in 2006, when I was still a Java developer, my friend Andre KDA came to me.
00:00:34.079
He asked, 'Hey, have you heard of that new language called Ruby?' I've been using Ruby ever since.
00:00:39.200
Professionally, I've been using Ruby since 2007. I live in Wroclaw but have been working remotely since before it was trendy.
00:00:47.079
I'm the author and the sole maintainer of WebMock.
00:01:01.600
Today, I'm a little stressed because this is my first presentation in 14 years.
00:01:08.920
After seeing all these great presentations today, I feel the bar has been raised.
00:01:16.400
Being the final speaker of the day is a unique experience.
00:01:23.159
This is also my second presentation about WebMock; the previous one was in London 15 years ago. I was still stressed then, and nothing has changed.
00:01:35.720
After seeing David Halas speak yesterday about formal languages, I thought I should add a slide about something I learned at my university.
00:01:41.560
This led me to think about extrapolation, so I extrapolated when my next talk about WebMock would be.
00:01:48.799
We have a unique chance to learn something about WebMock today.
00:02:04.119
So, what is WebMock? WebMock has two core functionalities.
00:02:10.000
First, it allows you to stub HTTP requests. This means you can declare fake responses so that when your application attempts to make an HTTP request, no connection is made, and instead, a fake response is returned.
00:02:21.480
Second, it allows you to verify that HTTP requests have been executed, whether they are stubbed or real.
00:02:32.959
The idea is not original; there was a gem called FakeWeb before. I tried to use it, but its request matching capabilities were too limited, and it only supported Net::HTTP.
00:02:45.919
Therefore, I decided to write my own solution and created WebMock during a hackathon.
00:02:55.440
The first version of WebMock was developed to address my own needs as a Ruby developer.
00:03:02.040
These needs turned out to be the needs of many other Ruby developers as well, and hence, WebMock has been downloaded more than 250 million times up until now.
00:03:15.799
It has almost 4,000 stars on GitHub; by the way, if you haven't starred it yet, please do. I'm aiming for 4,000.
00:03:23.280
Over 260 people have contributed to WebMock over the last 15 years, and I would very much like to thank everyone who has ever contributed.
00:03:30.800
WebMock is used by thousands of public projects on GitHub, and almost 8,000 registered gems depend on it.
00:03:37.440
Some of the projects that depend on WebMock are listed here, and you might have heard of or used some of them.
00:03:46.800
If you haven't had a chance to use any of them, you probably have used this one. Machi mentioned today that if you know something, you should publish it.
00:03:56.799
So I want to encourage you, if you have a solution that works for you, to publish it. It might be useful to others.
00:04:05.680
WebMock has also inspired developers in other languages, leading to the creation of solutions inspired by WebMock.
00:04:12.360
I've even seen the term 'web mocking' used to refer to mocking HTTP requests.
00:04:20.448
So, what can you do with WebMock? First, WebMock provides a nice domain-specific language (DSL) for declaring request stubs.
00:04:28.600
It is easily readable and understandable. Second, it offers a DSL for verifying that HTTP requests have been made. Both DSLs are HTTP client agnostic, so you can use them with any HTTP client.
00:04:46.680
WebMock follows the stub, execute, verify paradigm, which I borrowed from a Java testing framework called Mockito.
00:05:03.680
It separates the stubbing phase from the verification phase, which might be different from frameworks like RSpec, where a single expect declaration defines both the stub and the expectation.
00:05:12.960
In WebMock, the steps are separated, allowing you to stub without verifying or verifying without stubbing.
00:05:21.200
WebMock offers request callbacks, and with these callbacks, you can record requests.
00:05:29.200
There is a gem called VCR for recording HTTP requests, which uses WebMock under the hood.
00:05:37.680
WebMock offers many features for stubbing; for example, you can stub requests to timeout or with basic authentication.
00:05:46.080
You can even replay raw responses recorded with curl.
00:05:53.200
WebMock supports all popular HTTP clients in Ruby and all major testing frameworks.
00:06:03.680
But then you might ask, why mock HTTP requests at all? Why should we care?
00:06:15.760
First, it allows you to run your tests without needing an internet connection.
00:06:24.160
For anyone, like me, who arrived here by train and has been working on a train, this is a useful feature.
00:06:41.440
If you misconfigure your testing environment and accidentally use production credentials, with WebMock, your tests won't make any delete requests on a production third-party API.
00:06:56.200
You always get a stable environment for running your tests.
00:07:04.000
Even if a third-party service is down or temporarily returns 500 responses, your tests will still go through and pass.
00:07:15.000
You also get fast tests, as you don't have to wait for any API delays—your responses are instant.
00:07:24.400
You can also test various scenarios, such as different status codes and error responses, and see how your app reacts to them.
00:07:36.200
All this test setup is done in Ruby files.
00:07:46.200
Mocking HTTP requests is not only useful for testing environments; you can also define your request stubs in the development environment and work offline.
00:07:56.200
You can set request stubs for various scenarios and test your app manually while simulating different situations.
00:08:06.200
One feature that I have used many times in the past is that it allows you to develop without access to a third-party API.
00:08:14.200
For example, I once worked on a project that heavily relied on a third-party API.
00:08:23.760
We received access to it only after a month and a half.
00:08:33.600
In the meantime, we read the documentation and API specifications, created some stubs, and were ready to go.
00:08:40.200
You can do the same in your development environment to demo the features that have been implemented, even before you have access to the external API.
00:08:49.640
Now, let me show you a sample Rails initializer for development and staging.
00:08:56.200
So, we know that stubbing HTTP requests can be useful at times.
00:09:05.000
But why would we use WebMock instead of creating our own solution?
00:09:15.200
Let's say we find a piece of code in our app that makes an HTTP request with basic authentication.
00:09:25.120
We are going to make a change to it, so we search for a test because we do test-first development, and we change the test first before changing the actual behavior.
00:09:44.640
We find the test, and the setup looks completely unreadable.
00:09:50.679
So what do I do first? Well, I first do a Git blame and find out who wrote it, and then I realize it was me five years ago.
00:10:03.200
This has happened many times.
00:10:11.600
Then I take WebMock and change my test; the setup is much simpler.
00:10:19.280
Now I can change my code and don't have to use the complicated Net::HTTP interface.
00:10:25.920
Instead, I can use an HTTP gem that has a nice DSL for making requests.
00:10:34.400
WebMock is great for test-driven development (TDD). You can write your tests first, focusing on the expected behavior.
00:10:41.360
You don't have to think about which library you are going to use or what methods you are going to call.
00:10:49.200
You just want to define the expected behavior, such as what HTTP requests will be made and what responses will be returned.
00:10:55.920
You can think about the implementation later.
00:11:04.920
WebMock is also very useful for legacy applications.
00:11:11.600
One of the first things I would do when taking over some legacy code is to import WebMock and enable it.
00:11:20.560
Now I'm safe; I know that when I run the app in development or run tests, it won't make unexpected requests to a third-party API without my knowledge.
00:11:29.840
If it makes any requests, WebMock will alert me.
00:11:36.800
WebMock allows me to define request stubs with unescaped versions of URLs.
00:11:45.680
I find this feature very useful for having credible tests; escaped URLs are much harder to read.
00:11:54.800
Another feature I really like about WebMock is the stubbing instructions.
00:12:02.200
When you have a request stub that doesn't match, for any reason—whether the request stub is wrong or the wrong request is executed—WebMock will intercept it.
00:12:10.720
It raises an error and prints instructions telling you how to stub the request.
00:12:17.360
It also tells you what the currently registered request stubs are.
00:12:25.200
Basically, I created WebMock to simplify things for myself, and it also makes things simpler for others.
00:12:35.200
But with all this DSL, comes some magic.
00:12:42.560
You see the DSL declaration, but you lose control; you don’t know what actually happens underneath.
00:12:55.840
If you are like me, you like to understand how things work underneath.
00:13:05.040
About two months ago, I went to a presentation by Javier, who explained Sidekiq internals.
00:13:13.200
I found it very insightful and thought, 'Okay, I'll do the same for WebMock. I'll explain how WebMock works underneath.'
00:13:20.640
However, to understand how WebMock works internally, we first need to understand how HTTP clients in Ruby work.
00:13:30.960
Let's take a sample HTTP client in Ruby.
00:13:39.680
It usually has some public interface and public methods that represent HTTP verbs.
00:13:48.640
These methods are usually very simple; they don’t do much.
00:13:56.720
They typically call some private methods that do all the heavy lifting.
00:14:05.200
So let's look at how that method might be structured and what it does.
00:14:16.000
First, it will build the request, parse the headers, and add any default headers.
00:14:23.800
Then it creates the complete request object.
00:14:30.840
If the HTTP client supports it, it might run some request middleware on it.
00:14:38.000
Once all the request data is complete, it will send it through a TCP or SSL connection.
00:14:44.480
When a response is received from that connection, it might run some response middleware on it before returning the final response.
00:14:52.480
That's how a typical HTTP client works.
00:15:01.680
Of course, there are many differences; for example, there are asynchronous clients.
00:15:06.720
Some clients use C bindings instead of talking directly to TCP or SSL connections.
00:15:14.800
However, more or less, this is how a typical HTTP client in Ruby works.
00:15:24.640
Now that we know how HTTP clients in Ruby work, we can understand how WebMock works internally.
00:15:34.800
Let's first focus on stubbing and what happens when we call stub request.
00:15:41.680
If we have a stub request declaration, WebMock will create a request stub object and register it inside the global stub registry.
00:15:49.760
The request stub object consists of a request pattern that is used to match requests and a collection of fake responses that will be returned if a matching request is detected.
00:15:57.760
The stub request method returns the request stub itself, allowing us to chain additional calls.
00:16:07.360
For example, we can call 'with' to extend the request pattern with additional criteria.
00:16:16.480
Now our request stub will only match requests that have a body of 'F' and a content type of 'JSON' header.
00:16:24.080
Again, our request stub is being returned, which allows us to call the 'to_return' method to create a fake response object and associate it with the request stub.
00:16:33.599
Now, regarding registering callbacks, it’s quite simple: we pass a block, and that block gets registered in the global callback registry.
00:16:40.160
So now we have this global stub registry with registered request stubs and a registered callback.
00:16:48.480
But nothing actually changes right now; no requests are being intercepted.
00:17:01.160
That’s where WebMock uses adapters.
00:17:10.160
WebMock has an implementation adapter for each of the HTTP clients in Ruby; one adapter per client.
00:17:18.320
Let's implement an adapter for our sample HTTP client.
00:17:27.080
First, it will create a copy of the handle request method, which does all the heavy lifting.
00:17:35.360
WebMock will take the class that contains that method and create a copy of it by subclassing.
00:17:43.000
Then it will start monkey patching the class.
00:17:51.200
After the request is built, it will create a request signature and register that signature in the request registry.
00:18:00.800
This signature contains all required information to match the request.
00:18:07.680
Next, it checks the stub registry to see if there is any request stub matching the request.
00:18:16.160
If there is, it will return a fake response; if not, it will make a real request.
00:18:24.320
However, WebMock blocks all real connections by default.
00:18:32.480
So we have to extend the handle request method to allow making the real request if specifically configured.
00:18:41.680
If not, we raise an error, which will give us a nice stubbing instruction when the response is returned.
00:18:54.160
We also run any registered callbacks and then run any middleware supported by that HTTP client before returning the response.
00:19:02.560
Now we have a copy of a class that has been instrumented by monkey patching, but it still doesn't change the original behavior.
00:19:09.760
For instance, if I call sample HTTP client get, it will still call the original class and make the real request.
00:19:19.760
The adapter needs to do something more.
00:19:26.960
Let’s go over how handle request works. First, it builds the request, creates the request signature, and registers it in the request registry.
00:19:36.240
If a matching request stub is found, a fake response is used.
00:19:45.760
If a real request is allowed, we’ll make a request and return a real response.
00:19:53.920
Otherwise, we raise an error and print stubbing instructions.
00:20:02.240
Finally, we call any registered callbacks, response middleware, and return the response.
00:20:12.080
So, how does WebMock affect the original sample HTTP client class?
00:20:20.080
First, it creates a reference to the original class, which will be useful in the future.
00:20:30.680
Then, it replaces the original class using constant replacement.
00:20:40.280
Thus, the client constant will now point to our monkey-patched version.
00:20:49.760
While monkey patching can be problematic, I find this solution neat since we are not modifying the original class.
00:20:59.440
If needed, we can always revert all changes by doing constant replacement and restoring the original class.
00:21:10.080
When we enable WebMock, this will enable the adapter, replacing the original class with our monkey-patched class.
00:21:18.560
When we call stub request, it registers the stub in the stub registry with a fake response declared earlier.
00:21:26.320
Once we execute the request, it will return a fake response from the registry.
00:21:34.800
Now we can proceed further. Remember the request signatures we registered in the handle request method?
00:21:42.080
We can verify that those requests were executed.
00:21:50.760
We create a request pattern and check the request registry to find if there is any request signature matching that pattern.
00:22:02.120
This is how WebMock works.
00:22:10.480
I want to share something special and very important to me.
00:22:17.440
I created WebMock to address my own needs as a Ruby developer.
00:22:29.680
I created WebMock during a company hackathon.
00:22:37.520
But it wasn’t just any company and not with any people.
00:22:46.160
Back in 2009, I lived in London and worked for a company called New Bamboo.
00:22:54.080
New Bamboo comprised a team of passionate developers.
00:23:01.280
We called ourselves 'Bambinos,' and I believe each of us was very proud to be part of that team.
00:23:08.560
We shared our passion, motivated each other, inspired one another, supported each other, and encouraged one another.
00:23:15.680
During my over 20 years of building software, I have learned that the most important thing for your growth is the people you work with and what you learn from them.
00:23:24.240
When L. Stal said that Ruby people are strange, I believe he was right.
00:23:29.840
To me, Ruby has always been not just about writing nice code or using Rails.
00:23:36.560
It was mainly about the people—those who shared their passion, inspired each other, and were friendly.
00:23:43.200
This event, this conference, WrocLove.rb, is the essence of that.
00:23:51.000
I would very much like to thank everyone who organizes WrocLove.rb, or has ever organized it.
00:23:59.920
What you all are doing is really important for our community.
00:24:07.640
Thank you.
00:24:14.640
All right, do we have any questions?
00:24:18.080
Sorry, just ice breaker. Okay, no one? Alright, oh, there is one.
00:24:22.160
First, I just want to thank you because I've been using WebMock since the first version you pushed.
00:24:32.080
As long as I can remember, I've been using WebMock and still use it today, so that is great.
00:24:42.200
I want to mention something very important: it's crucial to know when a mock is no longer valid.
00:24:50.240
In the applications I’ve used, we’ve been doing contract testing to know when things changed and when the mocks had to be updated.
00:25:00.560
I wonder if you have thought about or tried to build anything for contract testing alongside WebMock or as a separate library?
00:25:08.240
No, I haven't. However, the scope of validation is something that is consistently on my mind.
00:25:15.680
The fact that WebMock uses global registries is a problem because you either have to reset them or keep using them.
00:25:24.560
It's not trivial to say, 'I want this mock to only be valid within this scope or this thread.'
00:25:30.960
I hear from time to time that users want to reduce this scope.
00:25:38.399
I hope that some major next version of WebMock will support this.
00:25:44.960
Thank you.
00:25:52.760
In other words, Bartosz is saying he needs contributors.
00:25:54.328
There's one more question. Thanks for a great presentation.
00:26:07.760
I wonder, because WebMock cooperates with various HTTP libraries, how do you keep track of the changes in those libraries?
00:26:16.560
How do you ensure that WebMock correctly monkey patches them?
00:26:30.440
WebMock tries to be up to date with the latest versions.
00:26:37.560
The WebMock gemfile doesn’t have hardcoded versions or any restrictions.
00:26:42.640
The tests will always update to the latest version of a gem when testing.
00:26:52.160
This means that some older versions may no longer work with WebMock.
00:26:59.520
WebMock always tries to stay up to date with the latest HTTP gem version.
00:27:04.080
However, there is no automated system for that.
00:27:08.720
Usually, I learn about incompatibilities either by rerunning the test suite or through users raising issues.
00:27:15.920
Often, users first open an issue in the HTTP client repository.
00:27:23.320
Then, the maintainer of that specific gem submits a pull request, or someone from the contributors raises an issue.
00:27:31.920
These types of issues are given priority because they affect people.
00:27:39.760
This is an emergency.
00:27:40.800
Okay, there’s one more chance to ask a question.
00:27:51.760
I see no one. Okay, ladies and gentlemen, Bartosz Blimke once again.