Ruby
Webmock Unmocked
Summarized using AI

Webmock Unmocked

by Bartosz Blimke

In this presentation titled "Webmock Unmocked," speaker Bartosz Blimke discusses the Ruby gem WebMock, which is used for mocking HTTP requests. With over 20 years of software development experience and as the creator and maintainer of WebMock, Blimke shares insights into the gem's functionalities and its contributions to the Ruby community.

Key points discussed include:

- Introduction to WebMock: WebMock is designed to stub HTTP requests and verify their execution without making real network calls. This ability allows developers to simulate responses in their tests, ensuring that the behavior of their applications can be reliably tested without dependency on external services.

- Development Journey: Blimke created WebMock during a hackathon when he found limitations with the existing gem FakeWeb. Since its inception, WebMock has been widely adopted, being downloaded over 250 million times and seeing contributions from over 260 developers.

- Functionality and Use Cases: The core functionalities include:

- Stubbing HTTP requests with fake responses

- Verifying that certain requests have been made

- Support for popular HTTP clients and testing frameworks

- Features allowing fast, stable testing environments, regardless of third-party service outages

- Methods for managing request stubbing without an internet connection, which facilitates working offline and during development.

- Implementation Details: Blimke explains the internal workings of WebMock, including how it creates and registers request stubs and integrates with various Ruby HTTP clients via monkey patching. He outlines how these strategies enable WebMock to intercept requests and facilitate the test-driven development process.

- Community and Contributions: Blimke emphasizes the importance of collaboration in development, sharing his gratitude towards the community and encouraging developers to publish their solutions. He highlighted how WebMock has inspired similar tools in other programming languages.

- Future Considerations: During the Q&A, he acknowledges challenges regarding the validity of mocks and keeping WebMock up-to-date with changes in HTTP libraries, expressing a desire to address these issues in future versions.

In conclusion, Blimke from his extensive experience and knowledge of Ruby emphasizes the utility of WebMock in providing stable, offline testing environments, the significance of community contributions, and the continuous evolution of software development tools.

00:00:12.120 Today I'm going to talk to you about WebMock, a Ruby gem for mocking HTTP requests.
00:00:20.480 First, a little bit about me.
00:00:26.560 I've been developing software for over 20 years. Back in 2006, when I was still a Java developer, my friend Andre KDA came to me.
00:00:34.079 He asked, 'Hey, have you heard of that new language called Ruby?' I've been using Ruby ever since.
00:00:39.200 Professionally, I've been using Ruby since 2007. I live in Wroclaw but have been working remotely since before it was trendy.
00:00:47.079 I'm the author and the sole maintainer of WebMock.
00:01:01.600 Today, I'm a little stressed because this is my first presentation in 14 years.
00:01:08.920 After seeing all these great presentations today, I feel the bar has been raised.
00:01:16.400 Being the final speaker of the day is a unique experience.
00:01:23.159 This is also my second presentation about WebMock; the previous one was in London 15 years ago. I was still stressed then, and nothing has changed.
00:01:35.720 After seeing David Halas speak yesterday about formal languages, I thought I should add a slide about something I learned at my university.
00:01:41.560 This led me to think about extrapolation, so I extrapolated when my next talk about WebMock would be.
00:01:48.799 We have a unique chance to learn something about WebMock today.
00:02:04.119 So, what is WebMock? WebMock has two core functionalities.
00:02:10.000 First, it allows you to stub HTTP requests. This means you can declare fake responses so that when your application attempts to make an HTTP request, no connection is made, and instead, a fake response is returned.
00:02:21.480 Second, it allows you to verify that HTTP requests have been executed, whether they are stubbed or real.
00:02:32.959 The idea is not original; there was a gem called FakeWeb before. I tried to use it, but its request matching capabilities were too limited, and it only supported Net::HTTP.
00:02:45.919 Therefore, I decided to write my own solution and created WebMock during a hackathon.
00:02:55.440 The first version of WebMock was developed to address my own needs as a Ruby developer.
00:03:02.040 These needs turned out to be the needs of many other Ruby developers as well, and hence, WebMock has been downloaded more than 250 million times up until now.
00:03:15.799 It has almost 4,000 stars on GitHub; by the way, if you haven't starred it yet, please do. I'm aiming for 4,000.
00:03:23.280 Over 260 people have contributed to WebMock over the last 15 years, and I would very much like to thank everyone who has ever contributed.
00:03:30.800 WebMock is used by thousands of public projects on GitHub, and almost 8,000 registered gems depend on it.
00:03:37.440 Some of the projects that depend on WebMock are listed here, and you might have heard of or used some of them.
00:03:46.800 If you haven't had a chance to use any of them, you probably have used this one. Machi mentioned today that if you know something, you should publish it.
00:03:56.799 So I want to encourage you, if you have a solution that works for you, to publish it. It might be useful to others.
00:04:05.680 WebMock has also inspired developers in other languages, leading to the creation of solutions inspired by WebMock.
00:04:12.360 I've even seen the term 'web mocking' used to refer to mocking HTTP requests.
00:04:20.448 So, what can you do with WebMock? First, WebMock provides a nice domain-specific language (DSL) for declaring request stubs.
00:04:28.600 It is easily readable and understandable. Second, it offers a DSL for verifying that HTTP requests have been made. Both DSLs are HTTP client agnostic, so you can use them with any HTTP client.
00:04:46.680 WebMock follows the stub, execute, verify paradigm, which I borrowed from a Java testing framework called Mockito.
00:05:03.680 It separates the stubbing phase from the verification phase, which might be different from frameworks like RSpec, where a single expect declaration defines both the stub and the expectation.
00:05:12.960 In WebMock, the steps are separated, allowing you to stub without verifying or verifying without stubbing.
00:05:21.200 WebMock offers request callbacks, and with these callbacks, you can record requests.
00:05:29.200 There is a gem called VCR for recording HTTP requests, which uses WebMock under the hood.
00:05:37.680 WebMock offers many features for stubbing; for example, you can stub requests to timeout or with basic authentication.
00:05:46.080 You can even replay raw responses recorded with curl.
00:05:53.200 WebMock supports all popular HTTP clients in Ruby and all major testing frameworks.
00:06:03.680 But then you might ask, why mock HTTP requests at all? Why should we care?
00:06:15.760 First, it allows you to run your tests without needing an internet connection.
00:06:24.160 For anyone, like me, who arrived here by train and has been working on a train, this is a useful feature.
00:06:41.440 If you misconfigure your testing environment and accidentally use production credentials, with WebMock, your tests won't make any delete requests on a production third-party API.
00:06:56.200 You always get a stable environment for running your tests.
00:07:04.000 Even if a third-party service is down or temporarily returns 500 responses, your tests will still go through and pass.
00:07:15.000 You also get fast tests, as you don't have to wait for any API delays—your responses are instant.
00:07:24.400 You can also test various scenarios, such as different status codes and error responses, and see how your app reacts to them.
00:07:36.200 All this test setup is done in Ruby files.
00:07:46.200 Mocking HTTP requests is not only useful for testing environments; you can also define your request stubs in the development environment and work offline.
00:07:56.200 You can set request stubs for various scenarios and test your app manually while simulating different situations.
00:08:06.200 One feature that I have used many times in the past is that it allows you to develop without access to a third-party API.
00:08:14.200 For example, I once worked on a project that heavily relied on a third-party API.
00:08:23.760 We received access to it only after a month and a half.
00:08:33.600 In the meantime, we read the documentation and API specifications, created some stubs, and were ready to go.
00:08:40.200 You can do the same in your development environment to demo the features that have been implemented, even before you have access to the external API.
00:08:49.640 Now, let me show you a sample Rails initializer for development and staging.
00:08:56.200 So, we know that stubbing HTTP requests can be useful at times.
00:09:05.000 But why would we use WebMock instead of creating our own solution?
00:09:15.200 Let's say we find a piece of code in our app that makes an HTTP request with basic authentication.
00:09:25.120 We are going to make a change to it, so we search for a test because we do test-first development, and we change the test first before changing the actual behavior.
00:09:44.640 We find the test, and the setup looks completely unreadable.
00:09:50.679 So what do I do first? Well, I first do a Git blame and find out who wrote it, and then I realize it was me five years ago.
00:10:03.200 This has happened many times.
00:10:11.600 Then I take WebMock and change my test; the setup is much simpler.
00:10:19.280 Now I can change my code and don't have to use the complicated Net::HTTP interface.
00:10:25.920 Instead, I can use an HTTP gem that has a nice DSL for making requests.
00:10:34.400 WebMock is great for test-driven development (TDD). You can write your tests first, focusing on the expected behavior.
00:10:41.360 You don't have to think about which library you are going to use or what methods you are going to call.
00:10:49.200 You just want to define the expected behavior, such as what HTTP requests will be made and what responses will be returned.
00:10:55.920 You can think about the implementation later.
00:11:04.920 WebMock is also very useful for legacy applications.
00:11:11.600 One of the first things I would do when taking over some legacy code is to import WebMock and enable it.
00:11:20.560 Now I'm safe; I know that when I run the app in development or run tests, it won't make unexpected requests to a third-party API without my knowledge.
00:11:29.840 If it makes any requests, WebMock will alert me.
00:11:36.800 WebMock allows me to define request stubs with unescaped versions of URLs.
00:11:45.680 I find this feature very useful for having credible tests; escaped URLs are much harder to read.
00:11:54.800 Another feature I really like about WebMock is the stubbing instructions.
00:12:02.200 When you have a request stub that doesn't match, for any reason—whether the request stub is wrong or the wrong request is executed—WebMock will intercept it.
00:12:10.720 It raises an error and prints instructions telling you how to stub the request.
00:12:17.360 It also tells you what the currently registered request stubs are.
00:12:25.200 Basically, I created WebMock to simplify things for myself, and it also makes things simpler for others.
00:12:35.200 But with all this DSL, comes some magic.
00:12:42.560 You see the DSL declaration, but you lose control; you don’t know what actually happens underneath.
00:12:55.840 If you are like me, you like to understand how things work underneath.
00:13:05.040 About two months ago, I went to a presentation by Javier, who explained Sidekiq internals.
00:13:13.200 I found it very insightful and thought, 'Okay, I'll do the same for WebMock. I'll explain how WebMock works underneath.'
00:13:20.640 However, to understand how WebMock works internally, we first need to understand how HTTP clients in Ruby work.
00:13:30.960 Let's take a sample HTTP client in Ruby.
00:13:39.680 It usually has some public interface and public methods that represent HTTP verbs.
00:13:48.640 These methods are usually very simple; they don’t do much.
00:13:56.720 They typically call some private methods that do all the heavy lifting.
00:14:05.200 So let's look at how that method might be structured and what it does.
00:14:16.000 First, it will build the request, parse the headers, and add any default headers.
00:14:23.800 Then it creates the complete request object.
00:14:30.840 If the HTTP client supports it, it might run some request middleware on it.
00:14:38.000 Once all the request data is complete, it will send it through a TCP or SSL connection.
00:14:44.480 When a response is received from that connection, it might run some response middleware on it before returning the final response.
00:14:52.480 That's how a typical HTTP client works.
00:15:01.680 Of course, there are many differences; for example, there are asynchronous clients.
00:15:06.720 Some clients use C bindings instead of talking directly to TCP or SSL connections.
00:15:14.800 However, more or less, this is how a typical HTTP client in Ruby works.
00:15:24.640 Now that we know how HTTP clients in Ruby work, we can understand how WebMock works internally.
00:15:34.800 Let's first focus on stubbing and what happens when we call stub request.
00:15:41.680 If we have a stub request declaration, WebMock will create a request stub object and register it inside the global stub registry.
00:15:49.760 The request stub object consists of a request pattern that is used to match requests and a collection of fake responses that will be returned if a matching request is detected.
00:15:57.760 The stub request method returns the request stub itself, allowing us to chain additional calls.
00:16:07.360 For example, we can call 'with' to extend the request pattern with additional criteria.
00:16:16.480 Now our request stub will only match requests that have a body of 'F' and a content type of 'JSON' header.
00:16:24.080 Again, our request stub is being returned, which allows us to call the 'to_return' method to create a fake response object and associate it with the request stub.
00:16:33.599 Now, regarding registering callbacks, it’s quite simple: we pass a block, and that block gets registered in the global callback registry.
00:16:40.160 So now we have this global stub registry with registered request stubs and a registered callback.
00:16:48.480 But nothing actually changes right now; no requests are being intercepted.
00:17:01.160 That’s where WebMock uses adapters.
00:17:10.160 WebMock has an implementation adapter for each of the HTTP clients in Ruby; one adapter per client.
00:17:18.320 Let's implement an adapter for our sample HTTP client.
00:17:27.080 First, it will create a copy of the handle request method, which does all the heavy lifting.
00:17:35.360 WebMock will take the class that contains that method and create a copy of it by subclassing.
00:17:43.000 Then it will start monkey patching the class.
00:17:51.200 After the request is built, it will create a request signature and register that signature in the request registry.
00:18:00.800 This signature contains all required information to match the request.
00:18:07.680 Next, it checks the stub registry to see if there is any request stub matching the request.
00:18:16.160 If there is, it will return a fake response; if not, it will make a real request.
00:18:24.320 However, WebMock blocks all real connections by default.
00:18:32.480 So we have to extend the handle request method to allow making the real request if specifically configured.
00:18:41.680 If not, we raise an error, which will give us a nice stubbing instruction when the response is returned.
00:18:54.160 We also run any registered callbacks and then run any middleware supported by that HTTP client before returning the response.
00:19:02.560 Now we have a copy of a class that has been instrumented by monkey patching, but it still doesn't change the original behavior.
00:19:09.760 For instance, if I call sample HTTP client get, it will still call the original class and make the real request.
00:19:19.760 The adapter needs to do something more.
00:19:26.960 Let’s go over how handle request works. First, it builds the request, creates the request signature, and registers it in the request registry.
00:19:36.240 If a matching request stub is found, a fake response is used.
00:19:45.760 If a real request is allowed, we’ll make a request and return a real response.
00:19:53.920 Otherwise, we raise an error and print stubbing instructions.
00:20:02.240 Finally, we call any registered callbacks, response middleware, and return the response.
00:20:12.080 So, how does WebMock affect the original sample HTTP client class?
00:20:20.080 First, it creates a reference to the original class, which will be useful in the future.
00:20:30.680 Then, it replaces the original class using constant replacement.
00:20:40.280 Thus, the client constant will now point to our monkey-patched version.
00:20:49.760 While monkey patching can be problematic, I find this solution neat since we are not modifying the original class.
00:20:59.440 If needed, we can always revert all changes by doing constant replacement and restoring the original class.
00:21:10.080 When we enable WebMock, this will enable the adapter, replacing the original class with our monkey-patched class.
00:21:18.560 When we call stub request, it registers the stub in the stub registry with a fake response declared earlier.
00:21:26.320 Once we execute the request, it will return a fake response from the registry.
00:21:34.800 Now we can proceed further. Remember the request signatures we registered in the handle request method?
00:21:42.080 We can verify that those requests were executed.
00:21:50.760 We create a request pattern and check the request registry to find if there is any request signature matching that pattern.
00:22:02.120 This is how WebMock works.
00:22:10.480 I want to share something special and very important to me.
00:22:17.440 I created WebMock to address my own needs as a Ruby developer.
00:22:29.680 I created WebMock during a company hackathon.
00:22:37.520 But it wasn’t just any company and not with any people.
00:22:46.160 Back in 2009, I lived in London and worked for a company called New Bamboo.
00:22:54.080 New Bamboo comprised a team of passionate developers.
00:23:01.280 We called ourselves 'Bambinos,' and I believe each of us was very proud to be part of that team.
00:23:08.560 We shared our passion, motivated each other, inspired one another, supported each other, and encouraged one another.
00:23:15.680 During my over 20 years of building software, I have learned that the most important thing for your growth is the people you work with and what you learn from them.
00:23:24.240 When L. Stal said that Ruby people are strange, I believe he was right.
00:23:29.840 To me, Ruby has always been not just about writing nice code or using Rails.
00:23:36.560 It was mainly about the people—those who shared their passion, inspired each other, and were friendly.
00:23:43.200 This event, this conference, WrocLove.rb, is the essence of that.
00:23:51.000 I would very much like to thank everyone who organizes WrocLove.rb, or has ever organized it.
00:23:59.920 What you all are doing is really important for our community.
00:24:07.640 Thank you.
00:24:14.640 All right, do we have any questions?
00:24:18.080 Sorry, just ice breaker. Okay, no one? Alright, oh, there is one.
00:24:22.160 First, I just want to thank you because I've been using WebMock since the first version you pushed.
00:24:32.080 As long as I can remember, I've been using WebMock and still use it today, so that is great.
00:24:42.200 I want to mention something very important: it's crucial to know when a mock is no longer valid.
00:24:50.240 In the applications I’ve used, we’ve been doing contract testing to know when things changed and when the mocks had to be updated.
00:25:00.560 I wonder if you have thought about or tried to build anything for contract testing alongside WebMock or as a separate library?
00:25:08.240 No, I haven't. However, the scope of validation is something that is consistently on my mind.
00:25:15.680 The fact that WebMock uses global registries is a problem because you either have to reset them or keep using them.
00:25:24.560 It's not trivial to say, 'I want this mock to only be valid within this scope or this thread.'
00:25:30.960 I hear from time to time that users want to reduce this scope.
00:25:38.399 I hope that some major next version of WebMock will support this.
00:25:44.960 Thank you.
00:25:52.760 In other words, Bartosz is saying he needs contributors.
00:25:54.328 There's one more question. Thanks for a great presentation.
00:26:07.760 I wonder, because WebMock cooperates with various HTTP libraries, how do you keep track of the changes in those libraries?
00:26:16.560 How do you ensure that WebMock correctly monkey patches them?
00:26:30.440 WebMock tries to be up to date with the latest versions.
00:26:37.560 The WebMock gemfile doesn’t have hardcoded versions or any restrictions.
00:26:42.640 The tests will always update to the latest version of a gem when testing.
00:26:52.160 This means that some older versions may no longer work with WebMock.
00:26:59.520 WebMock always tries to stay up to date with the latest HTTP gem version.
00:27:04.080 However, there is no automated system for that.
00:27:08.720 Usually, I learn about incompatibilities either by rerunning the test suite or through users raising issues.
00:27:15.920 Often, users first open an issue in the HTTP client repository.
00:27:23.320 Then, the maintainer of that specific gem submits a pull request, or someone from the contributors raises an issue.
00:27:31.920 These types of issues are given priority because they affect people.
00:27:39.760 This is an emergency.
00:27:40.800 Okay, there’s one more chance to ask a question.
00:27:51.760 I see no one. Okay, ladies and gentlemen, Bartosz Blimke once again.
Explore all talks recorded at wroc_love.rb 2024
+4