00:00:14.680
Hi everybody, my name is Kacper Walanus and I am a Ruby on Rails developer.
00:00:19.900
It all started seven years ago when I talked with my friend who was a developer. I told him that I had a lot of free time because I was studying and that I wanted to try coding. Programming sounded like fun, so I asked him if he had any programming language recommendations.
00:00:32.169
He answered, 'Try Ruby. Ruby is cool.' That's how I became a Ruby on Rails developer. After seven years, it is still my primary programming language. Currently, I work for a company called Oxford Evaluation Partners, which calculates the value of other companies based on data they send to Oxford. Essentially, we deal with financial calculations.
00:01:08.500
Today, I want to talk to you about consumer-driven contracts, or the CDC pattern. This pattern comes from Martin Fowler, and the concept is outlined in a rather comprehensive paper that can be a bit complex to grasp. Therefore, I am going to show you an implementation of the CDC pattern.
00:01:35.729
I hope this will help you understand the whole idea behind this pattern and how useful and powerful it can be in service-oriented architecture systems. So let's say we are building a system with service-oriented architecture, and we have two simple services: an API and a web client.
00:02:08.950
The web client relies on the API completely. It doesn't have its own database, so I will refer to the API as the provider and the web client as the consumer, simply because the API provides the data and the consumer uses it.
00:02:54.250
In this example, the consumer retrieves data from the provider in JSON format to display information about users in the browser. Both services have tests. Let's start with the API tests—it's a very straightforward one for users' calls.
00:03:22.390
What we typically do is expect that the response will be of a specific type of data. So, I expect that the response for this call is an array of objects. I expect that these objects have specific keys and values. It's quite simple. This test may not be perfect, but you get the idea.
00:03:50.560
Next, let's talk about the web client’s consumer tests. Our web client is a consumer, which means it cannot operate without its provider; it must have an API. To test the consumer properly, we obviously need some kind of API in the testing process. Otherwise, it wouldn't make sense.
00:04:21.430
However, we don't want to and shouldn't use an actual provider for that for obvious reasons. Therefore, we need some sort of imitation or simulation of the API when testing our consumer.
00:04:55.870
The most popular way to do this is to stub HTTP responses. We know that our application needs to send requests during testing, so we prepare all required responses in advance. This way, we can serve them whenever needed. For example, I'm testing this table with users, so I need to stub this user's call by hardcoding the response.
00:05:32.889
Then, I can verify that the HTML rendered based on this hardcoded response is correct. A quick note: writing these stubs is kind of like writing an API because if you gather all the steps together, they will create an API—albeit a strange one where all responses are hardcoded.
00:06:02.289
But still, it would be a web service that can respond to HTTP requests. So this is our system, and it's doing great. The consumer tests passed, the provider tests passed, and the whole system is working nicely.
00:06:23.440
Now, let's imagine that the API team just released the newest version of their software, and this new version returns user data not as an array, but as a single object.
00:06:46.870
Unfortunately, they forgot to inform the web client team about this small change. In reality, this would likely never happen because nobody is that irresponsible. But in this scenario, as a result of this honest mistake, production crashes.
00:07:09.790
Now, it's not a problem that production crashed, as it can be fixed within two minutes. The real issue is that it is likely to happen again and again because there is nothing preventing it.
00:07:40.300
Another reason this situation is somewhat terrifying is because our tests are still green. The production crashed, but if we look at the tests, everything seems fine. This is frightening because, despite having good test coverage, it turns out that our services' tests tell us nothing about the system as a whole.
00:08:09.820
Once again, the parts of the system are properly tested and work as expected from the developer's point of view. However, the thing that doesn't work is the collaboration of all these components to create the complete system.
00:08:34.710
One could say this is a simple compatibility problem: just update the web client and everything will be fine. However, it's not that simple. I can prove this by asking a simple question: why should the web application be the one that is updated?
00:09:05.060
We can assume the API team did the release for a reason, but maybe the web app team is the one that didn't follow the requirements. It could also be that this change wasn't supposed to be in this release or it's an old requirement that is no longer valid.
00:09:56.760
The point is that we don’t know. We cannot decide. Perhaps the consumer should be updated, but perhaps the provider's changes should be reverted. We cannot tell. Sometimes we have documentation, and if we do, we can use it to clarify this question.
00:10:40.820
You can point to it and say, 'Right here, it clearly states that it has to be an array, not an object.' This would allow us to rectify the situation.
00:11:16.210
In theory, it's a very good approach to have a set of expectations about the system, which all players should follow. In fact, this is the foundation of the CDC pattern, which focuses on maintaining these expectations as a contract between the provider and its consumers.
00:11:54.680
The CDC pattern doesn’t specify the implementation of contracts. Writing them down on paper is just one possible implementation. The problem with documentation is that it doesn’t come alive or light up when it is not being followed. Rules are easy to ignore if they exist only as documents.
00:12:36.490
So, we need to find a better implementation—essentially a more dynamic form of documentation that cannot be so easily ignored. This brings us to the problem of keeping system parts compatible with each other. I propose immediately notifying all system parts about any compatibility-breaking updates.
00:13:38.909
If we do not notify developers about the changes, they may not know how to handle the updates. Let’s return to the consumer tests, which are currently working well.
00:14:19.989
As long as there is something responding to the application's requests, everything runs smoothly. However, if I stop stubbing responses and start using an actual HTTP server, the application won’t even notice because it is still receiving a response.
00:15:05.230
So right now, I’m going to remove all stubs from my test suite and build a new API on top of them.
00:15:41.660
Henceforth, whenever I start testing, I will start a real HTTP server with this API, and this server will respond to all requests made by the application during testing.
00:16:21.500
Here is how it can be done. It’s an API that handles one request, written with Sinatra, and it responds to the users’ call by returning JSON we’ve seen before. There’s nothing magical about that.
00:17:03.210
The only thing left is to set up my test environment to use this API just like the real one. Thanks to the library WebMock, it can be done easily with one line. This stub request line tells the framework to intercept all requests and redirect them to the specified rack application, which in my case is the new API.
00:17:38.420
Now we are testing the consumer against this new external service, meaning whenever this service changes, my application must also change. For instance, if the server starts serving objects instead of arrays, my tests will fail.
00:18:17.890
So, they are tightly bound together and must always remain compatible. Otherwise, the system will show a red flag.
00:18:59.490
Of course, this new API is supposed to imitate the real API. Otherwise, this methodology wouldn’t make any sense. But if that is the case, it means we have just coded our expectations about the provider in our contract API.
00:19:25.330
We expect arrays, not objects. This expectation is now written in my new API. In other words, we have just created a consumer-driven contract in the form of an API.
00:19:49.790
Let’s call it the contract API from now on. However, maintaining another service doesn’t inherently make our lives easier. Instead, it complicates things.
00:20:26.830
So why would we do that? What’s the benefit? The answer is that with this contract API, we can also test the provider against it.
00:20:52.909
Think about it: the contract API is designed to be very similar to the real API. Now, I can test my real API; I will send a request to the contract API, then send the same request to the real one, and compare their responses.
00:21:46.630
If they don’t match, it signifies that something is wrong. It could mean that the provider does not fit the contract anymore or perhaps there are new expectations that have made the contract outdated and thus needs an update.
00:22:17.640
Now, this is our API test from before. This test contains information about the expected JSON. It has to be an array and the objects within it must have specific values, and so forth.
00:22:43.500
But now I have the same information coded in my contract API, which creates duplication. We don’t like duplication.
00:23:08.960
So we want to keep all expectations about our system in one place, and the contract is the perfect candidate for that.
00:23:36.200
I’m going to remove this redundancy from the API test because all the required information is now contained in my contract API.
00:24:10.000
Yes, that's right! This API test can look like this now. I only need to send a request to the real API and send a request to the contract API, then compare the responses. As I said before, if they don’t match, let’s let the test fail—because something is wrong.
00:24:46.000
So the problem was to keep this compatibility—keep the system parts compatible with one another, and the solution would be to notify all parts of the system about these breaking changes.
00:25:14.000
Let’s think about what would happen if we succeeded in that.
00:25:53.000
This is the scenario: the API team introduces a compatibility-breaking change. Automation runs its course, and their tests start failing because it doesn’t fit the contract anymore.
00:26:34.000
However, this change is very important; it is required. The people in charge decide that the contract can be updated.
00:27:05.000
As a result, the provider is fine, but this contract update automatically makes all other services' contracts outdated, and their tests now fail.
00:27:48.000
Consequently, the web client team must update their application codebase, which they do, and now everything is running smoothly. Right? Everybody is happy.
00:28:21.000
The key point here is that nobody could deploy before it was fixed because everyone immediately knew that something was wrong.
00:28:59.000
And if there are more services affected, this applies to all of them. All services will fail their tests and need to be updated.
00:29:38.000
It looks like we are really notifying all developers about changes by having their tests fail.
00:30:12.000
However, it’s not ideal. I don’t want to write a server every time I want to create a contract. That’s not very convincing.
00:30:37.000
I would prefer to write a specification, essentially a list of requests and responses, and have some external tool manage turning my documentation into a living service, like Apiary or Swagger.
00:31:05.000
For this purpose, I will use an external toolkit called Contractor. So let’s explore how the CDC pattern can be applied in a real-world scenario.
00:31:44.000
Here is this Contractor organization; there are many repositories, but we are interested in the Contractor server and Contractor format. Contractor server is a gem that can transform documentation written in this contractor format into a functional server.
00:32:16.000
Let’s say I have my system: on the left side, I have my consumer that responds with JSON. I can ask for a single user, and this is my web client.
00:32:52.000
I want to test this page with a single user. We are now focusing on the consumer tests, so I need a contract that describes a call for a single user.
00:33:22.000
This view requires a call for a single user, or at least that’s how I designed it. So, I need this call documented.
00:34:04.000
This is my contract, and at first glance, you can see this repository contains files with a .contractor JSON extension. Based on the names, I can tell that each file describes a separate call. I am specifically interested in the call for a single user.
00:34:42.000
This JSON indicates where and how to send the request. I know it must be a GET request sent to the /user/someID path.
00:35:29.000
I also want to inspect the possible responses, which may vary based on certain conditions. For example, I want different responses for users with ID 1 compared to user ID 2.
00:36:08.000
This JSON shows the expected response: for ID 1, I should receive a specific response body.
00:36:35.000
Next, I’ll check the corresponding file containing this response body.
00:37:06.000
Here I am, in the user directory where this file is located, and we can see that I’ve already described responses associated with different calls.
00:37:31.000
Thus, I expect this response if I send the call to /user/1. Once I have my contract set up, what I need to do now is to type 'contractor contract' to start it.
00:38:08.000
This command initiates the server, and now I need to pass the URL of my contract. I’ll copy my URL and execute it.
00:38:59.000
We see that the contract has been successfully downloaded from GitHub, and it informs me that the server is functioning. I can now make a request to a user by using the localhost.
00:39:40.000
I can also check for other user responses since there were three files, meaning I have three possible responses available.
00:40:24.000
This is essentially what the contractor server gem accomplishes: turning JSON into a server.
00:40:59.000
I can now employ it to test my consumer. This specific test involves checking the details for a single user along with a separate test for all users.
00:41:35.000
I have already added the contractor spectrum, which is a helper for implementing contractor in RSpec. This allows me to issue the command that tells the contractor to start a server before executing the tests.
00:42:20.000
Naturally, I have to specify where my contract needs to be downloaded from; it can also be located on the local machine.
00:42:54.000
Next, I will set up the endpoint for my API in the test. The application will operate as expected, with a port now designated as 3051 instead of the usual 3000.
00:43:36.000
Let me check if everything is functional. I will start my spec. The contractor gets downloaded successfully, the server starts, and then my tests pass.
00:44:14.000
However, it's not enough just to pass for proof. I will modify something in my contract.
00:44:33.000
Okay, I will change the response so that it returns my surname instead of 'Einstein'.
00:45:05.000
Let’s see what happens once I update it. I can run my tests again.
00:45:37.000
Of course, now it fails because it expects 'Einstein', but it's being set to my surname.
00:46:08.000
Now we can almost confirm that it actively utilizes the repository.
00:46:44.000
However, let’s move on to the provider tests. Remember, my contract is still broken due to the recent change.
00:47:30.000
Therefore, provider tests should also fail. The real API still uses 'Einstein' in its test data.
00:48:10.000
I will proceed to fix the issue.
00:48:54.000
After this update, both services' tests should pass. I am currently testing the provider once more, and yes, it works.
00:49:38.000
I think you can see the usefulness of this setup. Let me show you something more interesting.
00:50:06.000
With my integration system is working properly, I've noticed that whenever I update my specification, it triggers tests for each service immediately.
00:50:43.000
This is a very useful feature because it ensures that everyone is informed of important changes in the contract.
00:51:16.000
This information is crucial to the evolution of our system.
00:51:56.000
I am confident that I have incorporated this functionality into my work.
00:52:29.000
Now, let’s check the status of the webhooks in action.
00:53:07.000
It should fail due to the expected response no longer matching the required output.
00:53:44.000
Indeed, it seems the API expectations are misaligned.
00:54:22.000
This shows the reason and importance behind safeguarding consumer-driven contracts.
00:55:04.000
If any of our expectations break, the required updates must be enforced.
00:55:36.000
It’s imperative that everyone remains aware of these changes if they want to avoid failure.
00:56:04.000
Let’s address your feedback, as you have provided valuable insights throughout.
00:56:44.000
To summarize, consumer-driven contracts enhance interaction and create a more stable ecosystem for applications to communicate.
00:57:24.000
The crucial take-away is that ensuring consumer-driven contract maintenance benefits all parts of the development pipeline.
00:58:00.000
Questions?
00:58:42.000
In real life, it’s important to update both consumers and providers simultaneously. I imagine first you would need to update your consumer to support both the old and new versions of the API.
00:59:22.000
After doing that, the provider can be updated to enable the new version.
00:59:48.000
Undoubtedly, that’s the best way to avoid complications when you have multiple versions of an API.
01:00:20.000
This process helps minimize risks and streamline transitions.
01:00:39.000
Thank you for your engaging presentation! Could you name a tool that generates popular API documentation formats, like Swagger?
01:01:22.000
Actually, Contractor and Pact are quite simple products.
01:01:46.000
They don’t aim to replace existing formats, but rather to fulfill their own essential roles. However, there is a valid point regarding the utilization of existing API documentation tools.
01:02:29.000
One limitation of conventional formats is that they often don’t accommodate dynamic scenarios.
01:03:08.000
Contractor allows flexible responses depending on passed parameters, which isn’t typically possible with Swagger.
01:03:51.000
That’s significant because testing becomes easier. You want to test numerous users with diverse datasets.
01:04:41.000
I really appreciate your insights!
01:05:02.000
Could you share who has the authority to alter the contract?
01:05:42.000
The concept of consumer-driven contracts suggests that consumers should have a voice.
01:06:09.000
Previously, providers created contracts, and consumers had no room to negotiate.
01:06:33.000
With this new mindset, each consumer can establish their own contracts since their focus is on the API calls they require.
01:07:11.000
Moreover, collaboration is necessary, and the goal is to empower consumers.
01:07:36.000
As for the constraint involving an API offline during specific hours, I'm unsure how to implement that.
01:08:12.000
However, while some tools are limited in this regard, the flexibility of contractor allows for localized handling of requests.
01:08:42.000
Yet, I cannot offer a robust solution for that particular problem.
01:09:05.000
Could you comment on how to effectively inform mobile developers of contract changes?
01:09:58.000
Employing tools like Retrofit can be beneficial as it parses contractor formats.
01:10:23.000
It works well for Android developers validating their response schemas.
01:11:05.000
Thanks for sharing valuable information about using consumer-driven contracts, and thank you all for your questions.