Integration Tests Are Bogus

Talks

Piotr Szotkowski

@chastell

#test-driven-development

#mocking

#ruby

#unit-test

#behavior-driven-development-bdd

#code-quality

#rspec

Integration Tests Are Bogus

by Piotr Szotkowski

In the talk titled "Integration Tests Are Bogus," Piotr Szotkowski discusses the complexities and limitations of integration tests within the Ruby programming community. Szotkowski begins by defining integration tests and comparing them with other types of tests, including unit tests and end-to-end tests. He highlights the importance of different testing levels in achieving reliable software development. The talk emphasizes that integration tests, while useful, can often become problematic and lead to false security due to over-reliance on test doubles which can misrepresent interactions in code.

Key points discussed include:
- Types of Tests: Integration tests check the cooperation among objects, while end-to-end tests assess overall application functionality. Unit tests confirm that individual components behave as expected.
- Testing Strategy: Szotkowski advocates for unit tests because they run quickly and help in isolating behaviors, facilitating efficient debugging and coding practices.
- Command Query Separation: He introduces this principle, which proposes that methods should clearly distinguish between querying data (without side effects) and executing commands (which can change state).
- Risks of Test Doubles: He warns against the excessive use of mocks and stubs, which can lead to ineffectual tests that pass without the real functionality being verified. Integration tests are advised as a necessary check against these pitfalls.
- Verified Doubles: The introduction of the 'Bogus' library is pivotal; it ensures that methods being mocked or stubbed are correctly represented, thereby revealing discrepancies early.
- Mocking and Contract Tests: Szotkowski explains the functionality of mocking calls in testing services and introduces the concept of contract tests to ensure the accuracy of method interactions.

As a conclusion, he stresses the significance of efficient testing practices to foster robust software development. The audience is encouraged to engage with the 'Bogus' library and explore further resources for advanced learning on object-oriented design and testing methodologies in Ruby, reinforcing the idea that effective communication with testing doubles is critical for successful unit tests.

00:00:13.920 As you probably saw yesterday, the next talk after mine was supposed to start at 3:00 PM. Now it's at 2:30 PM, so I need to cut my talk from three hours to two and a half. We might have to move a bit faster.

00:00:22.160 As you have seen in the previous talks, the theme for today is basically telling you how you should code, so please keep that mindset for the next half hour at least.

00:00:28.320 I am between you and your lunch, so I will try to make it brief and interesting enough that you forget you're hungry. However, I also have a lot of material, so I will try to go fast.

00:00:40.600 If there are any questions, please let me know. The title of my talk is 'Integration Tests Are Bogus.' My name is Piotr Szotkowski, and I work at the University. I am also the Chief Software Officer at Rebased, where we work with Ruby and Rails.

00:00:53.079 If you have any projects you'd like us to work on, please let me know. Let's start by saying what integration tests are and, most importantly, why they could be seen as problematic.

00:01:04.439 If you haven't seen this talk before, integration tests can be thought of as a self-replicating virus that threatens the health of your codebase, your sanity, and your overall life. This is a direct quote, and you should definitely watch the talk that originates from.

00:01:29.360 What types of tests are there other than integration tests? There are numerous tests with different categories and taxonomies, but let’s concentrate on three types: end-to-end tests, integration tests, and unit tests.

00:01:47.320 End-to-end tests check whether your application still works after you make any changes, integration tests check whether an object works with its actual collaborators, and unit tests ideally test a single behavior to confirm that things work as expected.

00:02:02.360 So, why would you do all of these different levels of testing? We perform end-to-end tests because we believe our tests work, though not enough to deploy to production without first clicking around in the application.

00:02:23.080 They give us peace of mind. We conduct integration tests knowing that end-to-end tests can be slow, and we want to verify whether objects work with their collaborators without mocking or stubbing them out. We use unit tests to identify what broke when something does fail.

00:02:56.120 If something breaks and you only have end-to-end tests, you typically see half of them fail, which is not an ideal way to determine what actually broke.

00:03:03.400 Unit tests also allow for outside-in development, which is an interesting technique that I personally enjoy.

00:03:16.239 Speed is not the sole reason for conducting unit tests. While they give you fast tests, which is crucial for efficient testing, their primary purpose is to isolate the object under test from its collaborators.

00:03:29.799 To achieve this, we use test doubles that act as stand-ins for collaborators, providing specified responses rather than calling the actual collaborators. This way, we can verify that the right calls were made and that our code communicated as expected.

00:04:08.159 There's a wonderful Ruby book out there titled 'Practical Object-Oriented Design in Ruby.' It emphasizes that while testing our objects with collaborators, we should adhere to command query separation.

00:04:26.440 This principle suggests that outgoing calls from our objects should either be query calls—which ask the collaborator for something without changing the state—or command calls—which instruct the other object to perform a task that does change state.

00:04:59.080 In Ruby, a method cannot be entirely devoid of return values, but you can attempt to structure your code so that state-changing methods do not return meaningful values, thus minimizing any temptation to blend state changes with meaningful returns.

00:05:22.280 There are instances where you cannot maintain command query separation, such as with a stack where you must pop off the top item. In such cases, you definitely want to change the state of the stack and know what item was removed.

00:06:01.360 However, if you can opt for command query separation, you should. According to one practitioner, when writing tests, you should stub the outgoing query calls and mock the outgoing command calls.

00:06:35.280 You can then check how your code behaves when a query call returns a specific value and verify that the command call, which is state-changing, gets executed.

00:06:54.720 For your objects, you should also test incoming calls to all the public API of that object. If the object is a collaborator of another, you will test its API from the perspective of that other object.

00:07:12.360 There’s a perspective presented by a professional who discusses the integration tests and defines them differently. He asserts that the goal of unit tests is basic correctness—essentially checking whether our object behaves correctly.

00:07:53.520 Given the myth of perfect technology, we want to be sure that our object functions correctly, assuming the rest of the system works as intended. This addresses whether the object makes the right calls, handles responses appropriately, and whether the collaborator can adequately respond.

00:08:12.960 Understanding whether the collaborator provides the right response correlates with how this unit testing can ensure correctness. Heavy use of doubles is essential for unit testing to really isolate the object from its collaborators.

00:08:32.080 If you want to do this properly, especially when using outside-in testing, you need test doubles to create an effective design, as you don’t yet have the internal implementation to rely upon.

00:08:43.920 However, there's a real danger: when your code changes, and your mocks or stubs aren't updated, they can misrepresent the object under test by showing a non-existent world state.

00:09:10.680 You can find yourself in an unfortunate situation where you have a green test suite, heavily mocked, yet the application fails to boot. This false sense of security can erode your trust in your tests.

00:09:41.920 The core problem with doubles arises from the nature of outgoing calls. If an outgoing call is defined incorrectly, or it is right but the collaborator cannot handle it as expected, this can lead to trouble.

00:10:07.360 If you encounter an issue in your tests where the mocks and stubs don't interact correctly, it highlights the symptoms of this problem. People often think that end-to-end tests will catch these mistakes, but they won't.

00:10:57.920 You cannot test all code paths using end-to-end tests alone. Thus, the need for integration tests arises. However, integration tests can often test too much to be practical and can be slow.

00:11:19.160 Also, if there are too many interactions with too many collaborators, the scenarios can grow extensively, leading to an overload of test cases.

00:11:43.200 Watch that talk for more detail; I don’t want to repeat everything here. I apologize for this bad joke about glitter and powder, but let's illustrate what I mean through an example.

00:11:54.080 Suppose we have a weather service for vehicles. Imagine I just paid tribute to someone who brought me here in their Camaro. While the weather was unfavorable for coding in Wroclaw, I can say that as you can currently see, the weather has significantly improved.

00:12:24.800 We expect the weather object to collaborate with the vehicle to ask for its location. Depending on whether it's raining or clear, we implement functionality to get the correct weather response.

00:12:46.320 Now, we implement the vehicle, but I’ll spare you the awful details for simplicity’s sake in this presentation. We ask the vehicle for its location, and if it's raining, it provides one response; if it's clear, it provides another.

00:13:10.839 Our specs pass quickly, even as a fully integrated test that tests the weather collaboration with the vehicle. But let’s assume asking for the vehicle’s location is very costly.

00:13:27.440 This integrated test passes, but it could be very slow and require much time to verify whether weather collaborates correctly with its partner, making it less efficient.

00:13:50.040 Thus, we can opt to stub it. In Ruby, the simplest way to stub something is by using 'open struct.' You can define any calls as setters, and you can receive those as outputs.

00:14:02.440 We run the stub, and again we have a fast spec that passes. However, if we want to spice up our code a bit by changing 'location' to 'whereabouts,' we create a breaking change.

00:14:29.400 Our specs still pass, even though the vehicle class can no longer respond to a request for 'location.' This shows that the integration test would reveal the problem, but only if we hadn’t stubbed it.

00:14:50.960 With verified doubles, we can test behavior as well. These ensure that the mocks and stubs check if methods exist. After removing the open struct, we can easily replace it with the Verified Doubles when running the tests.

00:15:03.199 Now when we run our specs, 'Bogus' verifies that we shouldn’t stub 'location' on a vehicle since they don't have locations—but 'whereabouts' instead.

00:15:20.680 Verified doubles ensure that the methods you’re stubbing and mocking are indeed correct and available. They provide real runtime feedback based on expected behaviors.

00:15:41.560 They allow for accurate design and drive development as they highlight what to work on next, presenting requirements you have to implement based on previous passing specifications.

00:16:02.960 Let’s also explore mocking. If you want to check whether we are making some call to another collaborator, we may be concerned with making the call rather than the return value.

00:16:20.440 In this instance, we will expect a service to receive 'weather at waro' when we have a Camaro in Wroclaw. To do this, we will mock the service call and expect to receive that input.

00:16:55.680 Expecting and spying in 'Bogus,' there's a small difference. With stubbing, you define a setup and wait for a call. When calling the service afterward, you ensure expected behavior.

00:17:12.360 As a bonus, you can easily inject dependencies into your services, which remains convenient and natural in Ruby. This approach allows for simplifying mocking processes.

00:17:44.399 Through dependency injection, the tests become manageable because they focus on behaviors and expected outcomes without tightly coupling to the specifics of implementation.

00:18:03.199 One can use verified doubles to enforce stricter contracts, thus catching instances where the mock-api using the wrong interface might lead to failure in unit tests.

00:18:14.159 The final consideration is contract tests, which remain experimental but functional in 'Bogus.' They test the actual calls and returns made by your mocks.

00:18:35.760 If we define contracts for vehicles, if we mock with certain values, we also need to ensure that our unit tests verify that those calls can succeed with the actual implementation.

00:18:46.440 For example, we can mock calls with speed values, but we need to ensure that a vehicle can handle that, and so we must run those contracts.

00:19:02.360 Mocking lets us ensure we use interactions correctly, and contract tests back this finding by validating that mocks do not misrepresent capabilities.

00:19:17.199 So, the big picture with outgoing calls focuses on ensuring correctness with verified doubles, while failing integration tests lead us to necessity for more thorough integration testing.

00:19:48.159 As I finish up my arguments here, I remind everyone that despite being in between you and your lunch, it's truly about investing time to create efficient testing frameworks.

00:20:02.439 Visit the Bogus repository on GitHub, fork it, make use of the testing methodologies, and practice integration tests.

00:20:20.440 If you'd like to dive more in-depth about practical object-oriented design in Ruby, I strongly encourage you to check out the integration test or scam talk. It's highly rated!

00:20:56.600 This content offers a wealth of knowledge regarding testing and development practices. After all, developing a robust application requires thorough and well-structured testing.

00:21:13.680 Remember, the nature of successful tests revolves around the ability to communicate well with your doubles. Don't let them veer your output away from proper unit testing.

00:21:28.160 Thank you for listening! Are there any questions?

Piotr Szotkowski

@chastell

wroclove.rb 2014