Integration Tests Are Bogus

Talks

Piotr Szotkowski

#behavior-driven-development-bdd

#mocking

#unit-test

#software-design

Integration Tests Are Bogus

by Piotr Szotkowski

In his talk 'Integration Tests Are Bogus' delivered at RubyDay 2016, Piotr Szotkowski explores the role and effectiveness of integration tests in software development. He emphasizes that his insights are personal opinions aimed at sparking thought rather than as definitive prescriptions for testing methodologies. Szotkowski reveals multiple key points regarding integration tests and their alternatives:

Definition of Tests: He defines various types of software tests, notably unit tests, integration tests, and end-to-end tests, emphasizing that each serves a distinct purpose in validating application functionality.
Integration Tests Caution: The speaker shares anecdotal findings from Jon Rainsberger's talk, suggesting that integration tests can act like a 'self-replicating virus' that's detrimental to code health. He argues that while integration tests are not inherently bad, their necessity can sometimes be overstated.
Unit Test Importance: Szotkowski underscores the significance of unit tests for providing specific, reliable feedback when issues arise. Unlike integration tests, unit tests can identify failures swiftly and accurately, making them crucial for effective debugging.
Testing Strategies: He advocates for a careful balance of testing strategies, promoting a focus on well-structured unit tests alongside judicious use of integration tests as needed, rather than an over-reliance on the latter.
Use of Test Doubles: The concept of test doubles (stubs and mocks) is addressed, with Szotkowski explaining their role in isolating units of code during testing to avoid reliance on external resources. While these can be beneficial, their discrepancies from actual application behavior can lead to unreliable test results, hence the need for integration tests to catch these variations.
Best Practices: The importance of a clean separation of commands and queries in coding to facilitate effective unit tests is discussed, along with suggestions to utilize libraries and tools that verify interactions between components.
Conclusion and Advice: Szotkowski concludes by encouraging developers to reflect on the presented ideas and incorporate effective testing strategies that improve reliability and application performance. He calls for continuous education and engagement in discussions about best practices in testing.

Overall, the presentation invites developers to critically assess their testing methodologies and adopt a balanced approach to integrating unit and integration tests, enhancing the reliability of software applications.

00:01:05.759 Anyway, I want to talk to you about integration tests but primarily as a tool for achieving the things I want to show. The issues I’m addressing are generally universal across languages, as they tend to remain the same. This will be one of the approaches I use to display these issues. The title is a tongue-in-cheek reference to a talk by Jon Rainsberger, who claimed that integration tests are a scam. So, what are integration tests, and why should you care?

00:01:20.670 The idea behind integration tests is captured in a quote from another talk, which states that an integration test has become a self-replicating virus that threatens the health of your codebase and your sanity. I am not exaggerating when I say your life could depend on it. I really encourage you to watch this talk. It’s a bit dated, and there are parts where you might cringe at the delivery or some references, but its technical merits are solid. I will refer to some concepts from that talk throughout this discussion.

00:01:38.909 Let’s define integration tests. In general, there are various types of tests. One common method involves end-to-end tests that verify whether the entire application boots up and follows the successful path. These tests usually ensure that common features work, avoiding any significant failure points like login failures. However, the main idea behind end-to-end tests is that they provide reassurance about the application's functionality, even if the individual tests transition in some manner. Essentially, we invest in our tests to ensure everything works, even if the integration tests pass. Integration tests check whether parts of the application wire together correctly and function as a unit. They examine whether application components and their collaborators work together adequately. Additionally, unit tests assess if individual components of your code function correctly.

00:02:57.419 While it's essential to have all three types of tests, I want to be very deliberate about what unit tests should entail. Unit tests do not necessarily have to target a single class. A unit of your code might consist of several classes working on implementing a minor feature. Usually, these classes interact very closely. Therefore, one shouldn't strictly adhere to the idea that unit tests must correspond exactly to one single class. They could cover multiple classes taken from a single unit. Importantly, you do need unit tests because when you change something in your codebase, having only end-to-end tests might lead to most of your tests failing afterward, which is not convenient. I support the idea of having unit tests, as they provide at least some reliability when changes occur.

00:05:27.009 Integration tests raise the question of their overall value. Again, this is my opinion: they may not be required, but if you feel the need for them, then, by all means, use them. Generally, the more tests you have, the better, provided they are well-written. The last thing you want is an excess of tests giving a false sense of security. Therefore, it's crucial to keep your tests relevant and be open to removing those that do not add value. This is not an indictment against integration tests; rather, it's about how you can minimize the necessity for them if you prefer to focus on having more dependable unit tests along with reliable mocks and stubs.

00:06:12.490 Why perform end-to-end tests? They serve to confirm whether the application works and provide reassurance. If you only rely on unit tests, you could find yourself in a scenario where all your tests pass, but the application doesn't. This situation can lead to frustration because it feels deceptive when your tests fail to represent reality correctly. On the other hand, integration tests are typically slower to write and manage. They are meant to confirm if a few crucial components are communicating effectively and passing the right arguments to each other. Unit tests, however, provide precise feedback about what went wrong. If you encounter an issue with one failing unit test, it’s far simpler to diagnose and rectify than if half your end-to-end tests suddenly fail.

00:07:18.730 Testing can begin from the top-level down, allowing you to design your application outward by stubbing and mocking lower layers until they are fully implemented. The key point here is that speed should not be the primary reason for performing unit tests. The goal of unit testing is to isolate small, specific functions within your code, making it evidently clear where the issues lie. The fact that unit tests are often faster is merely a beneficial side effect. To emphasize, the purpose of unit testing is not solely speed but reliability and pinpoint accuracy when diagnosing failures.

00:07:52.950 To conduct unit tests, we utilize test doubles, which serve as stand-ins for actual collaborators. There are two primary types: stubs and mocks. Stubs return predetermined responses while mocks verify whether specific calls were made. The greatest advantage of test doubles is their ability to isolate the unit under test. If your test relies on functions that access external resources, like APIs or databases, this can cause issues, especially in situations where you cannot consistently execute your tests, such as on an airplane without internet access. Additionally, testing an object dependent on numerous database tables complicates matters; if I could, I’d go back and advise the original developer to streamline their design.

00:09:49.750 Many web applications face the problem of testing suites intertwined with databases, particularly when they contain excessive business logic. While it can be justified to have tightly coupled tests, it’s advantageous to understand how to extricate oneself from those situations should they arise. As with many technical questions, the answer often depends on context, and various valid reasons justify writing simple, cross-cutting tests, such as speeding up development. However, along with that comes the importance of developing solid unit tests. A book I highly recommend is 'Practical Object-Oriented Design' by Sandi Metz. This book highlights achieving the common object design principle of separation between commands and queries in your application.

00:11:27.090 The principle states that your code should ideally contain methods that perform either queries—functions that return information without modifying state—or commands—methods that alter state without returning meaningful responses. In Ruby, where methods return the last statement value by default, ensuring a clean separation can be challenging. When your methods adhere to this separation, and most of them are either commands or queries, you can establish precise unit tests. We can mock outgoing command calls to confirm that state changes actually occur while allowing queries to verify expected responses.

00:12:32.749 What about when calls don’t exhibit proper command-query separation? An example could be popping the top item off a stack. This represents a situation where one method modifies state while simultaneously returning the element modified. Understanding how to approach such cases can streamline your code's design, making them more effective without excessive complexity. By adopting and maintaining better code structures, you can avoid numerous issues that can arise from negligent design decisions.

00:13:29.029 In the context of unit tests, it's important to focus solely on incoming calls to the object's public API. I firmly believe that private methods should not be tested individually. If you think a private method is so complex it needs to be tested in isolation, it may indicate the necessity for refactoring—perhaps by extracting the behavior into its own object. Thus, unit tests should concentrate on interacting with public methods, while mocking the state-altering command calls. This approach ensures you’re aware of how query responses impact the workings of your objects.

00:14:06.440 In Jon Rainsberger's talk, he discussed basic correctness—ensuring that a piece of your technology reliably computes the right results with proper input. In establishing what your object under test can achieve, it’s critical to confirm that it handles responses correctly and that appropriate calls are made to its collaborators. However, if the mocks and stubs you employ do not accurately reflect what exists in the actual implementation of your system, you risk emerging problems: you may have green tests—indicating all is well—while your application ultimately fails to function as intended.

00:16:59.670 One significant issue with test doubles is that they can create scenarios where green tests become untrustworthy. If you don't have a method for ensuring mocks and stubs line up correctly with what's implemented in your application, you may find yourself in a regrettable position. Especially, if mocks and stubs differ significantly from actual objects, your tests may not accurately reflect real-world performance. This disconnect could leave your application in disarray, leading to failures during production deployment despite having validated your unit tests.

00:17:56.440 To solve these issues, integration tests are proposed. They catch the potential problems that arise when your mocks and stubs diverge from the actual objects they simulate. Moreover, unit tests will, by their nature, become less resilient as code evolves due to changes in dependencies. To ensure the reliability of your tests, you need to maintain a balanced relationship between unit tests and integration tests. This holistic view allows inaccuracies across the system to flag through integration tests, and we can glean useful information from these interactions.

00:18:59.840 Taking it a step further, if you don’t ensure your mocks and stubs connect to your system correctly, you risk your tests passing when they shouldn’t. This is where the importance of verification doubles comes into play—they validate that your tests accurately engage the components of your system. Likewise, it's advisable to employ tools to keeps checks on how your mocks and stubs are engaged. By ensuring they truly interact with stubs and mocks accurately, you bolster your testing suite against future fluctuations.

00:20:59.110 Contract tests also serve effectively in this context, ensuring that if a method exists in your codebase, it adheres to expected contracts. If you change a method or argument, your tests should indicate whether the application behaves as required. For instance, if mocking a connection from one point fails because external events intervene—like an airport closure—this should trigger a failure case indicating that you need to check those interactions. This back-and-forth enhances the quality requirements you need in your coding process.

00:22:52.800 In recap, it’s critical to verify not only that outgoing calls work but also that collaborators handle them appropriately. Contract tests ensure that your suite covers the necessary connections in your systems. Diligently employing verified doubles not only boosts accuracy but helps liberate the burdens associated with traditional mocking. It’s important to produce quality tests that amply cover the major functionalities while explaining the processes clearly enough to make them easy to follow.

00:24:54.859 So if you want to start implementing this in your codebase, utilizing libraries like Bogus can facilitate testing needed components effectively. With verified doubles, you can conduct tests quickly and utilize stubbing techniques appropriately. These are simple but essential design principles that should be kept at the forefront of your testing methodology.

00:26:30.370 When relying on dependency injection, for instance, you can clear the path for better-directed functionality. Adjusting for service connections or interfacing with new functionality should be straightforward with the right structure. In the long run, fostering a healthy testing environment will elevate your application’s reliability and performance.

00:28:10.190 By conducting effective testing across your application layers—including unit tests, integration tests, and higher-level contract tests—you can align functionality with requirements while troubleshooting as needed. Keep in mind that having a clean, accurate set of tests produces great confidence in your deployment quality. Ultimately, adopt techniques that breach modularity and trust between the various components of your architecture.

00:29:23.640 In conclusion, I urge you to reflect on the practical approaches discussed here and integrate them into your development processes. Look for resources to guide you on good practices and engage with discussions around these topics to gain further insight. Thank you for your attention, and I appreciate the opportunity to share my thoughts with you today. If you have additional questions or feedback on this topic, feel free to engage with me afterward. I hope you found this useful!

rubyday 2016