00:00:09.200
I want to start with a story today. This is a story about a friend of mine named John, who used to work as a software engineer at a big aviation contractor.
00:00:14.799
One day, he walked into a meeting and the client came to them and said, 'Here’s a million dollars and here's a deadline that's a year out. Here's a little chip, and your job is to make sure that chip doesn't kill people.'
00:00:21.680
The chip was already done; all they needed to do was test it. What the chip was is a sensor that goes inside of landing gear. It detects extreme temperatures, extreme pressures, and atmospheric conditions that might indicate a problem.
00:00:32.960
If there is a failure, the landing gear could stick, and the plane would crash while failing to recognize that the landing gear was stuck. Clearly, lives are on the line, making it crucial that the tests prove this device works as expected.
00:00:46.960
Normally, our response to testing is to ensure it works—prove that it works. However, in this case, a realistic test was not feasible. What could be a maximally realistic test for this situation? Of course, put the chip in multiple planes, load those planes with people to simulate the right weight, and then keep crashing planes until the chip works.
00:01:03.280
Thus, if our tests, when we use the word 'test', aim to verify our code’s functionality, the term 'fake', or anything less than real, feels like a cheat. It feels like we are diminishing the value of the tests, so we prefer to think that all our tests are real.
00:01:16.880
Web applications are certainly much simpler to test than landing gear on an airplane; they are usually not life-critical. However, we also often partake in various forms of cheating, justifying that they are acceptable ways to cheat without discussing it as if we are faking anything.
00:01:28.320
For example, when we use Selenium, we create a fake user and script that user on what to click on with the keyboard and mouse. It follows our instructions repetitively, without variations or randomness, which does not simulate human behavior.
00:01:36.240
In terms of test data, most testers will clean their database with every run. If you have a long-lived app with years of data migrations, your production data means that users have dealt with all the, let’s say, nuanced column changes for years, but your tests? No problem there—the tests don't encounter any issues because they get a clean run every time. Your tests won’t catch this lack of realism.
00:01:49.439
Moreover, the runtime is usually restarted for each individual test. If you have a subtle memory leak occurring over time, causing your application to lag, these long-running situations might not be visible to your tests. Your tests will pass, so we often cheat.
00:01:59.680
Typically, tests run against a single process or even a single-threaded runner or server. Meanwhile, in production, when scaled up, you have multiple servers running simultaneously, load balancers, and all kinds of potential race conditions in your code that your tests might not expose.
00:02:07.920
So, we are all cheaters. Reality is costly; it would be expensive to hire a multitude of people in focus groups living in a lab all day to ensure our code functions with each commit.
00:02:13.360
If reality is expensive, why not budget it? Why not be more intentional about evaluating how real versus how fake our tests should be? We cannot always have pure reality, so we must determine how much reality each of our tests needs.
00:02:22.400
Instead of asking if a test is too fake, we should ask how much reality it needs. We are flipping the question on its head, aiming to assess the reality worth of the test. Is it worth 8 bits of Mario, 16, or 64, or is that guy a real plumber?
00:02:38.960
Every test has different needs. These reflections come from my observations at conferences, many of which are ruby conferences, where I hear platitudes like 'mocks are good' or 'mocks are bad', or others saying, 'only mock what you own.'
00:02:57.600
Some assert that you should only mock external systems to speed up the tests, while others proclaim, 'mock everything.' Some individuals mock everything all the time to an uncomfortable degree, while occasional voices of reason suggest avoiding over-mocking without defining what that entails.
00:03:08.960
We have a plethora of platitudes surrounding faking in tests, and I want to step back to initiate a more formal discussion. I crave nuance and rigor in how we address testables and thoughtfulness regarding our testing goals.
00:03:21.200
It’s ultimately up to us to determine the value we seek from our tests and to assess how much reality is required for each type of test, along with the associated consequences of that reality.
00:03:35.760
This notion aligns well with the themes of the first two talks today, focusing on long-term sustainability in design and understanding our motivation for our actions.
00:03:50.560
I first want to discuss different types of tests to establish a common language we can use today.
00:03:57.040
Let's start with an end-to-end test. In an end-to-end test, the real application is a black box. Consider a dog app and a separate application that runs in its own process, acting as an interface to test the functionality of the dog app, likely using a web interface or command line.
00:04:11.560
Full stack integration tests involve using the same process as our main application, maintaining visibility into the object model. For example, a dog feeder class, responsible for feeding dogs, will access a bone repository to gather new bones as well as a dog repository to save the animal after receiving a bone.
00:04:25.440
An integration test typically works alongside all components, whereas a unit test, as defined by Michael Feathers, follows four rules. These rules include no network access and no file system access in the case of a unit test, which means it essentially fakes out the database layer to reduce the black box while still owning real repositories.
00:04:43.680
An isolation test features similar elements, but here the test orchestrates its own supporting components. For example, the dog feeder test creates its own fake dog and bone repositories, injecting them into the dog feeder to maintain complete control over that interaction.
00:04:56.760
This set-up allows our test to serve as a thorough specification of what the dog feeder does, including its relationships with others. When following Test-Driven Development, this approach helps clarify what collaborators the object might require, thus forming an isolation test.
00:05:11.080
When discussing mocks, we find the term 'mock' generally refers to any fake we use to represent a real entity in our tests. A 'test double' serves as a more specific term because 'mock' tends to describe a particular subtype of a fake item used during testing.
00:05:25.840
The term 'test double' was popularized by Gerard Meszaros in his book on xUnit patterns, emphasizing that we should visualize it as a stunt double—this fake object stands in for a real actor during tests.
00:05:39.440
We can categorize different types of test doubles further. A 'fake' often consists of makeshift objects we craft for testing simplicity, like a fake file data source instead of a slow, volatile network data source.
00:05:50.560
A 'stub' is a test double that pre-configures responses to certain messages. For instance, if we have a database class, we could create a stub that returns a specific order whenever it receives a message for 'find' with the argument '1'.
00:06:05.120
Mocks come with their distinctions as they pre-configure both responses and expected interactions. For example, using a mock in our test alerts if certain methods are called as expected—such as 'database.save!' with a specified dog object.
00:06:23.200
So, a spy is also a special subtype, designed to verify behavior over time without expecting specific invocations. Instead, it records all interactions so that we can later interrogate its actions against test assertions.
00:06:34.280
These various types of testables provide a framework for understanding why and how to utilize them. However, as we delve into their practical application, it’s essential to ask ourselves why we should use them.
00:06:43.360
Let’s analyze our motivations for testing. Acceptance testing serves as an end-to-end test, showcasing that the application functions as promised to a product owner, customer, or even ourselves in the future.
00:06:57.600
In this context, reality holds immense value, as we wish to ensure our acceptance tests mirror real-world conditions. This type of testing necessitates offering complete transparency about functionality to avoid any ambivalence.
00:07:12.159
Specification tests also hold importance; they exercise code while offering regression value. Their primary benefit is providing future developers with a chance to make safe changes—with clear readability and clarity being pivotal.
00:07:27.680
Regression tests come into play when a bug arises, leading us to write a test to reproduce the issue, correct it, and ensure the bug won’t return. Here, some realism is crucial to accurately replicate bugs; but we must realize that often, identifying bugs involves pinpointing a small, narrow issue.
00:07:39.280
In this case, we can afford a relatively minor level of reality—perhaps a 64-bit Mario's worth, just enough to identify core problems without needing a fully rigged application.
00:07:49.440
When we consider motivations based on test-driven development (TDD), it shifts towards a focus on test-driven design. Here, the objective is to discover the nuances of API design and understand the code's architecture.
00:08:04.480
Reality doesn’t play such a critical role in this context; the regression value of isolated tests is typically low, emphasizing more on layout rather than realism behind the tests.
00:08:18.559
Another situation we encounter is when dealing with legacy code. We might lack clarity about how it operates, yet we must alter it. Characterization tests help us make black boxes out of existing code by documenting inputs and outputs.
00:08:30.720
In such cases, it’s crucial to gain some realism about the potential changes without overly complicating how the surrounding application acts. It requires some realism but doesn't necessitate full, detailed coverage.
00:08:47.360
Overall, my point here is that we should consistently ask ourselves what value we anticipate derived from each test. If I am spending 10 hours on a test, I hope to realize significant returns from that investment.
00:09:00.560
Next, we will explore strategies regarding testing, assessing where and how testables fit into various strategies. Starting on one end of the realism spectrum, some teams prioritize writing tests simply to demonstrate functionality.
00:09:16.960
Under this philosophy, testing is regarded as primarily functional confirmation instead of design guidance. Thus, such teams invest primarily in end-to-end tests and integration tests to ensure things work, taking minimal risks in real-world testing.
00:09:32.960
The benefits here are high confidence in green builds, complementing that confidence with a lower volume of tests. You spend less effort on testing, but the trade-off is that you miss significant design clues.
00:09:48.320
If your primary goal is to make sure everything works, quick feedback becomes vital. You do not wish to wait an extended time to determine if your changes are valid. However, end-to-end and integration tests are often slow and may constrict your overall productivity.
00:10:07.760
Another strategy involves mocking boundaries; this strategy sees teams willing to fake out remote systems and third-party integrations but abstaining from faking things they completely own. This strategy focuses on unit tests and end-to-end tests, utilizing mocks on external resources.
00:10:18.800
On the plus side, this means each object has both unit tests and endorsements from collaborators, allowing for extensive coverage and higher regression significance when the object is used in different contexts.
00:10:30.479
The downside includes both redundancy and overload. One small change can potentially break thirty other tests, always requiring a convergence of mental effort from the initial state to the updated context. Unit tests alone do not guarantee that your entire application is functional, necessitating end-to-end tests.
00:10:45.680
A case-by-case approach intuitively allows developers to select the best tools and methods for each test. While this offers freedom, it may lead to incoherence or inconsistent expectations among teams and developers, blurring the lines of original intent.
00:10:58.960
In this chaotic approach, collaboration is essential to process changes effectively; however, it may enable test doubles being abused. In many Rails projects, I’ve observed a significant number of people using mocks and stubs within model tests.
00:11:14.080
Rails model tests are typically designed to communicate with real databases, and when mocks or stubs enter the mix, it muddles the clarity and the actual intent of the test.
00:11:26.679
In most situations, tests designed to verify key functionality may not effectively drive design decisions or architecture. Consequently, the results become ambiguous, leading to uncertainty over whether a test facilitates the correct approach.
00:11:39.440
As an alternative, we can shift towards the concept of growing object-oriented software, often referred to by its advocates as 'goose'. This method involves utilizing isolation tests to inform the design of the objects we create.
00:11:52.480
We still retain end-to-end tests to validate the software's capabilities. Although we might encounter skepticism surrounding high-level integration tests lacking detail, our real backing comes from integration smoke tests assessing overarching collaboration.
00:12:07.360
I’ve found this balance yields the best results, maintaining a handful of end-to-end tests while having my isolation tests run rapidly—for instance, a complete suite of thousands running in just seconds.
00:12:20.240
Jasmine encourages isolation testing by design due to difficulties surrounding integration tests. My experience showed me that high numbers of test cases quickly yield positive results, often allowing testers to maintain their focus without excessive time required for results.
00:12:33.599
This leads to the understanding that achieving quick feedback directly enhances productivity. Developer efficiency often suffers when feedback takes an excessively long time or has unresponsive delays.
00:12:46.080
A primary advantage of isolation testing is the feedback on abstractions and understanding the responsibilities of collaborating objects. If tests become too cumbersome, it may signify that our object structure requires refining.
00:12:56.080
As we establish and redefine these collaborations, effective testing helps regulate the workflow, keeping codebases clean and manageable without excessive setup for each component.
00:13:10.080
However, not all tests will fit into purely isolated or entirely integrated categories. Therefore, we may entertain the idea of maintaining numerous specialized test suites focused on various tasks.
00:13:24.120
Those categorized by motivation—characterization tests, design tests, and end-to-end assessments—can serve to balance our priorities effectively while allowing flexibility in addressing diverse testing needs.
00:13:39.200
Historically, I delved into goose-style isolation testing within Java projects. Yet, I’ve often questioned why many Rails users avoid test doubles.
00:13:54.080
I believe this stems fundamentally from Rails architecture. It’s not a bad architecture by any means, but it is a factor we must acknowledge and work within correctly.
00:14:09.720
Test doubles excel at writing isolation tests for our subject code, but they require first identifying and substituting all dependencies. For instance, understanding inherited surface area becomes vital—this relates to everything the subject interacts with.
00:14:24.120
While creating a simple application, I found that an ActiveRecord extension had over 300 public methods. The extensive inherited surface area complicates matters as we navigate toward more straightforward, manageable code.
00:14:39.920
In scenarios where classes require extensions, lacking clarity and assumptions complicate isolation significantly within Rails. Moreover, class loading can become a burden due to the performance hit incurred with heavy class reliance.
00:14:54.080
Consequently, we see the emergence of fantasies—these are tests using mocking without any runtime checks, generating a false sense of validity as real classes aren’t loaded during the tests. Should any method name change, tests would still succeed based on prior configurations.
00:15:08.080
Given these observations, Rails presents challenges in adopting a strict isolation testing structure. The goose methodology offers one potential route, but it intersects awkwardly with common challenges experienced in Rails.
00:15:24.120
Further considerations must evaluate existing Rails objects. Writing seamless integration tests, while respecting their inherent structures, yields dependencies that require clarification.
00:15:35.360
Ultimately, if we desire to avoid excessive functional coupling with our tests, we might need to reconsider how we leverage Rails altogether as the framework wasn’t necessarily built with isolation testing in mind.
00:15:50.320
For additional insights on this topic, I authored a post surveying various test libraries used in Ruby, from Mocha to FlexMock, addressing motivations behind my project decisions.
00:16:05.920
As a result of my experiences, I developed a test double library called 'Gimme,' which is open for exploration should anyone find value in it. Although I haven’t had the chance to deeply use it, I welcome any feedback.
00:16:11.440
In closing, I appreciate this opportunity to speak today. My company's name is 'TestDouble,' which seems fitting for this discussion.
00:16:18.560
You can contact me via my Twitter handle, @searles, and I value any feedback from all of you.
00:16:21.720
Thank you all very much for your attention! I truly appreciate it.
