00:00:12.080
All right, let's get this started. My name is Ryan Davis, and I am known pretty much everywhere as 'ryand-ruby', except for Twitter. I’m an independent consultant based in Seattle and a founding member of Seattle.rb, which is the first and oldest Ruby group in the world.
00:00:36.000
Setting some expectations: This is an introductory talk with very little code in it. I'm not going to teach testing or TDD; instead, I will talk about the 'what' and the 'why', not so much the 'how'. I have 218 slides, which puts me at just under five and a half slides a minute, so I have to go kind of fast.
00:00:49.520
Let’s get started. The simplest question we can ask is: What is Minitest? Minitest was originally an experiment to see if I could replace Test Unit for about 50 projects I had at the time—I'm now up to about 100—using as little code as possible. I achieved that in about 90 lines of code.
00:01:10.399
It's currently available as a gem; we didn’t have RubyGems back then when it was originally written. It now ships with Ruby as of version 1.9.1 and later. Minitest is meant to be small, clean, and very fast. While its size has increased to around 1600 lines of code, which sounds significant compared to 90, it’s still very small.
00:01:22.320
Minitest supports unit-style, spec-style, and benchmark-style testing, including very basic mocking and stubbing. It features a flexible plug-in system, among other things. Let's discuss the six main parts of Minitest: the runner, which is quite nebulous now; Minitest itself, which is the TDD API; Minitest Spec, which is the BDD API; and Mock, Pride, and Bench, though I will focus primarily on Minitest Test and Minitest Spec for the sake of this talk.
00:01:46.560
Now, let's jump into Minitest Test, which is the unit testing side of Minitest. Test cases are simple classes that subclass Minitest::Test or another test case, and tests are methods that start with 'test'. They make assertions about your code. It’s fundamentally classes and methods and method calls, all the way down. Everything is as straightforward as it gets, and it is magic-free.
00:02:04.320
Minitest Test includes the usual assertions you would expect from XUnit, plus several additions beyond what XUnit and Test Unit typically provide. Methods marked with a plus (+) are new to Minitest, while those marked with asterisk (*) do not have negatives or reciprocals. Unlike Test Unit, Minitest provides a lot more negative assertions, although it doesn't include some that you might expect, which I'll discuss shortly.
00:02:37.200
One common question is why there are so many extra features. I want my code tests to communicate as clearly as possible not only to me but also to other readers. Better communication in the code leads to more customized error messages, offering better information when things go wrong. Finally, 'assert_equal' has been enhanced to provide an intelligent diff mode, allowing you to see precisely what has changed instead of lumping a massive blob of information.
00:02:54.000
However, I mentioned earlier that some negative assertions are missing, which raises a lot of questions that I often hear: Where is 'refute' or 'assert_not_raised'? These are both absent in the same way that 'refute_silence' is. Let's look at that: Refute_silence indicates that a block of code must print something, but what does it print? I don’t care—this is a valueless assertion. You should assert a specific output instead.
00:03:18.560
Similarly, 'refute_raises' indicates that a block of code must do something, but I still do not care what it is. Both are valueless assertions. Instead, you should be asserting specific results or side effects that you intend. Some argue that it is useful; I disagree. It implies that side effects and return values are either not important or already checked, which is always false.
00:03:44.000
This either falsely inflates your code coverage metrics or provides a false sense of security. As an ex-lifeguard, I am aware that parents of children with water wings often believe their kids are safe, only to turn away and chat with friends while ignoring their kids who may be drowning.
00:04:06.400
In other words, it makes it seem like something has been tested when, in fact, it hasn’t. I have also heard it said that these assertions are more expressive; I contend that this is not true. Writing the test itself is an active expression, making it an explicit contract that states any unhandled exceptions are, by definition, errors.
00:04:30.720
The mere existence of the test states there are no unhandled exceptions via these pathways. I've had these arguments for years, and I have come to terms with the fact that some people will never be convinced, and that's okay. I can only try one more time to convince all of you.
00:04:53.040
Now, let’s talk about Minitest Spec, which is the example testing side. In short, Minitest Test is a testing API while Minitest Spec is a testing DSL (Domain Specific Language). Instead of defining classes and methods, you use a DSL to declare your examples. Test cases are 'describe' blocks that contain tests, while tests are 'it' blocks that call various expectation methods.
00:05:07.680
Here’s an example that is equivalent to the previous style: We have 'describe' instead of a class and 'it' instead of defining a test. However, in reality, describe blocks act as classes, while 'it' blocks act as methods. This same example transforms one-to-one into standard code where 'describe' creates a class and 'it' generates a method. There is no magic involved. All of your usual object design code tools exist and operate normally, which is essential.
00:05:39.760
It means that include works, 'def' works, and everything behaves as you would expect. You're just using a slightly different language. Similar to Minitest Test, Minitest Spec has many expectations defined and a similar set of negative expectations. It also shares some of the same missing assertions.
00:06:01.120
All of this is gained for free because each one maps from expectation to assertion directly. Underneath Minitest Test and Minitest Spec is the infrastructure that runs your tests and does so in a way that helps promote more advanced and robust testing.
00:06:13.520
Minitest has randomization baked in, which has always been enabled by default. This randomization feature helps prevent test order dependencies and ensures that your tests remain robust and able to run independently. As a rule, every single test, down to the lowest level, should be able to run by itself and pass; if it requires another test to run before it, then it is not standalone, and essentially, it is buggy.
00:06:32.200
As far as I know, Minitest was the first test framework to implement randomized run order. There's an opt-in system that allows you to promote a test case to run in parallel during execution, which takes randomization to a whole new level and ensures thread safety within your libraries.
00:06:53.760
Minitest was originally a tiny 90 lines of code, but features have been added over time, all while remaining relatively small compared to its capabilities. The reasoning behind the design of Minitest is that it isn’t special in any shape or form; my usual principles apply. If you have heard me ranting and raving before, Minitest is no different.
00:07:11.760
First and foremost, it’s just Ruby—it's classes, methods, and method calls. Everything is as straightforward as it can be. I believe that less is more; if I can accomplish something with less code, I absolutely will. Method dispatch tends to be the slowest process in Ruby.
00:07:32.080
More importantly, less code is nearly always more comprehensible than more code. For example, consider 'assert_in_delta', which is the equivalent of 'assert_equal' but for floats: never use 'assert_equal' on floats, and never use floats for money. There, I've done my usual cautions.
00:07:48.720
This is as simple as possible, with slight optimizations. Here, we use a block to delay rendering the error message in case there is no error—there is no need to incur that cost otherwise. This framework builds on just 15 other lines of code to understand how it works as a whole.
00:08:18.960
Indirection is the enemy; I want errors to occur as close to the real code as possible, with no delays or layers of indirection in between. I feel that Noel really emphasizes my point since he often discusses the layers of indirection that RSpec adds. I want to strip away as many of those layers as possible.
00:08:39.680
I want the responsibility of tests to fall in the right places—no managers necessary, no unnecessary coordination. Objects should handle their own duties. While I may not have the best example of this, 'must_equal' is an expectation that directly calls 'assert_equal' in the current test context.
00:08:58.080
The 'assert_equal' implementation is about three lines long, if I remember correctly, and that’s it—no magic allowed. Even test discovery avoids object space, using minimal meta-programming and relying on plain classes and methods for functionality.
00:09:17.200
I originally wrote Minitest partially to see if I could because I was the maintainer of Test Unit at the time, which terrified me. I also wanted Minitest to be the simplest implementation of a test framework for Rubinius, JRuby, and other Ruby implementations that hadn’t completed full Ruby compatibility. This way, they could receive feedback quickly.
00:09:42.239
Finally, Minitest has a flourishing plug-in ecosystem. I designed Minitest to be extensible, allowing the core framework to remain minimal. Here are just a few of the popular plugins available for Minitest.
00:10:02.559
So, what does this have to do with Rails? The official Rails stack utilizes Minitest. Each release peels back the testing onion, encouraging better practices—though peeling onions can make you cry. Fortunately, that's not the case with Minitest.
00:10:19.040
Rails 4.0 was the first version to transition away from Test Unit to Minitest, specifically using the Minitest 4.x line. At that time, I had already declared that I wouldn’t continue updating the standard library Minitest to Minitest 5; I would only maintain it at version 4.
00:10:39.120
This decision was made because Test Unit is built atop Minitest and has numerous hooks into its internals. That made it extremely difficult for me to update Minitest without breaking Test Unit functionality. So, I froze that version, and Rails updated to Minitest 4.
00:11:00.160
This change removed layers of complexity and indirection, allowing Rails a migration path to Minitest 5. Since Test Unit was already wrapping Minitest, there was minimal impact on users. Arguably, there might have even been a nearly imperceptible performance improvement, but I'm not claiming that definitively.
00:11:19.360
Rails 4.1 transitioned to the Minitest 5 line, which was painful due to outdated tests and many monkey patches present in Minitest itself. This update placed Rails onto the newer, more actively developed version of Minitest, making operations like exec-based isolation tests easier to implement.
00:11:38.640
Among the tests in Rails, some effectively fork a process to run a separate Rails app individually, ensuring that tests running in parallel do not interfere with one another. Although it may have been painful to get Rails switched over, you hopefully never noticed it.
00:11:59.200
Rails 4.2 turned off the alphabetical order of tests and removed a monkey patch that ultimately allowed tests to run in random order. This change was implemented solely to enhance the quality of Rails and your tests.
00:12:15.920
We did receive some reports stating that after updating, some previously passing tests began to fail. I had to isolate a number of bugs within Rails because of this issue. Honest to goodness, despite the pain it may cause, this is a beneficial step—testing order dependency bugs are problematic and notoriously difficult to track down.
00:12:35.279
Later in my talk, I will discuss a tool that can help identify these bugs. In future versions, Rails should be aligned more with Minitest; Aaron Patterson and I keep both in sync, and he informs me when important changes are approaching.
00:12:55.680
So, as a Rails developer, what does all of this mean? Hopefully, if I've done my job correctly, it means nothing. You shouldn’t even have to see Minitest most of the time unless you want to enhance it with specific plug-ins. That's because you’re subclassing Rails test cases, like ActiveSupport::TestCase or ActionController::TestCase.
00:13:13.840
There are about six of these test case subclasses—there might be more now. The basic architecture looks something like this: You write your own test class that subclasses ActiveSupport::TestCase, which then subclasses Minitest::Test. This provides features like per-test database transactions, so you don't have to worry about cleanup; instead, you simply add records that will disappear in the next test.
00:13:31.280
Aaron and I also introduced before and after hooks to assist Rails and other libraries or frameworks in extending Minitest with the extra wrappers they might need. It provides utilities for loading test data and various forms of declarations. Prefer them if you like, but if you don’t, you can still just use the traditional 'def'.
00:13:49.920
It also offers extra assertions such as assert_difference, assert_valid_keys, assert_deprecated, and assert_nothing_raised. Wait, what? Don’t worry! This is the actual implementation of assert_nothing_raised; it's only there for compatibility, and I personally believe that it should be deprecated and removed.
00:14:06.320
Perhaps this can be done in Rails 5. All this means you can write simple tests that describe your current task. Utilities like per-test transactions help prevent clutter in your test code, allowing you to focus on what the test intends to accomplish.
00:14:24.720
This is an essential example which I believe comes directly from the testing section of the Rails guide. ActiveController::TestCase is another Rails extension that subclasses ActiveSupport::TestCase, enabling you to leverage all of those features. It further extends to encompass the usual HTTP verbs.
00:14:43.200
It also simulates web server states and provides assertions specifically related to handling requests—allowing you to easily write concise functional tests, like the one you see here.
00:15:02.240
I love air conditioning so much that I am so dry right now. I’m just going to leave the lid off. ActiveDispatch::IntegrationTest provides full integration testing capabilities as it subclasses ActiveSupport::TestCase, so all the typical features are present.
00:15:22.080
It provides a wealth of assertions too many to fit on one slide, enabling you to write thorough integration tests that can cover multiple controllers.
00:15:42.080
If you are curious about all the functionality Rails adds on top of Minitest, you can find detailed descriptions that are quite informative. Thus, the Rails approach to subclass Minitest Test yields a very straightforward setup that remains powerful, providing everything needed for unit tests through to integration.
00:16:01.920
It leverages Minitest's capabilities with an emphasis on randomization and optional parallelization to ensure better testing outcomes, enhancing the robustness of the tests over time.
00:16:20.240
But what happens when something goes wrong? Perhaps you'd like to use spec-style testing; the problem is DHH disapproves of RSpec to such an extent that he won’t allow the Rails test framework to switch to Minitest Spec. He reverted that commitment.
00:16:39.040
From what I understand, he simply doesn’t want people submitting patches based on spec style, which makes it less accessible. That said, you are not without options if you prefer spec style. You can utilize Mike Moore's Minitest::Rails or Ken Collins's Minitest::Spec-Rails to varying degrees.
00:16:58.160
The two libraries have differences; one aligns more closely with RSpec's style than the other. Alternatively, if you upgraded to Rails 4.2 and now have failures—you might be thinking, 'Ryan, you broke all my tests!' I apologize. Unfortunately, it is not an easy fix, and to clarify, it is a good thing to identify and correct these issues sooner rather than later.
00:17:16.560
Why are my tests failing? This is a classic test order dependency bug, which essentially means tests will successfully pass when executed in a certain order, say A before B, but fail when B is executed before A.
00:17:32.000
If it were only three tests, it wouldn't be a problem and would be quite simple to spot and amend. However, in a large codebase with hundreds of tests, that becomes challenging. To tackle this, I devised Minitest::Bisect to assist in isolating the issues we observed during the transition to Minitest's randomization.
00:18:05.440
This tool helps isolate and debug random test failures by intelligently running and re-running your tests, narrowing down the list of potential culprits.
00:18:23.000
Here’s a simple example of it at work. Please ignore the fact that I'm pre-specifying the seed; pretend I'm running this anew, and we receive this failure.
00:18:38.720
We obtain the failure and grab the random seed, then we rerun the test using Minitest::Bisect. This action will re-execute the entire test suite to ensure reproduction.
00:18:58.240
Once confirmed, it begins to isolate the problem area down to the minimal subset of tests. As you can see, the number of tests reduces dramatically.
00:19:22.800
You are left with two tests run in a specific order that can cause reproduction every time. I've learned I’m not alone in facing this issue.
00:19:41.440
Or perhaps you are accustomed to a kitchen-sink approach to development. To start, just try it out—it might work! Many libraries, including Mocha, are already compatible with Minitest and perform quite well.
00:19:58.960
If not, you’re not alone; someone has likely already encountered your issue. Search for existing plugins listed in the Minitest README, check RubyGems.org, Stack Overflow, or other similar resources.
00:20:16.240
However, my suggestion—albeit contrary to popular opinion—is to try simpler testing first. Only integrate plugins after you've determined you truly require them. Starting fresh and clean could yield beneficial results; you might actually enjoy it!
00:20:32.560
Change takes time, so remember to measure your before and after states to gain an objective perspective. I can only provide anecdotes about projects speeding up after switching to Minitest. I would love to obtain more data supporting this view.
00:20:54.560
I've heard individuals report that they halved their test times by moving from RSpec to Minitest, but once more, I lack objective data.
00:21:17.040
So, you might be wondering why you should care about all this Minitest discussion. The first point I’d make is that if you’re not open to testing, you're a lost cause. There's ample data demonstrating the benefits of testing.
00:21:34.240
If you cannot accept that notion, I would prefer to assist others. That said, let’s consider this issue in detail.
00:21:52.080
The official Rails stack employs it, meaning that DHH uses Minitest. Tenderlove uses it as well, and Jeff Casimir and his cohort teach Minitest at a touring school. Other popular gems, like Nokogiri or Haml, also utilize Minitest.
00:22:14.480
In fact, more than 4,000 gems declare dependencies on Minitest. Unfortunately, since Minitest is part of the standard library, many gems neglect to declare their dependency, so there are likely even more.
00:22:37.040
What are the real functional differences between Minitest and RSpec? As testing frameworks, there is much overlap, so I won't delve into that. Where they diverge is where things become interesting.
00:22:52.160
To be fair, RSpec offers much more than Minitest; features like test metadata, filtering, additional hooks like 'before' and 'around', implicit subject, described class, complex mocking, and more.
00:23:12.080
Essentially, RSpec is fancier. However, what Minitest offers in return are benchmarking, speed, and parallelization, making it simpler and more pragmatic.
00:23:32.000
The cognitive differences between Minitest and RSpec fuel a significant divergence. This was driven home by Myron Marston a few years back, who provided a compelling response on Stack Overflow comparing RSpec and Minitest.
00:23:51.360
While there was a degree of bias, I believe it was a balanced overview. Unfortunately, there is a lot of information to digest, so let’s focus on an essential paragraph.
00:24:06.160
Let’s color-code the points: red for RSpec, blue for Minitest. Myron suggests this is why RSpec is great, though I’d argue it reflects everything wrong with RSpec, but we are both right—philosophically speaking, we have different goals and perceptions of quality.
00:24:27.360
To break it down further: Minitest compiles blocks down to simple classes, whereas RSpec refines testing concepts into first-class objects. The first term here signals a significant red flag for me regarding RSpec.
00:24:48.760
Examples in Minitest compile 'it' blocks into simple methods, while RSpec uses first-class constructs. Simple methods perform assertion expectations, whereas RSpec employs first-class matcher objects.
00:25:09.440
Further distilling this: Minitest uses classes, while RSpec uses first-class objects. Minitest relies on subclasses or includes, while RSpec relies on first-class constructs, meaning that the methods utilized in Minitest are more straightforward.
00:25:30.720
The term first-class indicates that you can assign a value to a variable and use it as you would any other value. Almost everything in Ruby is considered first-class, which doesn’t make it a distinguishing factor.
00:25:52.160
Consequently, where I could, I employed Ruby's mechanisms, whereas RSpec often reinvented the wheel. To illustrate how RSpec complicates matters, consider an example with two nested describes, each with a 'before' block and an example.
00:26:10.160
The 'before' blocks appear to be inherited, while examples are not. You end up with two runs: the first with one 'before' block and the second with two. For 'A' and 'B', classes nesting is a subclassing example; if they are classes, we require a method to prohibit inheriting tests.
00:26:31.440
Minitest’s approach is simpler: I loathe this runtime behavior that RSpec anticipates while also complicating the scenario. Presumably, if 'before' and 'after' work like included modules, we don’t need to define multiple methods, but we still need to generate inner modules, and setups must intelligently invoke super. That's cumbersome.
00:26:52.240
In essence, RSpec necessitates learning an additional object model that sits above Ruby's own object model. This presents complications, especially for those new to Ruby or learning it
00:27:15.040
at the same time, which can be overwhelming. Consequently, users may feel encouraged to gloss over details, using descriptions as a magic incantation rather than understanding their function.
00:27:30.799
If you missed the preceding talk, much occurs within 'describes,' and it usually boils down to a simplified instruction set to achieve the desired outcomes. Arthur C. Clarke remarked, 'Any sufficiently advanced technology is indistinguishable from magic.' I muse this means RSpec operates as such.
00:27:47.360
As previously mentioned, many people consider RSpec overkill, introducing a cognitive cost due to the added complexities. Therefore, it is valuable to highlight the raw numbers contributing to this added intellectual burden.
00:28:03.920
Let's visualize these numbers in the next slide. Here is a complexity metric called Flog, which indicates the difficulty of testing, debugging, or grasping the code. RSpec is approximately 6.6 times larger, with the combined code and comments at 8.5 times greater, akin to reading Dr. Seuss compared to James Joyce.
00:28:27.360
You can see my reference to Dr. Seuss perfectly represents my notion of distinction between the two frameworks.
00:28:42.160
Beyond cognitive burdens, performance variances also arise. These various abstractions that RSpec utilizes introduce real costs. The presented data illustrates runtime complexities for both systems.
00:29:02.319
What you will highlight are two significant problems: RSpec’s runtime for failures grows exponentially, while Minitest’s is low and linear. The impacts of failure in RSpec lead to consistently longer runtimes.
00:29:19.440
As for the assertion speed in both frameworks, they are linear. Therefore, no attempts should be made to reduce test quality for speed; Minitest will generally allow for faster testing.
00:29:35.520
You have the option of switching to Minitest or avoiding issues around refactoring or bugs. In summary, at the day's end, I don’t care what framework you utilize as long as you ensure your tests are in place.
00:29:52.160
Hopefully, I've illustrated the technical merits of Minitest. Choose the tool that works for your purposes, not based on popularity. For many, the argument arises surrounding RSpec having more available documentation.
00:30:08.240
Perhaps fewer articles arise due to testing in Minitest needing less explanation. Minitest tends to be much easier to comprehend, allowing you to read it in a matter of hours and grasp its entirety.
00:30:20.800
It might also suggest that Minitest users are busy accomplishing tasks instead of reading and writing articles. In any case, choose what functions best for you. You might just appreciate it; after all, it’s just Ruby.
00:30:38.239
Thank you.
00:30:56.239
And please, hire me!
