Ruby on Rails on Minitest

RailsConf 2015

by Ryan Davis

In the presentation "Ruby on Rails on Minitest" by Ryan Davis at RailsConf 2015, the speaker discusses the evolution of Rails' testing practices with the introduction of Minitest. The talk serves as an introduction to Minitest, emphasizing 'what' and 'why' rather than 'how', aimed at both existing users and those curious about the framework.

Key Points Discussed:
- What is Minitest?
- Originally created to replace Test Unit with minimal lines of code, Minitest is a small and fast testing framework available as a Ruby gem since version 1.9.1.
- Components of Minitest:
- Introduces various components including Minitest Test for unit testing and Minitest Spec for behavior-driven development (BDD).
- Testing Methodologies:
- Stresses the importance of clear communication in code tests through enhanced assertions and intelligent diff modes for better debugging.
- Randomization Feature:
- Highlights Minitest's default randomization of test execution to prevent order dependency, ensuring standalone functionality of tests.
- Integration with Rails:
- Rails versions 4.0 to 4.2 have progressively switched to Minitest, improving testing practices while addressing issues like test order dependencies.
- Comparison with RSpec:
- Discusses differences with RSpec, noting that Minitest is simpler, faster, and leverages Ruby's inherent capabilities, while RSpec offers more complex features.
- Practical Tools for Issues:
- Introduces Minitest::Bisect to help debug failures related to test order dependencies, making it easier to identify problematic tests.

Davis concludes that the choice between Minitest and other frameworks like RSpec lies in their respective merits. Minitest's focus on simplicity makes it accessible, while ensuring robust testing practices enhance the overall development experience in Rails. The speaker encourages maintaining good testing habits, regardless of the testing framework used, implying that Minitest should be used effectively without overwhelming the developer.

Overall, the session reaffirms Minitest's integral role in Rails' testing ecosystem and encourages Rails developers to embrace effective testing methodologies to enhance code reliability.

00:00:12.080 All right, let's get this started. My name is Ryan Davis, and I am known pretty much everywhere as 'ryand-ruby', except for Twitter. I’m an independent consultant based in Seattle and a founding member of Seattle.rb, which is the first and oldest Ruby group in the world.

00:00:36.000 Setting some expectations: This is an introductory talk with very little code in it. I'm not going to teach testing or TDD; instead, I will talk about the 'what' and the 'why', not so much the 'how'. I have 218 slides, which puts me at just under five and a half slides a minute, so I have to go kind of fast.

00:00:49.520 Let’s get started. The simplest question we can ask is: What is Minitest? Minitest was originally an experiment to see if I could replace Test Unit for about 50 projects I had at the time—I'm now up to about 100—using as little code as possible. I achieved that in about 90 lines of code.

00:01:10.399 It's currently available as a gem; we didn’t have RubyGems back then when it was originally written. It now ships with Ruby as of version 1.9.1 and later. Minitest is meant to be small, clean, and very fast. While its size has increased to around 1600 lines of code, which sounds significant compared to 90, it’s still very small.

00:01:22.320 Minitest supports unit-style, spec-style, and benchmark-style testing, including very basic mocking and stubbing. It features a flexible plug-in system, among other things. Let's discuss the six main parts of Minitest: the runner, which is quite nebulous now; Minitest itself, which is the TDD API; Minitest Spec, which is the BDD API; and Mock, Pride, and Bench, though I will focus primarily on Minitest Test and Minitest Spec for the sake of this talk.

00:01:46.560 Now, let's jump into Minitest Test, which is the unit testing side of Minitest. Test cases are simple classes that subclass Minitest::Test or another test case, and tests are methods that start with 'test'. They make assertions about your code. It’s fundamentally classes and methods and method calls, all the way down. Everything is as straightforward as it gets, and it is magic-free.

00:02:04.320 Minitest Test includes the usual assertions you would expect from XUnit, plus several additions beyond what XUnit and Test Unit typically provide. Methods marked with a plus (+) are new to Minitest, while those marked with asterisk (*) do not have negatives or reciprocals. Unlike Test Unit, Minitest provides a lot more negative assertions, although it doesn't include some that you might expect, which I'll discuss shortly.

00:02:37.200 One common question is why there are so many extra features. I want my code tests to communicate as clearly as possible not only to me but also to other readers. Better communication in the code leads to more customized error messages, offering better information when things go wrong. Finally, 'assert_equal' has been enhanced to provide an intelligent diff mode, allowing you to see precisely what has changed instead of lumping a massive blob of information.

00:02:54.000 However, I mentioned earlier that some negative assertions are missing, which raises a lot of questions that I often hear: Where is 'refute' or 'assert_not_raised'? These are both absent in the same way that 'refute_silence' is. Let's look at that: Refute_silence indicates that a block of code must print something, but what does it print? I don’t care—this is a valueless assertion. You should assert a specific output instead.

00:03:18.560 Similarly, 'refute_raises' indicates that a block of code must do something, but I still do not care what it is. Both are valueless assertions. Instead, you should be asserting specific results or side effects that you intend. Some argue that it is useful; I disagree. It implies that side effects and return values are either not important or already checked, which is always false.

00:03:44.000 This either falsely inflates your code coverage metrics or provides a false sense of security. As an ex-lifeguard, I am aware that parents of children with water wings often believe their kids are safe, only to turn away and chat with friends while ignoring their kids who may be drowning.

00:04:06.400 In other words, it makes it seem like something has been tested when, in fact, it hasn’t. I have also heard it said that these assertions are more expressive; I contend that this is not true. Writing the test itself is an active expression, making it an explicit contract that states any unhandled exceptions are, by definition, errors.

00:04:30.720 The mere existence of the test states there are no unhandled exceptions via these pathways. I've had these arguments for years, and I have come to terms with the fact that some people will never be convinced, and that's okay. I can only try one more time to convince all of you.

00:04:53.040 Now, let’s talk about Minitest Spec, which is the example testing side. In short, Minitest Test is a testing API while Minitest Spec is a testing DSL (Domain Specific Language). Instead of defining classes and methods, you use a DSL to declare your examples. Test cases are 'describe' blocks that contain tests, while tests are 'it' blocks that call various expectation methods.

00:05:07.680 Here’s an example that is equivalent to the previous style: We have 'describe' instead of a class and 'it' instead of defining a test. However, in reality, describe blocks act as classes, while 'it' blocks act as methods. This same example transforms one-to-one into standard code where 'describe' creates a class and 'it' generates a method. There is no magic involved. All of your usual object design code tools exist and operate normally, which is essential.

00:05:39.760 It means that include works, 'def' works, and everything behaves as you would expect. You're just using a slightly different language. Similar to Minitest Test, Minitest Spec has many expectations defined and a similar set of negative expectations. It also shares some of the same missing assertions.

00:06:01.120 All of this is gained for free because each one maps from expectation to assertion directly. Underneath Minitest Test and Minitest Spec is the infrastructure that runs your tests and does so in a way that helps promote more advanced and robust testing.

00:06:13.520 Minitest has randomization baked in, which has always been enabled by default. This randomization feature helps prevent test order dependencies and ensures that your tests remain robust and able to run independently. As a rule, every single test, down to the lowest level, should be able to run by itself and pass; if it requires another test to run before it, then it is not standalone, and essentially, it is buggy.

00:06:32.200 As far as I know, Minitest was the first test framework to implement randomized run order. There's an opt-in system that allows you to promote a test case to run in parallel during execution, which takes randomization to a whole new level and ensures thread safety within your libraries.

00:06:53.760 Minitest was originally a tiny 90 lines of code, but features have been added over time, all while remaining relatively small compared to its capabilities. The reasoning behind the design of Minitest is that it isn’t special in any shape or form; my usual principles apply. If you have heard me ranting and raving before, Minitest is no different.

00:07:11.760 First and foremost, it’s just Ruby—it's classes, methods, and method calls. Everything is as straightforward as it can be. I believe that less is more; if I can accomplish something with less code, I absolutely will. Method dispatch tends to be the slowest process in Ruby.

00:07:32.080 More importantly, less code is nearly always more comprehensible than more code. For example, consider 'assert_in_delta', which is the equivalent of 'assert_equal' but for floats: never use 'assert_equal' on floats, and never use floats for money. There, I've done my usual cautions.

00:07:48.720 This is as simple as possible, with slight optimizations. Here, we use a block to delay rendering the error message in case there is no error—there is no need to incur that cost otherwise. This framework builds on just 15 other lines of code to understand how it works as a whole.

00:08:18.960 Indirection is the enemy; I want errors to occur as close to the real code as possible, with no delays or layers of indirection in between. I feel that Noel really emphasizes my point since he often discusses the layers of indirection that RSpec adds. I want to strip away as many of those layers as possible.

00:08:39.680 I want the responsibility of tests to fall in the right places—no managers necessary, no unnecessary coordination. Objects should handle their own duties. While I may not have the best example of this, 'must_equal' is an expectation that directly calls 'assert_equal' in the current test context.

00:08:58.080 The 'assert_equal' implementation is about three lines long, if I remember correctly, and that’s it—no magic allowed. Even test discovery avoids object space, using minimal meta-programming and relying on plain classes and methods for functionality.

00:09:17.200 I originally wrote Minitest partially to see if I could because I was the maintainer of Test Unit at the time, which terrified me. I also wanted Minitest to be the simplest implementation of a test framework for Rubinius, JRuby, and other Ruby implementations that hadn’t completed full Ruby compatibility. This way, they could receive feedback quickly.

00:09:42.239 Finally, Minitest has a flourishing plug-in ecosystem. I designed Minitest to be extensible, allowing the core framework to remain minimal. Here are just a few of the popular plugins available for Minitest.

00:10:02.559 So, what does this have to do with Rails? The official Rails stack utilizes Minitest. Each release peels back the testing onion, encouraging better practices—though peeling onions can make you cry. Fortunately, that's not the case with Minitest.

00:10:19.040 Rails 4.0 was the first version to transition away from Test Unit to Minitest, specifically using the Minitest 4.x line. At that time, I had already declared that I wouldn’t continue updating the standard library Minitest to Minitest 5; I would only maintain it at version 4.

00:10:39.120 This decision was made because Test Unit is built atop Minitest and has numerous hooks into its internals. That made it extremely difficult for me to update Minitest without breaking Test Unit functionality. So, I froze that version, and Rails updated to Minitest 4.

00:11:00.160 This change removed layers of complexity and indirection, allowing Rails a migration path to Minitest 5. Since Test Unit was already wrapping Minitest, there was minimal impact on users. Arguably, there might have even been a nearly imperceptible performance improvement, but I'm not claiming that definitively.

00:11:19.360 Rails 4.1 transitioned to the Minitest 5 line, which was painful due to outdated tests and many monkey patches present in Minitest itself. This update placed Rails onto the newer, more actively developed version of Minitest, making operations like exec-based isolation tests easier to implement.

00:11:38.640 Among the tests in Rails, some effectively fork a process to run a separate Rails app individually, ensuring that tests running in parallel do not interfere with one another. Although it may have been painful to get Rails switched over, you hopefully never noticed it.

00:11:59.200 Rails 4.2 turned off the alphabetical order of tests and removed a monkey patch that ultimately allowed tests to run in random order. This change was implemented solely to enhance the quality of Rails and your tests.

00:12:15.920 We did receive some reports stating that after updating, some previously passing tests began to fail. I had to isolate a number of bugs within Rails because of this issue. Honest to goodness, despite the pain it may cause, this is a beneficial step—testing order dependency bugs are problematic and notoriously difficult to track down.

00:12:35.279 Later in my talk, I will discuss a tool that can help identify these bugs. In future versions, Rails should be aligned more with Minitest; Aaron Patterson and I keep both in sync, and he informs me when important changes are approaching.

00:12:55.680 So, as a Rails developer, what does all of this mean? Hopefully, if I've done my job correctly, it means nothing. You shouldn’t even have to see Minitest most of the time unless you want to enhance it with specific plug-ins. That's because you’re subclassing Rails test cases, like ActiveSupport::TestCase or ActionController::TestCase.

00:13:13.840 There are about six of these test case subclasses—there might be more now. The basic architecture looks something like this: You write your own test class that subclasses ActiveSupport::TestCase, which then subclasses Minitest::Test. This provides features like per-test database transactions, so you don't have to worry about cleanup; instead, you simply add records that will disappear in the next test.

00:13:31.280 Aaron and I also introduced before and after hooks to assist Rails and other libraries or frameworks in extending Minitest with the extra wrappers they might need. It provides utilities for loading test data and various forms of declarations. Prefer them if you like, but if you don’t, you can still just use the traditional 'def'.

00:13:49.920 It also offers extra assertions such as assert_difference, assert_valid_keys, assert_deprecated, and assert_nothing_raised. Wait, what? Don’t worry! This is the actual implementation of assert_nothing_raised; it's only there for compatibility, and I personally believe that it should be deprecated and removed.

00:14:06.320 Perhaps this can be done in Rails 5. All this means you can write simple tests that describe your current task. Utilities like per-test transactions help prevent clutter in your test code, allowing you to focus on what the test intends to accomplish.

00:14:24.720 This is an essential example which I believe comes directly from the testing section of the Rails guide. ActiveController::TestCase is another Rails extension that subclasses ActiveSupport::TestCase, enabling you to leverage all of those features. It further extends to encompass the usual HTTP verbs.

00:14:43.200 It also simulates web server states and provides assertions specifically related to handling requests—allowing you to easily write concise functional tests, like the one you see here.

00:15:02.240 I love air conditioning so much that I am so dry right now. I’m just going to leave the lid off. ActiveDispatch::IntegrationTest provides full integration testing capabilities as it subclasses ActiveSupport::TestCase, so all the typical features are present.

00:15:22.080 It provides a wealth of assertions too many to fit on one slide, enabling you to write thorough integration tests that can cover multiple controllers.

00:15:42.080 If you are curious about all the functionality Rails adds on top of Minitest, you can find detailed descriptions that are quite informative. Thus, the Rails approach to subclass Minitest Test yields a very straightforward setup that remains powerful, providing everything needed for unit tests through to integration.

00:16:01.920 It leverages Minitest's capabilities with an emphasis on randomization and optional parallelization to ensure better testing outcomes, enhancing the robustness of the tests over time.

00:16:20.240 But what happens when something goes wrong? Perhaps you'd like to use spec-style testing; the problem is DHH disapproves of RSpec to such an extent that he won’t allow the Rails test framework to switch to Minitest Spec. He reverted that commitment.

00:16:39.040 From what I understand, he simply doesn’t want people submitting patches based on spec style, which makes it less accessible. That said, you are not without options if you prefer spec style. You can utilize Mike Moore's Minitest::Rails or Ken Collins's Minitest::Spec-Rails to varying degrees.

00:16:58.160 The two libraries have differences; one aligns more closely with RSpec's style than the other. Alternatively, if you upgraded to Rails 4.2 and now have failures—you might be thinking, 'Ryan, you broke all my tests!' I apologize. Unfortunately, it is not an easy fix, and to clarify, it is a good thing to identify and correct these issues sooner rather than later.

00:17:16.560 Why are my tests failing? This is a classic test order dependency bug, which essentially means tests will successfully pass when executed in a certain order, say A before B, but fail when B is executed before A.

00:17:32.000 If it were only three tests, it wouldn't be a problem and would be quite simple to spot and amend. However, in a large codebase with hundreds of tests, that becomes challenging. To tackle this, I devised Minitest::Bisect to assist in isolating the issues we observed during the transition to Minitest's randomization.

00:18:05.440 This tool helps isolate and debug random test failures by intelligently running and re-running your tests, narrowing down the list of potential culprits.

00:18:23.000 Here’s a simple example of it at work. Please ignore the fact that I'm pre-specifying the seed; pretend I'm running this anew, and we receive this failure.

00:18:38.720 We obtain the failure and grab the random seed, then we rerun the test using Minitest::Bisect. This action will re-execute the entire test suite to ensure reproduction.

00:18:58.240 Once confirmed, it begins to isolate the problem area down to the minimal subset of tests. As you can see, the number of tests reduces dramatically.

00:19:22.800 You are left with two tests run in a specific order that can cause reproduction every time. I've learned I’m not alone in facing this issue.

00:19:41.440 Or perhaps you are accustomed to a kitchen-sink approach to development. To start, just try it out—it might work! Many libraries, including Mocha, are already compatible with Minitest and perform quite well.

00:19:58.960 If not, you’re not alone; someone has likely already encountered your issue. Search for existing plugins listed in the Minitest README, check RubyGems.org, Stack Overflow, or other similar resources.

00:20:16.240 However, my suggestion—albeit contrary to popular opinion—is to try simpler testing first. Only integrate plugins after you've determined you truly require them. Starting fresh and clean could yield beneficial results; you might actually enjoy it!

00:20:32.560 Change takes time, so remember to measure your before and after states to gain an objective perspective. I can only provide anecdotes about projects speeding up after switching to Minitest. I would love to obtain more data supporting this view.

00:20:54.560 I've heard individuals report that they halved their test times by moving from RSpec to Minitest, but once more, I lack objective data.

00:21:17.040 So, you might be wondering why you should care about all this Minitest discussion. The first point I’d make is that if you’re not open to testing, you're a lost cause. There's ample data demonstrating the benefits of testing.

00:21:34.240 If you cannot accept that notion, I would prefer to assist others. That said, let’s consider this issue in detail.

00:21:52.080 The official Rails stack employs it, meaning that DHH uses Minitest. Tenderlove uses it as well, and Jeff Casimir and his cohort teach Minitest at a touring school. Other popular gems, like Nokogiri or Haml, also utilize Minitest.

00:22:14.480 In fact, more than 4,000 gems declare dependencies on Minitest. Unfortunately, since Minitest is part of the standard library, many gems neglect to declare their dependency, so there are likely even more.

00:22:37.040 What are the real functional differences between Minitest and RSpec? As testing frameworks, there is much overlap, so I won't delve into that. Where they diverge is where things become interesting.

00:22:52.160 To be fair, RSpec offers much more than Minitest; features like test metadata, filtering, additional hooks like 'before' and 'around', implicit subject, described class, complex mocking, and more.

00:23:12.080 Essentially, RSpec is fancier. However, what Minitest offers in return are benchmarking, speed, and parallelization, making it simpler and more pragmatic.

00:23:32.000 The cognitive differences between Minitest and RSpec fuel a significant divergence. This was driven home by Myron Marston a few years back, who provided a compelling response on Stack Overflow comparing RSpec and Minitest.

00:23:51.360 While there was a degree of bias, I believe it was a balanced overview. Unfortunately, there is a lot of information to digest, so let’s focus on an essential paragraph.

00:24:06.160 Let’s color-code the points: red for RSpec, blue for Minitest. Myron suggests this is why RSpec is great, though I’d argue it reflects everything wrong with RSpec, but we are both right—philosophically speaking, we have different goals and perceptions of quality.

00:24:27.360 To break it down further: Minitest compiles blocks down to simple classes, whereas RSpec refines testing concepts into first-class objects. The first term here signals a significant red flag for me regarding RSpec.

00:24:48.760 Examples in Minitest compile 'it' blocks into simple methods, while RSpec uses first-class constructs. Simple methods perform assertion expectations, whereas RSpec employs first-class matcher objects.

00:25:09.440 Further distilling this: Minitest uses classes, while RSpec uses first-class objects. Minitest relies on subclasses or includes, while RSpec relies on first-class constructs, meaning that the methods utilized in Minitest are more straightforward.

00:25:30.720 The term first-class indicates that you can assign a value to a variable and use it as you would any other value. Almost everything in Ruby is considered first-class, which doesn’t make it a distinguishing factor.

00:25:52.160 Consequently, where I could, I employed Ruby's mechanisms, whereas RSpec often reinvented the wheel. To illustrate how RSpec complicates matters, consider an example with two nested describes, each with a 'before' block and an example.

00:26:10.160 The 'before' blocks appear to be inherited, while examples are not. You end up with two runs: the first with one 'before' block and the second with two. For 'A' and 'B', classes nesting is a subclassing example; if they are classes, we require a method to prohibit inheriting tests.

00:26:31.440 Minitest’s approach is simpler: I loathe this runtime behavior that RSpec anticipates while also complicating the scenario. Presumably, if 'before' and 'after' work like included modules, we don’t need to define multiple methods, but we still need to generate inner modules, and setups must intelligently invoke super. That's cumbersome.

00:26:52.240 In essence, RSpec necessitates learning an additional object model that sits above Ruby's own object model. This presents complications, especially for those new to Ruby or learning it

00:27:15.040 at the same time, which can be overwhelming. Consequently, users may feel encouraged to gloss over details, using descriptions as a magic incantation rather than understanding their function.

00:27:30.799 If you missed the preceding talk, much occurs within 'describes,' and it usually boils down to a simplified instruction set to achieve the desired outcomes. Arthur C. Clarke remarked, 'Any sufficiently advanced technology is indistinguishable from magic.' I muse this means RSpec operates as such.

00:27:47.360 As previously mentioned, many people consider RSpec overkill, introducing a cognitive cost due to the added complexities. Therefore, it is valuable to highlight the raw numbers contributing to this added intellectual burden.

00:28:03.920 Let's visualize these numbers in the next slide. Here is a complexity metric called Flog, which indicates the difficulty of testing, debugging, or grasping the code. RSpec is approximately 6.6 times larger, with the combined code and comments at 8.5 times greater, akin to reading Dr. Seuss compared to James Joyce.

00:28:27.360 You can see my reference to Dr. Seuss perfectly represents my notion of distinction between the two frameworks.

00:28:42.160 Beyond cognitive burdens, performance variances also arise. These various abstractions that RSpec utilizes introduce real costs. The presented data illustrates runtime complexities for both systems.

00:29:02.319 What you will highlight are two significant problems: RSpec’s runtime for failures grows exponentially, while Minitest’s is low and linear. The impacts of failure in RSpec lead to consistently longer runtimes.

00:29:19.440 As for the assertion speed in both frameworks, they are linear. Therefore, no attempts should be made to reduce test quality for speed; Minitest will generally allow for faster testing.

00:29:35.520 You have the option of switching to Minitest or avoiding issues around refactoring or bugs. In summary, at the day's end, I don’t care what framework you utilize as long as you ensure your tests are in place.

00:29:52.160 Hopefully, I've illustrated the technical merits of Minitest. Choose the tool that works for your purposes, not based on popularity. For many, the argument arises surrounding RSpec having more available documentation.

00:30:08.240 Perhaps fewer articles arise due to testing in Minitest needing less explanation. Minitest tends to be much easier to comprehend, allowing you to read it in a matter of hours and grasp its entirety.

00:30:20.800 It might also suggest that Minitest users are busy accomplishing tasks instead of reading and writing articles. In any case, choose what functions best for you. You might just appreciate it; after all, it’s just Ruby.

00:30:38.239 Thank you.

00:30:56.239 And please, hire me!