00:00:10
Looks like that's everyone, so hello! Thank you so much for joining me. I'm Ryan Davis, and today I may be talking about Minitest 6 and what's coming to help you test in a feistier way.
00:00:22
Setting expectations up front is something I always like to do in my talks. This is an overview or idea talk, also known as a hand-wavy talk. There will be little code in it. It is targeting testers at all levels.
00:00:38
I actually updated my presentation to include 265 slides within 40 minutes. That's about 7 slides a minute, which is an okay pace. Something I don't normally do, but I'm going to go outside of my comfort zone and share my excitement.
00:01:01
Last night, I realized, a bit drunkenly, that I have now released more than a thousand gems. I'm thrilled to have hit that milestone. Now that's taken care of, let's go meta for a second.
00:01:15
What's the narrative for a talk about a test framework? I can't think of one either. As such, I'm stuck doing what is known as a presentation sandwich.
00:01:30
A presentation sandwich is where you tell the audience what you're going to tell them, then you tell them, and then afterward, you tell them what you just told them. This makes me sad because I don't enjoy these types of talks. They always feel basic and a bit insulting, as if the audience can't track without the extra reminders.
00:01:54
As an aside, I thought it'd be fun to do my entire talk in emoji, but you end up having to learn to speak in a Neanderthal-like manner to do it. Go hamburger, bubble, eagle, bad! It was taking too much time. So, this is about the last one; I certainly don't like giving these types of talks, as it feels hypocritical to me.
00:02:14
I really prefer a nice story arc or some device that ties everything together, but there isn't one here. My goal today is to inform you about what's happening with Minitest 6, and for that, I apologize. This just isn't the type of talk where a narrative is easy to come by.
00:02:33
I'll do my best to keep you engaged. To help with that, I'd like to know why you are here. If I can get you to raise your hands for the following categories, that would be great.
00:02:50
Maybe you're new to testing and want to see what your options are? All right, about five of you. You're a Minitest user already, and you want to know what's new?
00:03:05
That looked to be about two-thirds of you—half to two-thirds. You're an RSpec user, and you want to know how Minitest differs? Nice! Awesome!
00:03:19
You're an entrenched RSpec user, and you want to heckle. All are welcome, but just know that I heckle back.
00:03:32
Minitest is my baby; this project is a labor of love for me. Testing is something I've been passionate about. I've been doing unit testing in one form or another since about 1995.
00:03:43
This was actually before Kent Beck's seminal XP book came out, which really popularized the practice. TDD stands for Test-Driven Design and Development, coupled with YAGNI and other practices from XP.
00:04:04
I believe that TDD leads to better software almost every time. That's where Minitest comes in. I wrote Minitest using those practices, and I wrote Minitest to help support those practices.
00:04:15
What am I talking about today? I'm going to give a quick introduction to Minitest so you understand what it's about, in case you don't know.
00:04:30
I will also give a very brief history of Minitest, so you understand where it's coming from. Then, I'll discuss the philosophy and purpose of Minitest, so you better understand the decisions I've made over time.
00:04:47
Finally, I'm going to detail the major changes coming in Minitest, both architectural and mostly invisible changes that you may not notice as a user, and code changes that you are more likely to see.
00:05:04
I am not trying to convince you to test. That ship has sailed! If you missed it, you can't be convinced. This talk is not for you, and you're probably better off in a different session.
00:05:19
I am also not trying to convince you that you should use Minitest. If you do, that's fantastic! If you don't, that's okay too. Hopefully, this talk will be informative for you.
00:05:31
And I'm not here to bash RSpec, as much as it is fun to do sometimes. After all, the maintainer likes to get drunk together.
00:05:41
While I might provide statistics that you may not like, the following conditions hold true for all testing frameworks, whether it's test time, memory impact, or the number of users and downloads.
00:05:56
But really at the end of the day, I don't care what you use, as long as you test. Please understand that in its entirety; I do not care, as long as you test.
00:06:09
That said, maybe Minitest is faster and more direct, and it always will be. I'm also not here to explain how Minitest works, just the 'what' and the 'why' about Minitest and how it's changing.
00:06:22
If you want to know more about how Minitest works, Nate Birkett gave a really good talk at RailsConf three years ago, walking through Minitest 3.
00:06:37
I've also given a couple of talks on how to create a test framework from scratch that builds up closely to the design of Minitest. My approach is more bottom-up, while his is more top-down.
00:06:52
If you watch both, you'll cover all the bases.
00:07:05
Thus concludes the first slice of bread. With that done, let's start with a quick introduction. Minitest is a testing framework that supports unit and functional testing using regular classes, methods, and assertions. This is known as XUnit style for some or Test style for older developers, and it also supports using describe blocks and expectations, which is known as Spec style.
00:07:34
It supports benchmark tests for platform-agnostic performance testing and has simple stubbing and mocking. Some might say it's too simple, but that's actually intentional, as we will see.
00:08:11
Minitest supports both TDD and BDD. This has nothing to do with whether you prefer Spec style or not; that's a common misconception.
00:08:34
It has a very clean architecture so you can understand all the code in one short reading, which usually takes about an hour. It's fast—it's very fast! It's the fastest out there of the full-fledged test frameworks, and its job is to get out of your way.
00:09:07
Minitest has a flexible reporting architecture that allows you to customize the output and integrate it with other systems like CI, and it's highly extensible using plugins from RubyGems. There are over 200 gems for Minitest. I learned that recently.
00:09:29
Now, with that other way, let's do the briefest history of Minitest that I can muster without being sarcastic and skipping the whole thing. History talks bore me to death, so this will be very brief.
00:09:39
I became the maintainer of TestUnit, which shipped with Ruby in standard lib, back in 2004. It was big and complex. I just didn't understand it, and frankly, I was scared of it, especially as a new maintainer with a core commit bit.
00:10:07
That was terrifying, with a lot of confusing files and a fair amount of code. It was overly complex for what little it did and had five different runners, including four different GUI libraries.
00:10:23
I had two collectors, or ways of discovering the tests to run, and it used every design pattern that you could find. Some of this was not the fault of TestUnit; it was just following in Ji's footsteps, which were following Smalltalk’s Test footsteps.
00:10:41
Even having its own text template engine for generating error messages for every assertion, even those that passed. Everything boiled up to something called assert_block instead of assert, and I could never figure out why.
00:11:05
Generating those extra blocks and closures was expensive and needlessly slow. I thought, what does it take to implement only what I use? What is the simplest thing I can do to ensure that my code is correct?
00:11:31
That was Failure-Driven Development at its most extreme. I ran my tests bare, defining everything that was undefined. The first thing was Test, then Unit, then Test Kitchen, and so on, until it ran and eventually passed all the tests across all my projects.
00:12:07
I had about 25 projects, and the best part was that it was only 75 lines of code. I don't use a diverse set of assertions. I'm a big fan of a specific few, and I use others now and then.
00:12:34
So, really, I had to implement just what I was using, which is part of why it’s called Minitest. You can sort of read this: on the left, you have the entire framework in one method, finding all the classes, running their tests, and setting up.
00:13:00
On the right is the TestCase class, with setup, teardown, stubs, asserts, and all the other very simple assertions underneath it. That's it! Here's the history in a nutshell.
00:13:20
Version 1 came around in 2006. Version 1.2 included all the assertions that I didn't use. Specs and mocks came in around 2008, and then benchmarks in 2010.
00:13:49
Version 5 arrived in 2013, which is really where we're at now. In February, I released the latest version, which is 5.11.3. That's the current function. The only real difference between that and 5.0 is that I've expanded the reporters a bit, and it's mostly an internal refactoring to make things easier for plugin writers.
00:14:06
To clarify, Minitest has been around for 11 years, and I hadn't realized that until writing this talk. This feels like an obligatory reference.
00:14:32
11 years! This stuff really sneaks up on you. So I think this is more interesting data than a table of dates and descriptions.
00:14:58
This is a visualization over time; the blue line represents the lines of code, while the red line shows a complexity metric from a tool I wrote called Vlog.
00:15:36
By comparison, here's RSpec. Unfortunately, due to a repo split when they changed hands, I could only gather numbers from version 2.0 and 3.5, which I think is the latest.
00:15:57
Here are the Minitest numbers on the same scale. Minitest has had a number of firsts as a test framework, possibly in any language; it’s difficult to verify.
00:16:22
It was the first test framework to shuffle tests by default, which prevents test order dependency bugs that can lead to production bugs.
00:16:46
It was the first to include both test and spec frameworks, as well as TDD and BDD style support, providing options for a developer to choose any style they want.
00:17:07
Mike Moore, who used to run the Mountain West Conference, really likes to do what he calls Spec Assert style. He uses describe blocks but uses asserts instead of expectations.
00:17:27
It was the first framework that I know of to provide benchmark testing, offering platform and hardware-agnostic performance testing.
00:17:46
The advantage of this is that both your very fast laptop and your very slow CI can pass tests, and you don't need fixed numbers or have to figure out the hours of complex math between the two.
00:18:03
Now that you know a bit about the 'what' and 'why' of Minitest, let’s look at the purpose and philosophy behind it.
00:18:15
As I said before, I subscribe to a lot of XP principles: one failing test at a time, then make it pass. This approach seems intuitive at first glance.
00:18:29
But it serves as a perfect carrot to keep you going without leading too far ahead of your actual needs. Do the simplest thing possible.
00:18:49
I'm not going to try to say that phrase; I've tried like ten times, and I can't. This principle pairs well with the test-first approach because you're never more ignorant than at the beginning of a project.
00:19:12
So, why pretend that you can design for all complexities up front when you aren't? You aren't going to need it.
00:19:33
At least YAGNI (You Aren't Going to Need It). Don't implement anything until you actually need it. This avoids the 'what about' switch that kills projects.
00:19:53
All of this tries to focus on being incredibly simple, or at least as simple as is reasonable for your needs. Ruby has one of the most complex grammars among today’s programming languages.
00:20:12
And don't even get me started on the semantics; they're crazy! There's a lot to learn just to get up and running, let alone semi-productive.
00:20:30
Both Minitest and RSpec look like magic to a beginner. To a beginner, Ruby itself is magic.
00:20:48
Beginners don't know about singleton methods on instances or why you'd want to use them.
00:21:01
Sometimes we lose touch and forget that. I don’t want Minitest to contribute to the bewilderment or at least contribute as little as possible to it.
00:21:19
Your test framework shouldn't get in the way of learning or doing, but once you do learn a new construct, that beginner can start to use that new tool anywhere.
00:21:31
Minitest tries to expose Ruby with little magic involved. To use Minitest, all you need is a basic understanding of classes, objects, methods, method calls, and the ability to read some basic API documentation. That's it!
00:22:01
There’s nothing extra to learn just to use it. Once you learn a new tool from Minitest, like singleton methods, you can use it anywhere.
00:22:18
That said, you should be careful because singleton methods will bite you!
00:22:31
Here's a minimal test that will actually run as is, and then it will fail because you haven't implemented the calculate method yet.
00:22:48
There's a required 'autorun' at the start of the file. You have a class with a descriptive name; it doesn't have to have tests in there, but I usually do.
00:23:01
It subclasses Minitest::Test, then you have a method that starts with the word 'test'. That's important, and you have an assertion that compares an expected value with an actual value.
00:23:15
Here's the equivalent in Spec style: the same require statement, a describe block naming the class under test or appropriately wrapping it.
00:23:33
You open a block, stating what it should handle, and you're performing the calculation while asserting that it must equal the expected value.
00:23:55
You've probably noticed that Minitest's test and Spec styles are nearly identical. This is because Spec things are essentially bridges to assertions. I don’t want to reinvent the wheel, nor do I want extra things to learn or teach.
00:24:15
Minitest strives to be incredibly simple while also driving to be incredibly fast. It's been a while since I ran my benchmarks, but Minitest is the fastest of the full-fledged test frameworks out there.
00:24:36
By full-fledged, I mean it has full functionality as you'd expect, and it's test-safe. There's a library out there called Riot that doesn't instantiate an object to run your tests, which means you can use it to test everything.
00:24:58
I don't consider that to be full-fledged. It is slightly faster than Minitest, but that's because I'm doing the real thing, and it’s not quite as pretty.
00:25:13
Minitest will always be the fastest out there and achieves that by doing nothing extra. In short, 'autorun' at the top triggers off 'run', which gathers all the test classes.
00:25:33
Then, each class runs its own tests, and they go off to do their thing. Another way to visualize this is through a call diagram to run tests.
00:25:50
This call stack is pretty clean and easy to understand. In the future, I’m thinking of ways to clean this up even more.
00:26:08
One way to remain incredibly fast is to store nothing extra. We do this to reduce the impact on garbage collection; we avoid creating unnecessary objects and dispose of everything as soon as we can.
00:26:24
You'd expect memory consumption to be linear with the amount of testing you're doing, but in Minitest, it's linear to the amount of failures and errors that you encounter.
00:26:43
The statistics reporter class only keeps track of failures and errors. In RSpec, the memory usage is exponential relative to the number of tests you're running.
00:27:00
I'm not sure why that is, but I think it’s easy to see the overhead of generated tests and basic UNIX tools. I think that's one of the main reasons why it’s much slower.
00:27:22
So here's a story: there’s a gem called 'minitest-rspec' that ironically uses my Ruby parser and the 'sexp' processor to convert my test suites to RSpec.
00:27:42
For fun, I ran this against one of my projects that was simpler and cleaner; it didn't have any shared tests or anything.
00:28:02
The tests in Minitest were already fast, which is what I wanted; I wanted to benchmark the frameworks against each other, not the test content.
00:28:20
However, in RSpec, the tests were 30% slower. Is that definitive? Of course not; this is not science; it's just funny statistics.
00:28:40
Minitest is opinionated in order to improve testing overall. I have this idea of a 'testing knob.' The idea is that your framework either helps or hinders your testing mojo or your feistiness.
00:29:00
Starting with an incredibly fast framework means providing only meaningful assertions to reduce nonsensical tests that offer a false sense of security to testers.
00:29:20
I'm looking at you, assert_nothing_raises! You give the user the choice of test or spec style, provide test randomization to prevent test order dependency bugs, and design testing for results and side effects.
00:29:38
Encouraging a mock-last strategy makes good testing practices easy while making questionable practices harder.
00:29:58
You benchmark tests against your algorithms, improve expressiveness through custom assertions, and abstract test classes to require assertions, ensuring that you're actually checking something in every test.
00:30:20
This was a user contribution I hadn't thought of; I realized that when I turned this feature on, I ran all my test suites and got failures because sometimes checks slip through the cracks.
00:30:39
I’d also like to mention parallel testing. I'm going to try to enunciate this clearly throughout my talk to drive home solid and thread-safe strategies.
00:30:59
Later, we will discuss my ideas for aiming to go to eleven.
00:31:19
Reproducibility is crucial. Minitest is pretty much just a fancy tool to reproducibly run things. All it does is call methods.
00:31:39
In testing, reproducibility is paramount. Without it, there’s really no point in testing at all. You might as well throw out your tests alongside your implementation!
00:31:59
Running with a fixed seed should always reproduce output, allowing you to see failures in the order they occurred previously.
00:32:21
Being able to selectively run failures helps reduce the problem set to a minimal reproduction, and if absolutely necessary, you can declare that your tests need to run in a specific order.
00:32:43
Finally, Minitest does not push a testing style, strategy, process, or pedagogy. Personally, I like the flexibility of this approach, but some people feel lost because of it.
00:33:05
If I were to endorse a testing style, it would be something like test-first. I love to conduct quick and dirty sprints, but when writing anything I know I’m going to keep—like during exploratory programming—I always write a failing test first.
00:33:27
The design is always better in the end, especially when you're still figuring out what you want your creation to be. Sticking to the essence of what you know keeps complexity, and therefore bugs, at bay.
00:33:49
Just like premature optimization, I believe over-mocking is the root of all evil. I've seen tests mocked so heavily that they can't fail, even if you remove the entire implementation.
00:34:09
So, set up a thing, call a method, and verify the result or side effects. Nothing can be simpler than simple stateless tests. If you know this game and don't know what to test, this is a perfect rule of thumb.
00:34:31
I've talked to Kent about testing strategies, and one thing he told me that I really liked is that getting a pass (green) means you don't have enough tests yet! You have a gut feeling that you know when you need more tests; it’s instinctual.
00:34:53
To understand where Minitest 6 is trying to go, it might help to see Minitest compared to versions. One of the biggest things about version 4 to 5 was the namespace changes.
00:35:12
I added a compatibility layer for users, but it still bit plugin authors. I moved benchmark testing to its own class, so they could be isolated.
00:35:30
I removed all the cruft from Minitest 4 and 5.
00:35:48
Old compatibility assertions that have been deprecated for ages are gone. If you still need that cruft while working on the upgrade path, here's a compatibility chart.
00:36:03
If you need your test suites unshuffled, you'll need to pin to version 5.3.3.
00:36:14
5.11.x will be the last real content release. Version 5.12.0 will simply turn on deprecations.
00:36:31
But who cares, let’s get to the meat. Parallelism has been a significant feature discussion for quite a while, but I’m considering an overhaul.
00:36:51
Currently, I’m exploring using parallelism to opt into parallelization on a class-by-class basis. I'm seriously considering making this the default.
00:37:07
Maybe we can use serialize_me to opt out or perhaps use a different superclass for serial testing.
00:37:23
Parallelization is the next step to ensure a solid, thread-safe design. It is the next logical step to make implementations scale.
00:37:45
I’m also seriously thinking about running all tests completely interweaved rather than on a class-by-class basis; I will discuss this more in a moment.
00:38:01
Easy worker-based distributed testing would allow running tests across multiple CPUs and avoiding locks while helping with tests that stub global methods, like Time.now.
00:38:22
Here are some real speed benefits. Jeremy Evans contributed numbers from his gem, ParallelFork.
00:38:39
I think the only difference between that project and what I’m exploring is its ability to actually generate tests across multiple machines.
00:38:54
However, Forking allows you to keep tests running concurrently and distribute your test runs across machines or VMs.
00:39:12
This worker-based distributed testing minimizes run times automatically and helps with CI and other systems that are resource-constrained.
00:39:29
How do you configure and set up for distributed testing? Well, I don't know yet!
00:39:40
What easy means at this point is still up for debate. Aaron Patterson has contributed a patch to provide distribution without any configuration.
00:39:58
Everything is up for discussion now. Maybe there are some assumptions about how to access your tests across machines, such as needing SSH access, and that you have the proper ports open.
00:40:14
Or that you have the same source code layout across all machines. This should be reasonably safe for container-based testing.
00:40:29
However, I have no feedback on non-UNIX systems, so I will definitely need some help here.
00:40:46
If you're interested in existing libraries for extending parallelization or distributed testing, you can check out TestQueue used by GitHub, or Knapsack, a popular project with a paid service.
00:41:05
I know that Shopify has a really elaborate setup.
00:41:20
Currently, the result of a test is the test instance itself. But tests can assign instance variables that cannot be serialized, like a Hash with a default block.
00:41:32
Minitest is switching over to a new setup that guarantees safe serialization. This deletes custom Marshal code and adds a Result class that pulls the bare minimum needed for proper reporting.
00:41:53
This simplifies parallel and distributed testing, and makes garbage collection easier, as we aren't holding on to closures accidentally.
00:42:06
Remember, closures see everything above them that they can touch.
00:42:29
They hold onto that until they get garbage collected, so if one's live, all those objects are live.
00:42:42
Ruby 2.5 added the ability for an exception to point back at its previous exception.
00:43:00
If you have an error that raises and rescues, and then raises again, both of those point back to the original.
00:43:12
This leaves the serialization hole open, so I need better tests to expose and fix it.
00:43:27
Minitest 6 will feature a simpler project structure. I'm falling behind on some of this.
00:43:36
Some files were getting hefty, like reporters, which have been broken out into their own file.
00:43:49
Before, I would really like to make the call stack cleaner, but I'm not sure it's possible without bringing complete havoc.
00:44:14
I previously gave a walkthrough of the current call stack, which runs through everything.
00:44:36
Auto run fires off 'run', which fires off 'double_underscore_run', which walks through all the runnable tests and runs them.
00:45:00
If we refactor some things, we might be able to pull out filtering from 'double_underscore_run'.
00:45:17
Extending 'runnable_x' to return both classes and their methods could allow us to shuffle across all tests.
00:45:35
I have a final vision: clustering everything together in chaos, but I'm not set on how far to take it.
00:45:50
This could break a lot of current tests, but I’m not sure if they are meant to break.
00:46:05
There's a lot to say about parallel by default. It’s an incredibly powerful tool for driving safer design, but it also has some problematic aspects.
00:46:22
Testing strategies like non-transactional database testing or stubbing class methods face challenges since class methods are essentially global functions.
00:46:38
That's all for the architectural changes; it’s not much, but there’s still plenty to consider.
00:46:55
Regarding code changes, many people have been requesting a command line runner, so I'm adding one.
00:47:12
I'd like to add the usual bells and whistles without pulling in extra dependencies, but running tests by line number might be difficult to manage.
00:47:30
What I appreciate about this is that it leads to better integration with other tools, like bisect or auto-test.
00:47:47
Rake's built-in test tasks are cumbersome at best; I actually hate it and do not use it.
00:48:02
I have my own, and it makes passing flags around complicated. Perhaps I should stop hoarding my own and just provide a clearer way.
00:48:16
There are some minor but important improvements to Minitest, including mocks and cleaning up testing stubbed methods that take blocks, which can get quite hairy.
00:48:37
It's much more consistent now than in the past. I'm considering adding a 'fake', which is just a fancy wrapper around 'Object.new'.
00:49:07
This singleton method initializer seems simple, but perhaps it doesn’t need to be added; that’s still open for discussion.
00:49:23
Assertions have been notably improved. Using an 'L' value is no longer allowed in certain statements; this drives out tautological testing.
00:49:41
Randy Coleman has a fantastic blog post about this; I highly recommend it. I can’t explain it all in just six minutes.
00:50:01
Assertions now reuse themselves more effectively. For example, 'assert' checks will call 'assert_response' first.
00:50:16
This significantly improves error messages, transforming a vague error into a failure that tells you what's wrong.
00:50:33
I've also improved failure message construction. If messages are passed as a proc, then that proc overrides all other output.
00:50:49
This allows user-defined messages to win when passed down to wrapped assertions. I've removed the last of the nonsensical assertions like 'assert_send'.
00:51:06
These have been replaced with better assertions like 'assert_predicate', 'assert_operator', 'assert_includes', etc.
00:51:22
Specs have also been cleaned up a bit. Since expectations are just assertions, all those improvements now apply, but I've also deleted all Minitest expectations from objects.
00:51:55
Adaptation to the new setup is pretty straightforward: instead of calling 'must_equal' directly, you now have to go through the wrapper.
00:52:12
I prefer to wrap the expected value using an underscore since it’s short and visually unobtrusive; however, the word 'value' is more descriptive.
00:52:26
Honestly, I find 'expect' terrible, but it's there too, which helps clear up the global namespace.
00:52:41
To summarize, first, I've given enough background to understand where Minitest is currently and how it got there.
00:53:05
I've gone through the major changes coming to Minitest 6, including architectural revisions like parallelism, distributed testing, and serialization guarantees needed to run smoothly.
00:53:25
I've also covered the possible overhaul of the call stack and the implications that may carry with it.
00:53:48
Additionally, I discussed the code changes coming that will impact both test plugin writers and the Minitest community, such as the command line runner.
00:54:08
I've introduced better rake integration, cleaned up stubs, improved assertions, and clarified failure messages.
00:54:31
Remember that testing knob? That's really what Minitest is about—to increase it as much as possible! Be feisty, but select the settings that work for you and your project.
00:54:59
If you want to help Minitest get to version 12, I’m always open to ideas that promote better testing practices.
00:55:19
So, in conclusion, when is this coming out? That's a good question. Much of this is being done in an experimental stage and I could release an alpha release soon.
00:55:34
However, some aspects aren't fully complete yet. There’s still extensive debate and evaluation of practices that need to be baked in to see how well they work.
00:55:51
I’m thinking in terms of weeks or months and not really beyond that. You know stalling on releases isn’t my style, but this depends on community feedback.
00:56:12
I could really use your feedback and contributions. My current draft of this code is available on GitHub, at the URL above, and on the MT6 branch.
00:56:34
Today, I will also post these slides on my website and tweet them. Thank you! If you appreciate and benefit from my work, I would appreciate it if you or your company could consider sponsoring me.