Writing a Test Framework from Scratch

Ruby on Ales 2016

Ryan Davis

@zenspider

#test-framework

#minitest

#test-driven-development-tdd

Writing a Test Framework from Scratch

by Ryan Davis

In the video 'Writing a Test Framework from Scratch', Ryan Davis, a renowned Ruby consultant, guides viewers through the fundamentals of constructing a test framework, starting with assertions. The presentation emphasizes the significance of assertions as the core components of any testing framework and demonstrates how to develop one from the ground up.

Key points discussed throughout the video include:

Introduction of Assertions: Davis begins by explaining assertions, citing them as the simplest way to evaluate expressions in tests. If an assertion fails, it throws an exception, halting execution and highlighting errors promptly.
Building Assertions: The talk elaborates on creating an equality assertion, emphasizing the importance of clearer error messages for easier debugging. An additional assertion function is introduced for comparing floats, avoiding direct equality tests due to potential precision issues.
Organization of Tests: Emphasis is placed on wrapping tests in methods and classes to maintain independence of test variables, thereby reducing side effects between tests.
Dynamic Test Execution: The concept of class instances managing the execution of their test methods is covered, promoting efficient handling of tests without external management.
DRY Principles: Subclassing is recommended to leverage shared behaviors and streamline the reusable code amongst tests.
Reporting Results: The need for a robust reporting mechanism is highlighted, allowing for clear feedback on test outcomes and efficient output handling, separating successes from failures.
Final Enhancements: The presentation concludes with suggestions for future refinements that can emerge from the foundational work established during the talk, including dynamic test registration and enhanced output clarity.

Overall, Davis insists on the vital nature of core principles in testing frameworks, suggesting that understanding these principles leads to informed decisions in building and improving test frameworks. He encourages attendees to explore assertions in their coding practices and engages with the audience for further questions and collaboration.

Conclusively, the foundation laid during this talk serves as an essential springboard for further advancements in test framework development, promoting the principles of simplicity and maintainability in programming.

00:00:15.440 Thank you everyone for coming here. As he said, my name is Ryan Davis. I'm no stranger to the world of Ruby; I'm a founding member of the CLR Be, the first and oldest Ruby aid in the world. Currently, I'm an independent consultant based in Seattle and I'm available for work.

00:00:30.810 I'm also the author of many tests, which happens to be the most popular test framework in Ruby. I mention this because I’m somewhat astounded that the r-spec is also quite popular, apparently ranking high.

00:00:38.190 Toby's talk earlier was amazing; in just 49 slides he managed to describe the physics of the solar system, which is quite mind-blowing. I feel like I've missed the mark here, only describing a test framework with 30 to 32 slides. So, let’s segue into setting expectations. This talk will be code-heavy; the focus will be on the what, why, and how of writing a test framework.

00:01:06.299 There are 337 slides, which amounts to about 9.5 slides per minute—fifty percent more than I’ve ever done before. That means I’ll be talking quickly, and I encourage you to open an editor if you want to follow along, as I was informed that many attendees got halfway through last time and then lost me. However, do not worry if you fall behind; you can download the GitHub repository and follow along at your own pace.

00:01:39.030 This talk is an adaptation from a chapter of a book I’m writing on many tests. By the end of the session, you should have a solid grasp of how to write your own basic test framework from scratch, focusing on assertions, which are the atomic unit of any testing framework.

00:02:22.900 The famous quote not actually attributed to Benjamin Franklin states: "Tell me and I forget, teach me and I may remember, involve me and I learn." The exact origins of this quote are somewhat murky; it has been attributed to many, but the essence highlights a significant issue with code walkthrough talks. While some code walkthroughs are beneficial for understanding implementation, they can often lead to a passive learning experience where you might remember less than you hoped.

00:03:02.790 The real challenge is when the focus shifts solely to the implementation itself—what each line of code does—resulting in a dry and forgettable experience. With that in mind, I won't be merely discussing a specific implementation, but will instead demonstrate the principles of building a test framework from the ground up, using assertions as our starting point.

00:03:42.970 Neil Gaiman's famous quote could be paraphrased as: "For storytelling, begin with a simple assertion." So let’s start with the simplest form of an assertion, which basically takes a single expression and evaluates its truth. If the evaluation fails, the assertion throws an exception. To begin with, here is the plain old assert in its simplest form.

00:04:22.460 The assert function will evaluate to true if the expression passes; it fails on anything that does not. At this point, all you really need to know is how to create an assertion. There are various approaches to asserting values; for example, you could choose to raise an exception or simply collect results—and while each method has its trade-offs, the critical aspect is to report a failed assertion at any point.

00:05:04.639 I opted to raise an exception because it interrupts execution and immediately draws attention to the error. Now if we run the expression with a true value, the assert function would handle it just fine. However, when tested with a false value, it raises an exception and halts the test suite until resolved.

00:05:41.080 It’s important to note that this failure reporting currently indicates where in the assert the raise occurred; however, for practical clarity, we want it to reflect where in the actual test the failure originated. The stack trace must lead back to the line of code where the assertion was made, rather than where the assertion function was called.

00:06:22.270 We can clean this up by altering our exception handling code. By using Ruby's built-in caller function, we can specify that we want the backtrace to show the relevant lines where the failure occurred, ideally enhancing the developer's ability to debug effectively.

00:06:49.055 With the plain old assert mechanism established, let’s now add a second assertion function. The next most required assertion for my tests is to check for equality, a requirement in at least ninety percent of my tests. The implementation is remarkably straightforward: simply pass the result of a equals b to assert.

00:07:36.880 However, while the implementation passes functionality tests, the error message for failed assertions needs improvement. The error message references the assert equal in the backtrace, which, while useful, lacks immediate clarity if it doesn't point back to the specific line that failed.

00:08:29.780 To enhance clarity further, let’s allow for optional error messages when assertions fail. This adjustment will lead to clearer output that identifies exactly what went wrong. When it comes to floats, a critical lesson I wish to emphasize is: never test floats for equality.

00:09:20.890 Instead of using assert equal as you would for integers, we will develop a new assertion solely for comparing floats that checks whether the difference between the two numbers is within an acceptable margin of precision, which we're defining as 0.001 for now.

00:10:03.420 With this newly created assertion function, we now have a generalized assert function that can cater to various assertion types, including our special float assertions.

00:10:37.240 From here, it would be beneficial for you, after the conference, to think about your favorite items to test and consider how you would write assertions for them. Writing your first test should encourage you to write many tests, prompting you to separate them in the name of organization, refactoring, reuse, and clarity.

00:11:39.770 Methods offer a straightforward approach to separating individual tests from one another, allowing for greater organization and clarity in their execution. By simply defining a test method that accepts a descriptor and a block of code, we can ensure that the tests stay organized and visible.

00:12:30.640 However, we run into difficulties with shared variables when using local variables, as they can lead to interdependencies that cause one test's modification to affect the outcome of another, ultimately violating essential testing principles.

00:13:28.020 To resolve this, we can encapsulate each test in its own method, thus independent variables can be created within each scope without affecting others. Ruby provides a simple way to do this, establishing three methods, each with their own localized variables that won’t interfere with each other.

00:14:32.440 Methods don't add any overhead costs; instead, they reduce hidden dependencies and complexities in your tests. Once we have these separate tests wrapped in methods, we should also ensure that they are executed within their own context when called.

00:15:22.700 Next, wrapping methods within classes allows us to organize our tests even better. However, we must address how to invoke these test methods after wrapping them in a class structure. We can take the responsibility for running each test method out of the user’s hands and instead allocate that to the class instance itself.

00:16:21.780 This means, while we instantiate a class for our tests, we can also build an instance method that adds responsibility for running individual tests to the class instance for easier management.

00:17:18.330 It's efficient to have the class itself manage its tests rather than relying on external externalization. Therefore, we can utilize instance methods to handle the execution of test methods systematically, ensuring that they run themselves.

00:18:11.850 Further simplification of running tests can be accomplished by using public instance methods that contain the tests and filtering them accordingly, thus creating a method in our class that can dynamically collect all public test methods.

00:19:00.450 In order to ensure that our tests maintain DRY (Don't Repeat Yourself) principles, we should consider subclassing for shared test behaviors. This leads us to create a central test class to inherit from, thus allowing all our test classes to benefit from code reuse.

00:19:50.000 By subclassing a common ancestor, we also streamline our organization, ensuring that each test suite is easily accessible and runnable. Once we establish a common test class, managing the execution of tests through the class itself enhances usability.

00:20:45.460 Additionally, we’ll automate the execution of all tests, using Ruby's built-in class-inherited hook to ensure that new subclasses automatically register their tests. This creates a dynamic workflow of initiating these tests.

00:21:32.120 At this stage, while we can initiate tests quite effectively, we could enhance the framework to report outcomes effectively as well; the goal now is to give feedback on what the results of running tests were.

00:22:28.040 To improve the reporting process, we can print a dot for each test run, giving immediate feedback on how many tests have executed, and this enhances user understanding of the test progress.

00:23:20.060 Now, when tests fail, they should not halt the entire process nor lead to convoluted backtraces. Instead, we will allow all tests to run. Additionally, we will implement a mechanism for cleaner output while catching exceptions.

00:24:15.360 We will refactor the reporting system to better handle output while separating test logic from output handling, which will lead to cleaner and more maintainable code.

00:25:08.070 To do this, we can extract reporting logic into its own class while preserving functionality within the test classes. This will cleanly segregate responsibilities for better maintainability and understanding.

00:25:55.830 In simplifying our output, we can keep track of any failures without mixing them with successful test reporting, creating a clearer summary in the end for better final results.

00:26:43.810 Once we've streamlined the structure of our reporting and failure tracking processes, we can take additional steps to further clarify the output by renaming key functions to better reflect their new responsibilities.

00:27:39.060 Incorporating randomization into our test execution can assist in identifying tests that may depend on previous test states, thus ensuring that our tests remain independent and continuously check for hidden dependencies.

00:28:39.300 With all those pieces in place, we end up with a robust yet straightforward testing framework that builds upon simplicity and principled design.

00:29:27.830 While we've accomplished a lot in terms of functionality and usability, the final array of features is essentially a foundation that can be built upon, leading to further enhancements down the line.

00:30:08.850 With that said, the project of micro test offers a swift, clean, and effective alternative to broader testing frameworks, ushering in not only speed but also maintainability.

00:30:52.680 As we conclude, I want to share that there's always room for improvements and adaptations. The feedback cycle, the changes, and performance enhancements I touched on today may lead to even more refined approaches in future iterations.

00:31:47.780 As I move to wrap up my talk, keep this in mind: the journey into testing frameworks— even simple ones— begins with understanding core principles of testing, which allows for informed decisions on how best to move forward.

00:32:35.590 Thank you again for attending. I’m also looking forward to feedback on my book draft that's forthcoming and I'm optimistic about the response to come. Please follow me on Twitter for announcements, and don't hesitate to reach out with further questions.

00:34:02.760 I am now available for any remaining questions you may have.

00:34:18.620 To address the question on whether whiskey is required for using MiniTest, the answer is a definitive no; I only started drinking recently.

00:34:32.630 I would encourage using micro tests in team collaborations; understanding core frameworks demystifies tools.

00:34:42.070 Comparing MicroTest to MiniTest, both share significant functionality, but MicroTest is clearer and simpler in execution.

00:35:09.660 Moving ahead, I plan that MiniTest can adapt features to make it more efficient. Adding an abstract reporter and refining plugins could be beneficial.

00:35:53.360 Ultimately, becoming acquainted with which additional features provide context-specific enhancements and user-friendly adjustments will be crucial as time moves forward.

00:36:27.950 Overall, I welcome any additional questions and I'm eager to assist wherever possible.

00:36:51.510 Thank you once more for your time, and I hope you learned something valuable today.

Ruby on Ales 2016