Test Driven Development: A Love Story

Talks

Nell Shamrell

@nellshamrell

#test-driven-development

#legacy-code

#refactoring

#ruby

#developer-experience

#software-craftsmanship

#code-quality

Test Driven Development: A Love Story

by Nell Shamrell

The video titled "Test Driven Development: A Love Story" presented by Nell Shamrell at the Ancient City Ruby 2013 event explores the concept of Test Driven Development (TDD) through the metaphor of love. Shamrell shares her personal journey of grappling with legacy code and her transition to embracing TDD, which parallels the emotional stages of grief. It reflects on the challenges she faced, her frustrations with legacy systems, and ultimately how she learned to appreciate and leverage TDD in her coding practices.

Key points discussed in the video include:

Initial Experience: Shamrell describes her blissful start in a new coding job and her sudden responsibility for a troublesome legacy codebase, leading to stress and panic.
Five Stages of Grief: She outlines her progression through denial, anger, bargaining, depression, and finally acceptance. Each stage represents a different emotional response to handling the chaotic code before her.
The Role of TDD: TDD emerged as a critical tool for improving her coding approach and reducing her anxiety around changes in the legacy code. She emphasizes that the ideal time to write tests is always the present, rather than waiting for a better time.
Learning Through Testing: Shamrell discusses how writing tests helped her better understand the legacy code and improve it. She advocates for starting with testing new code and gradually rewriting legacy code to be testable.
Workflows and Best Practices: She details her evolving workflow, transitioning from manual testing to a more structured approach involving writing failing tests first (red-green-refactor).
Community and Continued Learning: Shamrell highlights the importance of seeking help from the developer community when faced with complex testing challenges. She underscores that TDD evolves and encourages continual learning.
Final Reflection: The closing thoughts center on the lasting benefits of TDD, such as fostering better code quality and documentation, alleviating the fear of making changes, and cultivating a more enjoyable coding practice.

In conclusion, Shamrell's journey illustrates that while TDD can be challenging, it ultimately enhances a developer’s ability to manage legacy code effectively. TDD transforms coding from a source of panic into an opportunity for longevity and craftsmanship in software development, creating a healthier relationship with the code we write and maintain.

00:00:00.000 A couple of years ago, I started my dream job. I would be doing what I love: coding and building software. What's even better? I would be doing it in Ruby.

00:00:07.140 The first six weeks of this job were absolute bliss. I was working on a relatively new codebase, a codebase that had some rough patches but, for the most part, was very maintainable and understandable.

00:00:19.680 During this time of bliss, I did hear whispers of another codebase—a codebase so terrifying that even the bravest developers feared to tread there. This was 40,000 lines of gnarly, nasty, largely untested legacy code.

00:00:37.530 I figured I would be introduced to this codebase in due time, piece by piece. However, as always, the unexpected occurred. The only other full-time developer at the company left, and suddenly this legacy codebase was my responsibility.

00:00:58.190 My days of bliss turned into days of receiving frantic bug reports, panicking, writing fixes as fast as I could, deploying those fixes, and breaking more things, leading to more panic. This cycle repeated over and over.

00:01:15.570 Test-driven development wasn't part of the equation at this point. I felt it was taking all my time and energy just to keep the system above water, and I didn't see how I could incorporate tests into that.

00:01:33.030 As scared as I was of the bugs that I did see, what scared me the most were the ones I knew I didn't see—the silent bugs lurking in the depths of this legacy code.

00:01:54.270 During this time, I once woke up screaming. I dreamed I had been working on this legacy codebase on my laptop when all of a sudden, the code turned into a mutant bat and flew directly into my face. It scared the heck out of my fiancé.

00:02:08.250 This code was invading my conscious mind, beyond that, it was seeping into my unconscious mind as well. It felt dark, hopeless, and like an endless night that I would never escape.

00:02:29.000 But dawn did break. I don't wake up screaming anymore. I don't dread opening my inbox in the morning, sifting through a deluge of errors and exceptions that came in overnight.

00:02:44.840 I used to feel like I was just applying band-aid fix after band-aid fix to the legacy codebase. However, I actually build things again. I do what I love again.

00:03:06.230 I want to tell you a story—a story that starts with love, a love for code, and a love for building software. This love was severely tested when I inherited this legacy codebase.

00:03:20.959 I actually passed through the five stages of grief: denial, anger, bargaining, depression, and finally acceptance. I also want to share how this grief turned into love again—a love that is more passionate and stronger than it has ever been before.

00:03:44.930 Test-driven development was the key to learning to manage my grief, moving through it, and learning to manage this legacy code system. It didn't happen overnight, and it certainly wasn't easy.

00:04:05.419 Even now, after working as a test-driven developer for some time, it's still not all unicorns and rainbows. I still have to work at it.

00:04:11.209 There are still rough patches, and there is change, but the difference is that now those changes are not insurmountable. So, who here has had to deal with legacy code? Just about every hand went up.

00:04:28.030 Does that code ever scare you? Does it haunt your dreams?

00:04:37.040 I want to share my story to help you move beyond the grief and anger associated with legacy code and rediscover the joy in coding. You can tame a legacy system.

00:04:53.500 I'm here to share how, after inheriting this legacy code, I found myself at the first stage of grief: denial. This legacy codebase had some tests, but the vast majority of it was completely uncovered.

00:05:19.030 I was relatively new to test-driven development when I started this job, and I had only tried it on greenfield applications. I thought that it wasn't the right time to try adding it to an existing application.

00:05:34.610 Faced with errors and exceptions left and right, I felt I had to stabilize the codebase first and then worry about tests. To do this, I resolved to stick to manual testing for the time being.

00:05:54.440 I would code a fix, test it manually, deploy it, and if that fix broke things, I would code another fix, deploy it again, and if that broke more things, I would code another fix and deploy it yet again.

00:06:06.020 The problem with this approach was that I had no way of knowing if a change in one class would affect other classes. With every deploy, I was flying blind.

00:06:39.720 I knew I needed to add my tests when the crisis passed, when I had a moment to breathe. I convinced myself I just needed to hold out a little longer.

00:06:51.270 Surely things would get easier soon. Surely, then I would have time to add tests. The truth, though, is that a magical better time in the future never comes.

00:07:04.350 Every untested line of code I committed made the system worse and spiked my stress level even higher. Kent Beck writes in 'Test-Driven Development: By Example' that the more stress you feel, the less testing you will do.

00:07:19.160 The less testing you do, the more errors you will make. The ideal time to write tests is not some magical time in the future but always now.

00:07:34.610 I could only stay in denial for so long. I knew things were falling apart. I even knew that adding automated tests would make things better but I didn't know where to start.

00:07:52.520 Every bug fix or new feature I added depended heavily on untested legacy code; writing tests seemed impossible.

00:08:05.240 How could I write tests for someone else's code—even someone who had left the company years ago? How could the code have gotten to this state?

00:08:20.360 I honestly started to get angry. Now, has anyone else looked at legacy code and felt angry? Did you think to yourself, 'What was this developer thinking?' I would go in circles in my mind, ranting and raving about the sins of past developers.

00:08:37.740 'Why did I have to clean up their mess? Why was it my responsibility?' Fortunately, someone pointed out to me that the truth is, the past developers were probably just as panicked as I was.

00:08:52.080 Does anyone really write bad code just for the sake of writing bad code? Does anyone write untested code just for the fun of it? I could rant and rave about the past all I wanted, but it wouldn't accomplish anything.

00:09:13.300 The truth is, you simply can't change the past. I couldn't go back in time and write the legacy code better or prevent it from being written the way it was. I needed to move forward.

00:09:29.600 It turned out the thing that needed to change the most wasn't the code; the thing that needed to change most was me. I needed to take ownership of this code—as ugly as it was.

00:09:44.000 I could curse the past day in and day out, but the only thing I could do was affect the future. The only thing I could do was make it better from here on out.

00:09:58.270 Now, it is possible to add tests to legacy code, but it does take time. The problem with code that was written without tests in mind is that often it's untestable.

00:10:14.420 To be testable, it needs to be refactored, and refactoring a working codebase—particularly one that doesn't have test coverage to begin with—is always risky.

00:10:34.790 Noel Rappin writes in 'Rails, Test Prescription' that you should take small steps that can be verified. I couldn't add test coverage to this 40,000 lines of code all at once; that wouldn't be practical from either a technological or a business standpoint.

00:11:02.910 But there were certain steps that I could take. Any new code must have tests. When I read 'Rails: Test Prescriptions,' I found a good piece of advice: separate my new testing code, put it in new tested classes and methods, and call those tested classes and methods from within my legacy mess.

00:11:29.040 This is actually the first step in refactoring that legacy code: separating things out. Bugs in code must be reproduced with a test.

00:11:43.520 Now the way I found to do this, because that is honestly a lot easier said than done, was to use the bug report. When a user submits a bug report, if they're nice, they'll include steps to reproduce that bug.

00:12:06.780 I used the steps that the user used from the graphical user interface to figure out how to reproduce that bug on a code level. For example, if a user gave me a bug report stating that when they visited a form—let's say it's a form for a new record in the database—they would leave a certain field blank and then click submit, the application would crash completely.

00:12:30.060 To do this in a test, I created a new object. Chances are, a form is probably rendered by a controller. One of the things that a controller action does is create a new object in the database.

00:12:53.690 I would then leave a field on that new object blank, save that object, and when I did this in a test, I found that the test returned an exception.

00:13:12.120 This begged the question: what should my code do? Should it crash completely when a user leaves a required field blank, or should it return a useful error to that user?

00:13:31.050 I tend to lean toward the useful error. In order to do this in a test, I created a new object, assigned it to a variable, set a required field on that object to nil, saved that object, and then specified what I wanted my code to do.

00:13:46.880 I wanted that new object to return an error saying that a certain required field was blank. I then wrote the code to make this test pass. You can use tests to learn about legacy code.

00:14:11.680 My tests let me interact with my legacy code directly on a code level. Separate documentation might lie, comments in the code might lie, but exercising the code itself doesn't lie.

00:14:30.949 With every new test I added, I learned more about this legacy codebase. When you're adding tests to existing code, it's okay if those tests have an extensive setup. The point is to learn about the system.

00:14:51.250 The more I learn about this legacy system, the more I see how I could change it for the better. Once I realized I could add tests to my legacy code and use these tests to improve the system, my anger dissipated.

00:15:15.080 However, I wasn't done with my grief yet. I moved on to bargaining. I embraced the need for tests but I was still in the middle of a crisis.

00:15:28.750 Bugs and exceptions were still coming in left and right—urgent bodies that needed fixing. I decided to fix the problem first, write my code, and then write my tests.

00:15:40.400 After all, how could I write tests for code when I didn't know what the code was supposed to do in the first place? I envisioned my workflow as a simple two-step process: write my code, then write my tests.

00:16:03.650 In actuality, however, this process had many more steps. I would write my code, manually test that code, modify the code, manually test the modifications again, write an automated test, and realize I had to modify my code to make it testable.

00:16:32.880 Then, I'd manually test the modifications again and modify my tests to work. This actually wasted a lot of time. When you test last, it inevitably leaves holes in your test coverage.

00:16:49.260 When I did get to writing tests, I wrote them fully expecting them to pass. When they passed, I deployed. However, sometimes after the deploy, the code would break but the tests would still pass.

00:17:06.370 This was a red flag that something was very wrong in my approach to testing. I found a workflow that works a lot better: I'd write a failing test, make the test pass, and then refactor the code.

00:17:21.809 Chances are, this looks very familiar to all of you. This is also known as red-green-refactor—the failing test, making the test pass, and then refactoring. In 'Growing Object-Oriented Software, Guided by Tests,' Steve Freeman and Nate Price lay out the golden rule of test-driven development: never write new functionality without first writing a failing test.

00:17:49.800 It's called test-driven development for a reason. When I test first, I'm forced to clarify my intentions to find exactly what I want my code to do.

00:18:05.570 Then, after I have a failing test in place, I write code that meets my intention and does only that. Robert C. Martin, in his 'Clean Coder' screencast series, states that testing is about trust.

00:18:24.480 Trust that when you write code and all your tests pass, and then you deploy that code, it won't break anything in your system. The only way to have full 100% trust is to have every line of your production code be tested.

00:18:44.920 The only way I found to come even remotely close to this number is to write my tests first. When I write my tests first and then make that test pass, I know my code is covered by tests; it wouldn't have passed the test otherwise.

00:19:04.490 It was actually disheartening when I realized that my attempts at bargaining and my intention of doing things on my terms had failed. I moved on to the fourth stage of grief: depression.

00:19:23.670 I felt hopeless. Sure, I could write tests first from now on, but so much of the legacy code— including code that I myself had written—was still so bad.

00:19:40.090 What was worse was that even the tests I had written or that other developers had written in the past had some unreliable tests. There were gaping holes in their coverage because those tests had been written last.

00:19:55.680 I found myself asking, 'What's the point? What's the point of adding new tests when so much of the existing code and tests are still terrible?' I honestly felt like giving up. I felt like writing off this legacy codebase as a lost cause that I unfortunately still had to use.

00:20:21.300 Now, it sometimes seems easier to do this—just to give up and resign yourself to living with the legacy mess. But the truth is, it's not.

00:20:39.330 Production code doesn't stay in stasis. You will have to add things and you will have to change things to a system that's still in use. When faced with a legacy mess, you have two choices.

00:20:53.990 One: you can add to the legacy mess, let the code get worse, and let it rot and collapse in on itself. Or two: you can move forward—you can draw a line in the sand and make it better from now on.

00:21:09.090 Every line of tested code is a reliable piece of code. Every reliable piece of code is a gift to yourself, to your teammates, to your users.

00:21:29.430 At this point in the story, I was not the only developer at the company anymore. There were several other people working to tame this legacy mess.

00:21:47.510 Every piece of reliable code was a step out of the mess. Every piece of tested reliable code was a beacon of hope in the dark snarl of legacy code.

00:22:06.280 By contrast, every line of untested code is an unreliable piece of code. Every piece of unreliable code only makes things worse—for yourself, for your teammates, and for your users.

00:22:28.240 The legacy code may have been bad then, but if I kept on committing unreliable untested code, it would only get worse. Like it or not, I now own this code.

00:22:52.900 I was steering its direction. As painful as it was, and as tempting as it was to give up, it wasn't hopeless. It's never hopeless to start doing the right thing.

00:23:07.050 Now, the right thing to do to save my sanity—and the sanity of my teammates and my users—was to test-drive my code from now on. It was to move forward.

00:23:29.630 At this point, I found myself at the final stage of grief: acceptance. I finally came to terms with my situation. I knew I would survive this legacy codebase and that test-driven development was vital.

00:23:58.790 It was time to leave the past exactly where it belongs: in the past. It was time to move on to a brighter future.

00:24:12.160 This is at this point that I realized the true benefits of test-driven development. One of the most obvious benefits is that tests prevent breaking production code.

00:24:32.840 Production code doesn't live in a vacuum, as I said before. You will have to add things and you will have to change things. Change is inevitable.

00:24:54.350 What we can control is how painful these changes need to be. The most painful changes for me are when I make a change and it breaks something completely unrelated in the code.

00:25:08.634 Has anyone else had that experience? I see a lot of heads nodding. When I have a full test suite—especially one written with tests first—it prevents this from happening.

00:25:33.550 Now, even when I run that test suite and a test fails unexpectedly, if I'm on that test suite, chances are I'm going to know exactly what change broke that test.

00:25:59.220 The best time to fix something is directly after you break it. This meant I could keep coding rather than constantly chasing down bugs.

00:26:20.840 Testing turns out didn't just keep my code from breaking; it kept me from breaking as well. Tests are documentation—they're living fully functional documentation that's entwined with your code.

00:26:43.680 Now, rather than keeping my documentation in a separate place, say a wiki or, God forbid, a Word document, my tests were embedded directly in my code.

00:27:01.370 I couldn't change one without changing the other. The documentation was embedded directly with my code. Some of the hardest cases to debug are when the original developer's intent is unclear.

00:27:17.360 How can I fix a piece of code when I have no idea what that code was supposed to do in the first place? Tests document the intent of the developer.

00:27:39.690 They state in no uncertain terms that when the code receives certain input, it should return a certain output or take a certain action.

00:28:00.300 For example, I have my first test stating that when I call a method called 'tax_rate' and pass in the argument 'Seattle,' it should return 0.095. The tax rate in Seattle is 9.5%.

00:28:21.960 I don't actually know what it is in August, but this tells me exactly what this method is expected to do, what input it should receive, and what output it should return with that input.

00:28:42.060 Next, let's say I want to call a controller action. I want to say that controller action assigns a variable called 'new_object' as a new record in my database.

00:29:01.950 It should not be an existing record; it should be a new record. I can also say I expect that a certain method, which saves this record to our database, will change the table count by one—not by two, not by zero.

00:29:18.910 It will add one record to that table. These tests state exactly what my intention is with my code.

00:29:33.680 Test-driven code is better code. Test-driven code is modular, loosely coupled, and has smaller methods. It's really hard to test something that doesn't have these attributes.

00:29:54.780 These are the hallmarks of good code. Testing doesn't just tell me to write clean code; it compels me to write clean code—code that is maintainable, more readable, and all around better crafted.

00:30:11.660 Finally, tests remove fear. When I have tests for my code, I can refactor that code fearlessly. I can try new ways of implementing the same functionality.

00:30:29.700 Maybe something I heard at a conference, something I read in the news, or something from a user group or a newsletter. I can constantly make my code better, and I can do this without fear.

00:30:46.790 Even with this gnarled, nasty legacy mess, I could still improve the code. I can move forward rather than staying stuck in the past. It will be a step by step, piece by piece process, but I knew this code would get better. I would make it better.

00:31:07.220 Now, I wish I could say that after I passed through these stages of grief, this legacy code and I rode off into the sunset and lived happily ever after in test-driven development bliss. But that wouldn't be reality.

00:31:40.590 I would be lying if I said that testing ever truly gets easy. It does get easier, and you will see the benefits very quickly, but much like love itself, it's something that you have to keep working on.

00:31:53.479 Now, about a year ago, I found myself completely baffled when it came time to test-drive code that interacted with an external API. I knew I couldn't connect to this external API every time I ran my tests.

00:32:05.090 This would be impractical at best and would get me banned from the API service at worst. I was convinced I had to figure this out myself; if I couldn't do it, surely it had to be impossible.

00:32:13.990 Fortunately, a colleague pointed out to me that I didn't have to go at it alone. When testing gets hard—and it will—it's okay to consult a teammate.

00:32:29.520 It's okay to do a search on Stack Overflow or a search on Google. Once I learned from the experiences of others, it turned out many others had already solved the problem of test-driving code that interacts with an external API.

00:32:41.920 What I expected my code to do was call a method on the API service, and that API service would return a certain response that my code would then handle.

00:32:57.000 It turns out that for testing purposes, it was actually irrelevant whether my code connected to the live API. The way to do this was by using mocks and stubs.

00:33:07.660 Now, mocks and stubs are hard. It took me a long time just to conceptually figure out what they actually do. A stub is a stand-in for an object called by your code.

00:33:24.440 It receives messages from your code while it's under test and returns scripted responses that you tell it to return. Under test conditions, my code would call an API method, and that call would be received by the API stub.

00:33:39.820 The stub would then return a scripted response that I told it to return. Now, mocks are specialized types of stubs. In 'Eloquent Ruby,' Russ Olsen—who just spoke before me—points out that mocks are stubs with attitude.

00:33:53.600 So, a mock acts a lot like a regular stub. My code will call an API method; that call will be received by the mock, and the mock will return a certain scripted response.

00:34:15.060 What's special about a mock, as opposed to a regular stub, is that a mock knows exactly which of its methods should be called and with what arguments. If my code calls an unexpected method, my mock will fail the test.

00:34:31.360 Mocks are much more specific than regular stubs; if their expectations are not met exactly, they will fail the test. The answers were out there; I just needed to reach out.

00:34:53.720 I wasn't just being my code all alone in the universe. There's an entire community, an entire world of developers all working on who maybe already resolved the same problems that I was working on.

00:35:07.480 Just as I need to constantly learn and adapt the way I code, I constantly need to learn and adapt the way I test.

00:35:14.720 Test-driven development is evolving. New test frameworks come and go, and new best practices are being uncovered in regards to testing every day.

00:35:32.240 I recently read 'Practical Object-Oriented Design in Ruby' by Sandi Metz, who also happens to be in attendance. The chapter on testing opened my eyes to how I can make my tests not only have better coverage, but also be smarter.

00:35:55.730 It introduced concepts that I never even conceived of, even after working as a test-driven developer for some time. I know that I will never stop learning about testing, just as I'll never stop learning about coding.

00:36:24.130 The way I code and tests will change over time, but even with these changes, the core benefits of testing—preventing breaking, acting as documentation, and compelling you to write better code—remain the same.

00:36:36.590 Now, my relationship with test-driven development is not perfect, but no relationship—whether between two people or between a person and their technology—is ever going to truly be perfect.

00:36:54.900 That's what makes it worth doing. That's what makes it rewarding. So, test-driven development changed the way I code.

00:37:13.050 It gave me new tools for dealing with a legacy mess, but even more than that, test-driven development changed me. It transformed me from a developer who panics in the face of legacy code to a craftsperson who cares not only that my code just works for the current moment.

00:37:37.620 A craftsperson who designs code to build things that last. A craftsperson doesn't just rant and rave about legacy code but takes action to make it better.

00:38:04.150 Test-driven development is developing beyond myself, beyond my immediate needs in the present moment. It's developing my code to handle change without pain.

00:38:23.950 It's providing a safety net and a guided path for any developer who might come after me. Now, change is inevitable; we hold the power to make that change something we can manage, maybe even enjoy, rather than something we dread.

00:38:39.780 Test-driven development empowers us to do this. I better understand why I do what I do now. I love to build things that last—things people can add to and change over time.

00:38:58.000 This love was severely tested when I inherited this legacy code. I had to pass through the five stages of grief, but I did rediscover that love, and that love is stronger than it ever has been before.

00:39:07.200 Coding is a major part of my life; it's one of the ways I'm leaving my mark on this world. Test-driven development helped me move beyond the fear and anger associated with legacy code.

00:39:17.000 I no longer code out of panic; I code out of love. Love, ultimately, is what life is all about.

00:39:27.000 I'm now ephemeral; I'm a software development engineer with Blue Box. Thank you very much!

00:39:39.000 You.

Ancient City Ruby 2013