"RSpec no longer works with ActiveRecord"

by Sam Phippen

In the talk titled 'RSpec No Longer Works With ActiveRecord,' presented by Sam Phippen at RubyConf 2017, the central theme revolves around a critical investigation into a significant bug affecting RSpec's compatibility with ActiveRecord. The session highlights the importance of robust testing in application development, especially for open-source projects like RSpec.

Key points discussed include:
- The initial concern arose from an RSpec Mocks issue indicating that RSpec was not functioning correctly with ActiveRecord classes, a crucial feature for many users.
- A detailed account was provided of a past issue where a Rails upgrade led to a wave of broken tests, demonstrating the real-world implications of bugs in widely used libraries.
- The importance of high-quality bug reports was emphasized, showcasing both exemplary submissions and missed details that could hinder debugging efforts.
- The talk described the step-by-step debugging process, showcasing tools like Git Bisect to efficiently track down the commit responsible for the bug.
- Throughout the investigation, community collaboration was highlighted as key to resolving issues in open-source projects.
- Finally, best practices for filing effective bug reports were discussed, advocating for context-rich submissions that include versions, backtraces, and reproduction steps.

Significant examples included a personal anecdote where a small change in RSpec led to severe implications for users, demonstrating the interconnected nature of libraries in software development. The conclusion urged developers to take supportive measures in reporting issues to ensure swift resolution, ultimately contributing to the health of the open-source ecosystem. Phippen also ended the talk by inviting attendees to connect with him regarding job opportunities at DigitalOcean, indicating his interest in recruiting proficient Ruby developers.

Overall, the talk serves both as a cautionary tale about the fragility of software dependencies and a guide on the best practices for collaborating in the open-source community.

00:00:10.520 We're not quite starting yet; we have a minute before the timer officially kicks in.

00:00:16.260 Before I start, I just want to say thank you to everyone for coming to this talk instead of going to watch Aaron Patterson talk about garbage collection and his cats.

00:00:30.650 That's absolutely 100% where I would be right now. As an RSpec maintainer, knowing how the garbage collector relocates pages is actually super important to my day-to-day work.

00:00:37.440 However, I suspect most people in this room are actually application developers, and being able to effectively test your code is perhaps more important than knowing about all the shiny new features under the hood of the Ruby VM.

00:00:50.760 I hope that resonates with you, or maybe you just like writing specs or care about open source.

00:01:03.300 So, I guess they’ve closed the doors, which means I should probably start. This is 'RSpec No Longer Works With ActiveRecord.' My name is Sam Phippen.

00:01:11.580 Just to be absolutely clear, this is not a feature announcement. Sometimes, when you wake up in the morning, working through your email with that first cup of coffee, a line comes in with a subject that fills your heart with dread.

00:01:24.240 For me, that was this RSpec Mocks issue number 972: 'RSpec No Longer Works with ActiveRecord Classes.' While the title of this talk is slightly hyperbolic—it's not that RSpec was completely broken with ActiveRecord—it is nonetheless one of the most core and important functionalities of the RSpec gem.

00:01:50.070 When RSpec doesn't work with ActiveRecord classes, it leads to a lot of unhappy maintainers and users. There's much gnashing of teeth, things break, and no one is happy.

00:02:20.629 An equivalent expression would be that when RSpec doesn’t work with Rails, my email inbox suddenly explodes. I have a poignant example in mind, which is a different RSpec issue: 'undefined method cache in tests' for Rails 4.2.5.1.

00:02:35.250 That particular issue had 37 comments come in overnight while I was sleeping. You see, this version of Rails got released late at night Pacific Time while I was still in the UK, meaning I was in bed, minding my own business. I woke up to discover that all of your tests were broken.

00:03:07.349 Everyone in this room who did the security upgrade had non-functional tests. I had an inbox full of sad emails, people sending pull requests, issues, and debugging inquiries trying to work out what was going on. I was just like, 'Everyone, please calm down. It will be fine.'

00:03:20.010 It may sound like I'm complaining, but I'm really not. In fact, I'm glad people care that RSpec works. I think it's a sign of an incredibly mature open-source project that people care enough about how the RSpec gem operates that they file issues the moment there’s even the slightest tremor of a problem.

00:03:34.019 This is a GIF of me at the end of the day after work most days. You have to remember that open-source projects are mostly worked on during evenings and weekends. Maintainers must divide their attention very carefully among their projects, or everything will be left to waste.

00:04:12.870 So, let's get back to our issue and focus on what we're here to discuss. The Rails issue I mentioned previously affected expressions like 'allow user to receive' followed by some kind of scope method. In this particular case, we were using 'confirmed,' but it could really be anything.

00:04:24.000 This simple test would raise an exception, and when I look at that, I face a moment of screaming fear because this has to work. ActiveRecord objects are one of the core things that people want to mock on, making this situation particularly urgent.

00:04:51.489 When I saw this issue, I thought to myself, 'Oh, this is not going to be fun.' Before we dive into debugging, I wanted to examine the bug report more closely to think about its quality, what was good, what was lacking, and how it might be improved.

00:05:15.849 If you look at the full text of the issue as submitted, it mentions that updating a Rails 4.2.1 project from RSpec 3.2.0 to 3.3.0 resulted in many failing specs. The user noted that the issue could be reproduced simply, and then they provided this test we just examined.

00:05:44.049 There are a few really great aspects of this bug report. For instance, the user provided both the Rails version and the RSpec version, which helps me hone in quickly on what to investigate. If you're filing an issue on any project, these details are incredibly useful.

00:06:06.809 However, there were some downsides to the report. We did not have much in the way of steps to reproduce the issue; we only had a single failing test. We lacked the production code, the Rails configuration, and other context that might help understand the issue better.

00:06:30.009 We also didn’t receive a Ruby version. In modern times, if you're using Ruby, variance between Ruby versions is usually not significant, but providing that information can be incredibly helpful, especially if someone is using an alternative interpreter like JRuby or if something particularly odd is occurring.

00:06:54.409 For this specific issue, it may not have been critical, but supplying the Ruby version would have been a helpful addition. Furthermore, we were missing a backtrace. I don’t know what the exception the user is encountering is, or where in the spec the exception is occurring.

00:07:12.790 As a maintainer, this means I have to dig into the issue and investigate thoroughly before I can figure out where the bug is coming from. While I appreciate having the Rails and RSpec versions, we also need extended dependency information, like which other gems are in use.

00:07:43.200 To improve the issue, I think it would be beneficial to provide a complete application reproduction case that I could clone and execute. Rather than just filing an issue with a single spec, I'd prefer an issue that includes a link to a Git repository containing everything needed for debugging.

00:08:15.130 What's great about RSpec is that it's a multi-maintainer project. So, what happened shortly after this user filed the issue is that another maintainer, Myron, stepped in and asked for a complete backtrace to help with debugging.

00:08:47.260 Following this request, another user—distinct from the one who initially reported the issue—arrived and provided the extra information. This collaborative experience is what open source is all about. Multiple people can experience the same bug, collaborate, and work on a solution because our tools are open.

00:09:35.930 If you are using an open-source framework and encounter an issue, you can visit the issue tracker to find that it is at the top of the list and take part in the collaboration.

00:09:47.880 Providing more information—beyond what the initial reporter gave—is always encouraged. No maintainer is ever going to reprimand you for providing extra data points; it can only help.

00:10:00.260 However, we still didn’t have quite enough information to begin debugging. We need a reproducible issue, something automated and easy to use, to debug any real problem in the open-source world today.

00:10:35.110 Fortunately, another RSpec maintainer stepped in and provided exactly that. They listed complete steps to reproduce, such as creating a new Rails application, specifying what to put in the Gemfile, generating models, and how to write a test.

00:11:12.150 So let's actually debug this issue together live on stage and work out what's going on. To begin, I will create a Rails application with version 4.2. The reason for that is that this issue was filed against that exact version.

00:11:46.400 Once the application is created, we can prepare to debug. The first step is to add RSpec to our Rails application so we can test the bug. Here, I have specified path dependencies to the RSpec version instead of the official gem versions.

00:12:06.530 I’ve done this because I have all of the RSpec repositories checked out locally on my computer for debugging purposes. This makes switching versions very easy. I can check out a specific git tag and get the exact version needed.

00:12:44.230 I’m also specifying rake version 10 here because the current version is rake 12, and earlier versions of RSpec are not compatible with rake 12. Now that we've updated our Gemfile, let’s proceed with 'bundle update' and 'bundle install' to get everything ready.

00:13:23.800 Next, we'll do the standard Rails generator to install RSpec. This may take some time as Bundler can be slow. Then we can proceed to generate a User model, which is the first step toward reproducing our bug.

00:13:52.100 We are experiencing some video troubles, as I had to rebuild these videos half an hour ago because they were a bit blurry. Please bear with me as I resolve that.

00:14:04.810 Once we've generated the User model, we must migrate our database; otherwise, we won’t be able to run the tests. Since we ran 'rspec install' in our Rails application, it automatically generated a spec file for the User model.

00:14:15.460 We can now open it and replace the default test with one that reproduces our bug. Since we haven’t created a scope method on this user object, I'm going to stub the 'new' method instead of stubbing a specific scope method.

00:14:36.920 This should allow us to reproduce the bug exactly as expected. If we run the tests now, we should see that this test passes, which is because we've checked out RSpec version 3.2.0 and want to ensure it fails on the subsequent version.

00:15:02.330 Now, I'll navigate to my RSpec development tooling repository, which is a meta repository used by RSpec maintainers to make working with different versions easier. Here, I will run a command to check out version 3.3.0.

00:15:30.180 Then I will switch back to my Rails application and run my tests. We can see that they are now failing, so we confirm that somewhere between versions 3.2.0 and 3.3.0, the tests broke. We have a good commit range to work with now.

00:15:57.330 I think having a passing and failing test is usually sufficient for most people to start debugging. This issue is likely the result of a small change that wouldn’t be too difficult to investigate.

00:16:13.440 However, we can do better. The year is 2017, and we have great tools for debugging regressions. If something used to work and now it doesn’t, one tool stands out as particularly beneficial: 'git bisect.'

00:16:49.350 For those unfamiliar, git bisect is a command that allows you to tell git where something was functioning correctly and where it is not. Git will then move you through the commit range in halves, helping you identify where the problem first arose.

00:17:15.650 The important commands you need to know are 'git bisect start', which initiates the process, 'git bisect good', which labels a particular commit as functional, and 'git bisect bad', which marks a commit as non-functional.

00:17:49.080 By using these commands, git will guide you through a series of commits interactively, helping you to identify the correct commit that introduced the issue. Unfortunately, our situation is complicated because there are potentially two repositories involved, RSpec and RSpec Mocks.

00:18:16.330 For those not familiar with RSpec’s internal workings, RSpec is broken down into multiple repositories that allow us to ship features independently, providing flexibility in our approach.

00:18:59.180 I generally assume that bugs exist in RSpec Rails since I lead that repository, so I will begin my bisecting process there to see what's happening.

00:19:21.560 Here I've moved into the RSpec Rails repository, initiated the bisect process, and labeled the version 3.3.0 as bad while checking out 3.2.0 as good. Now, we begin to debug.

00:19:53.000 Git has moved us to a specific revision, and back at our Rails application, we will type 'bundle exec rspec' to execute the test suite.

00:20:02.000 The reason we may see an explosion at this point is that Bundler cannot resolve 'rspec-core' version 3.3.0 with RSpec Rails version 3.3.0-pre.

00:20:41.000 Let me explain what's happening here. All RSpec gems are released at the same version numbers for the sake of compatibility. RSpec provides a stable public API, but our internal APIs might not follow the same rules.

00:21:00.080 Let's say we have RSpec Rails version 3.2.0, I must also have all other RSpec gems at 3.2.x. Similarly, all gems need to be at the same version level for '3.3.0', but what's happened is that the RSpec Rails version tag has changed to 3.3.0-pre.

00:21:24.890 This version tag does not correspond to a single RSpec commit; it indicates the version number in the master branch during the development of the next minor version.

00:21:49.680 Earlier, we checked out all other RSpec dependencies to version 3.3.0, but '3.3.0-pre' is not equal to '3.3.0', which is causing Bundler to fail.

00:22:05.370 To resolve this issue, we can use the command 'git checkout HEAD^', which instructs git to revert to the revision just before the one currently checked out.

00:22:35.630 I will execute this process for all my RSpec repositories and make sure each one is set to the appropriate version; then I can return to my Rails application.

00:23:07.920 From there, I run my tests, and we finally get to an operational state where the tests are running again, and we can confirm if they are failing.

00:23:20.110 So, let’s get through this process by hitting return on test runs until we arrive back at the failing commit. Each time we run our tests and confirm they fail, we mark that revision as bad.

00:23:56.300 Eventually, we work our way through until git provides the specific revision where the tests first started failing. This indicates where a change may have broken functionality.

00:24:40.420 Upon examining the specific revision, we find that what changed could be attributed to a patch introduced that altered RSpec's handling of method calls and expectations.

00:25:11.550 After searching through the method definitions on GitHub, we can identify the problematic code, which is designed to check if ActiveRecord subclasses exist. However, the check was mistakenly applicable to the ActiveRecord base itself.

00:25:40.600 This oversight resulted in a spec explosion when the method was invoked. Therefore, we will implement a check to ensure this code does not run on the ActiveRecord base.

00:26:06.800 Furthermore, since RSpec is a testing framework, we will add appropriate tests to verify that this behavior is corrected.

00:26:42.030 As an aftermath, while this solution works for ActiveRecord, it’s also pertinent to consider more generic fixes. If you can, you should aim for generic solutions that extend beyond the immediate problem.

00:27:09.890 In capturing the essence of this debugging experience, I want to summarize the steps we took: we identified the problem, implemented a fix, and documented the entire process.

00:27:43.080 An essential part of this experience is advocating for comprehensive bug reporting. Basic reports need improvement; maintaining a continuous flow of information strengthens the open-source ecosystem.

00:28:12.480 So, what can you do to elevate your bug reports? Start by providing clear, concise information to the maintainers, detailing the bug, the relevant code, and supplementary data.

00:28:45.030 Providing a backtrace filled with helpful context elevates the quality of your report significantly. It can tell maintainers specific versions and additional context leading to the failure.

00:29:20.070 Dependency information is also critical; knowing every gem and tool in use when the bug occurred helps maintainers quickly identify the underlying issue. If a gem is monkey-patching RSpec, it may not be RSpec’s fault at all.

00:29:55.780 The ideal addition to your bug report would be a reproduction case that I could clone immediately and test. Many maintainers won’t look at your report unless you provide this kind of context because it saves time in debugging.

00:30:30.490 Finally, having already conducted a bisect test before filing your report can revolutionize your contribution to the community. If you can pinpoint the commit where the breakage occurred, you can save the maintainers considerable time.

00:31:05.180 In conclusion, the more work you do beforehand, the simpler it becomes for maintainers like myself to address and fix the issues at hand.

00:31:33.570 As a parting note, I work at DigitalOcean, and we are hiring. I want to build an amazing team filled with Ruby developers who can help ship great software. If any of you are interested, feel free to approach me after the talk to discuss.