00:00:10.520
We're not quite starting yet; we have a minute before the timer officially kicks in.
00:00:16.260
Before I start, I just want to say thank you to everyone for coming to this talk instead of going to watch Aaron Patterson talk about garbage collection and his cats.
00:00:30.650
That's absolutely 100% where I would be right now. As an RSpec maintainer, knowing how the garbage collector relocates pages is actually super important to my day-to-day work.
00:00:37.440
However, I suspect most people in this room are actually application developers, and being able to effectively test your code is perhaps more important than knowing about all the shiny new features under the hood of the Ruby VM.
00:00:50.760
I hope that resonates with you, or maybe you just like writing specs or care about open source.
00:01:03.300
So, I guess they’ve closed the doors, which means I should probably start. This is 'RSpec No Longer Works With ActiveRecord.' My name is Sam Phippen.
00:01:11.580
Just to be absolutely clear, this is not a feature announcement. Sometimes, when you wake up in the morning, working through your email with that first cup of coffee, a line comes in with a subject that fills your heart with dread.
00:01:24.240
For me, that was this RSpec Mocks issue number 972: 'RSpec No Longer Works with ActiveRecord Classes.' While the title of this talk is slightly hyperbolic—it's not that RSpec was completely broken with ActiveRecord—it is nonetheless one of the most core and important functionalities of the RSpec gem.
00:01:50.070
When RSpec doesn't work with ActiveRecord classes, it leads to a lot of unhappy maintainers and users. There's much gnashing of teeth, things break, and no one is happy.
00:02:20.629
An equivalent expression would be that when RSpec doesn’t work with Rails, my email inbox suddenly explodes. I have a poignant example in mind, which is a different RSpec issue: 'undefined method cache in tests' for Rails 4.2.5.1.
00:02:35.250
That particular issue had 37 comments come in overnight while I was sleeping. You see, this version of Rails got released late at night Pacific Time while I was still in the UK, meaning I was in bed, minding my own business. I woke up to discover that all of your tests were broken.
00:03:07.349
Everyone in this room who did the security upgrade had non-functional tests. I had an inbox full of sad emails, people sending pull requests, issues, and debugging inquiries trying to work out what was going on. I was just like, 'Everyone, please calm down. It will be fine.'
00:03:20.010
It may sound like I'm complaining, but I'm really not. In fact, I'm glad people care that RSpec works. I think it's a sign of an incredibly mature open-source project that people care enough about how the RSpec gem operates that they file issues the moment there’s even the slightest tremor of a problem.
00:03:34.019
This is a GIF of me at the end of the day after work most days. You have to remember that open-source projects are mostly worked on during evenings and weekends. Maintainers must divide their attention very carefully among their projects, or everything will be left to waste.
00:04:12.870
So, let's get back to our issue and focus on what we're here to discuss. The Rails issue I mentioned previously affected expressions like 'allow user to receive' followed by some kind of scope method. In this particular case, we were using 'confirmed,' but it could really be anything.
00:04:24.000
This simple test would raise an exception, and when I look at that, I face a moment of screaming fear because this has to work. ActiveRecord objects are one of the core things that people want to mock on, making this situation particularly urgent.
00:04:51.489
When I saw this issue, I thought to myself, 'Oh, this is not going to be fun.' Before we dive into debugging, I wanted to examine the bug report more closely to think about its quality, what was good, what was lacking, and how it might be improved.
00:05:15.849
If you look at the full text of the issue as submitted, it mentions that updating a Rails 4.2.1 project from RSpec 3.2.0 to 3.3.0 resulted in many failing specs. The user noted that the issue could be reproduced simply, and then they provided this test we just examined.
00:05:44.049
There are a few really great aspects of this bug report. For instance, the user provided both the Rails version and the RSpec version, which helps me hone in quickly on what to investigate. If you're filing an issue on any project, these details are incredibly useful.
00:06:06.809
However, there were some downsides to the report. We did not have much in the way of steps to reproduce the issue; we only had a single failing test. We lacked the production code, the Rails configuration, and other context that might help understand the issue better.
00:06:30.009
We also didn’t receive a Ruby version. In modern times, if you're using Ruby, variance between Ruby versions is usually not significant, but providing that information can be incredibly helpful, especially if someone is using an alternative interpreter like JRuby or if something particularly odd is occurring.
00:06:54.409
For this specific issue, it may not have been critical, but supplying the Ruby version would have been a helpful addition. Furthermore, we were missing a backtrace. I don’t know what the exception the user is encountering is, or where in the spec the exception is occurring.
00:07:12.790
As a maintainer, this means I have to dig into the issue and investigate thoroughly before I can figure out where the bug is coming from. While I appreciate having the Rails and RSpec versions, we also need extended dependency information, like which other gems are in use.
00:07:43.200
To improve the issue, I think it would be beneficial to provide a complete application reproduction case that I could clone and execute. Rather than just filing an issue with a single spec, I'd prefer an issue that includes a link to a Git repository containing everything needed for debugging.
00:08:15.130
What's great about RSpec is that it's a multi-maintainer project. So, what happened shortly after this user filed the issue is that another maintainer, Myron, stepped in and asked for a complete backtrace to help with debugging.
00:08:47.260
Following this request, another user—distinct from the one who initially reported the issue—arrived and provided the extra information. This collaborative experience is what open source is all about. Multiple people can experience the same bug, collaborate, and work on a solution because our tools are open.
00:09:35.930
If you are using an open-source framework and encounter an issue, you can visit the issue tracker to find that it is at the top of the list and take part in the collaboration.
00:09:47.880
Providing more information—beyond what the initial reporter gave—is always encouraged. No maintainer is ever going to reprimand you for providing extra data points; it can only help.
00:10:00.260
However, we still didn’t have quite enough information to begin debugging. We need a reproducible issue, something automated and easy to use, to debug any real problem in the open-source world today.
00:10:35.110
Fortunately, another RSpec maintainer stepped in and provided exactly that. They listed complete steps to reproduce, such as creating a new Rails application, specifying what to put in the Gemfile, generating models, and how to write a test.
00:11:12.150
So let's actually debug this issue together live on stage and work out what's going on. To begin, I will create a Rails application with version 4.2. The reason for that is that this issue was filed against that exact version.
00:11:46.400
Once the application is created, we can prepare to debug. The first step is to add RSpec to our Rails application so we can test the bug. Here, I have specified path dependencies to the RSpec version instead of the official gem versions.
00:12:06.530
I’ve done this because I have all of the RSpec repositories checked out locally on my computer for debugging purposes. This makes switching versions very easy. I can check out a specific git tag and get the exact version needed.
00:12:44.230
I’m also specifying rake version 10 here because the current version is rake 12, and earlier versions of RSpec are not compatible with rake 12. Now that we've updated our Gemfile, let’s proceed with 'bundle update' and 'bundle install' to get everything ready.
00:13:23.800
Next, we'll do the standard Rails generator to install RSpec. This may take some time as Bundler can be slow. Then we can proceed to generate a User model, which is the first step toward reproducing our bug.
00:13:52.100
We are experiencing some video troubles, as I had to rebuild these videos half an hour ago because they were a bit blurry. Please bear with me as I resolve that.
00:14:04.810
Once we've generated the User model, we must migrate our database; otherwise, we won’t be able to run the tests. Since we ran 'rspec install' in our Rails application, it automatically generated a spec file for the User model.
00:14:15.460
We can now open it and replace the default test with one that reproduces our bug. Since we haven’t created a scope method on this user object, I'm going to stub the 'new' method instead of stubbing a specific scope method.
00:14:36.920
This should allow us to reproduce the bug exactly as expected. If we run the tests now, we should see that this test passes, which is because we've checked out RSpec version 3.2.0 and want to ensure it fails on the subsequent version.
00:15:02.330
Now, I'll navigate to my RSpec development tooling repository, which is a meta repository used by RSpec maintainers to make working with different versions easier. Here, I will run a command to check out version 3.3.0.
00:15:30.180
Then I will switch back to my Rails application and run my tests. We can see that they are now failing, so we confirm that somewhere between versions 3.2.0 and 3.3.0, the tests broke. We have a good commit range to work with now.
00:15:57.330
I think having a passing and failing test is usually sufficient for most people to start debugging. This issue is likely the result of a small change that wouldn’t be too difficult to investigate.
00:16:13.440
However, we can do better. The year is 2017, and we have great tools for debugging regressions. If something used to work and now it doesn’t, one tool stands out as particularly beneficial: 'git bisect.'
00:16:49.350
For those unfamiliar, git bisect is a command that allows you to tell git where something was functioning correctly and where it is not. Git will then move you through the commit range in halves, helping you identify where the problem first arose.
00:17:15.650
The important commands you need to know are 'git bisect start', which initiates the process, 'git bisect good', which labels a particular commit as functional, and 'git bisect bad', which marks a commit as non-functional.
00:17:49.080
By using these commands, git will guide you through a series of commits interactively, helping you to identify the correct commit that introduced the issue. Unfortunately, our situation is complicated because there are potentially two repositories involved, RSpec and RSpec Mocks.
00:18:16.330
For those not familiar with RSpec’s internal workings, RSpec is broken down into multiple repositories that allow us to ship features independently, providing flexibility in our approach.
00:18:59.180
I generally assume that bugs exist in RSpec Rails since I lead that repository, so I will begin my bisecting process there to see what's happening.
00:19:21.560
Here I've moved into the RSpec Rails repository, initiated the bisect process, and labeled the version 3.3.0 as bad while checking out 3.2.0 as good. Now, we begin to debug.
00:19:53.000
Git has moved us to a specific revision, and back at our Rails application, we will type 'bundle exec rspec' to execute the test suite.
00:20:02.000
The reason we may see an explosion at this point is that Bundler cannot resolve 'rspec-core' version 3.3.0 with RSpec Rails version 3.3.0-pre.
00:20:41.000
Let me explain what's happening here. All RSpec gems are released at the same version numbers for the sake of compatibility. RSpec provides a stable public API, but our internal APIs might not follow the same rules.
00:21:00.080
Let's say we have RSpec Rails version 3.2.0, I must also have all other RSpec gems at 3.2.x. Similarly, all gems need to be at the same version level for '3.3.0', but what's happened is that the RSpec Rails version tag has changed to 3.3.0-pre.
00:21:24.890
This version tag does not correspond to a single RSpec commit; it indicates the version number in the master branch during the development of the next minor version.
00:21:49.680
Earlier, we checked out all other RSpec dependencies to version 3.3.0, but '3.3.0-pre' is not equal to '3.3.0', which is causing Bundler to fail.
00:22:05.370
To resolve this issue, we can use the command 'git checkout HEAD^', which instructs git to revert to the revision just before the one currently checked out.
00:22:35.630
I will execute this process for all my RSpec repositories and make sure each one is set to the appropriate version; then I can return to my Rails application.
00:23:07.920
From there, I run my tests, and we finally get to an operational state where the tests are running again, and we can confirm if they are failing.
00:23:20.110
So, let’s get through this process by hitting return on test runs until we arrive back at the failing commit. Each time we run our tests and confirm they fail, we mark that revision as bad.
00:23:56.300
Eventually, we work our way through until git provides the specific revision where the tests first started failing. This indicates where a change may have broken functionality.
00:24:40.420
Upon examining the specific revision, we find that what changed could be attributed to a patch introduced that altered RSpec's handling of method calls and expectations.
00:25:11.550
After searching through the method definitions on GitHub, we can identify the problematic code, which is designed to check if ActiveRecord subclasses exist. However, the check was mistakenly applicable to the ActiveRecord base itself.
00:25:40.600
This oversight resulted in a spec explosion when the method was invoked. Therefore, we will implement a check to ensure this code does not run on the ActiveRecord base.
00:26:06.800
Furthermore, since RSpec is a testing framework, we will add appropriate tests to verify that this behavior is corrected.
00:26:42.030
As an aftermath, while this solution works for ActiveRecord, it’s also pertinent to consider more generic fixes. If you can, you should aim for generic solutions that extend beyond the immediate problem.
00:27:09.890
In capturing the essence of this debugging experience, I want to summarize the steps we took: we identified the problem, implemented a fix, and documented the entire process.
00:27:43.080
An essential part of this experience is advocating for comprehensive bug reporting. Basic reports need improvement; maintaining a continuous flow of information strengthens the open-source ecosystem.
00:28:12.480
So, what can you do to elevate your bug reports? Start by providing clear, concise information to the maintainers, detailing the bug, the relevant code, and supplementary data.
00:28:45.030
Providing a backtrace filled with helpful context elevates the quality of your report significantly. It can tell maintainers specific versions and additional context leading to the failure.
00:29:20.070
Dependency information is also critical; knowing every gem and tool in use when the bug occurred helps maintainers quickly identify the underlying issue. If a gem is monkey-patching RSpec, it may not be RSpec’s fault at all.
00:29:55.780
The ideal addition to your bug report would be a reproduction case that I could clone immediately and test. Many maintainers won’t look at your report unless you provide this kind of context because it saves time in debugging.
00:30:30.490
Finally, having already conducted a bisect test before filing your report can revolutionize your contribution to the community. If you can pinpoint the commit where the breakage occurred, you can save the maintainers considerable time.
00:31:05.180
In conclusion, the more work you do beforehand, the simpler it becomes for maintainers like myself to address and fix the issues at hand.
00:31:33.570
As a parting note, I work at DigitalOcean, and we are hiring. I want to build an amazing team filled with Ruby developers who can help ship great software. If any of you are interested, feel free to approach me after the talk to discuss.
