00:00:12.049
Hi everyone! Uh, there are a few of you way down at the back there, and this room is very big. You can come forward; it's fine. I won't bite, I promise! It'll be great. Alright, this is 'Teaching RSpec to Play Nice with Rails.' Let's get started! I could not be more excited and happy to be back here at RailsConf.
00:00:23.100
This stage is a lot bigger than what I thought I would be presenting on this year. Last year, I didn't make the conference. Some of you know the story, and some of you don't, but I was suffering from a life-threatening infection in my leg. In fact, exactly a year ago to this day, I took a picture of myself in the hospital and texted it to my mum. She had a bit of a freak out, but it was fine. That is a bag of a drug called Calf Triac Zone. Calf Triac Zone is a powerful broad-spectrum antibiotic, and were it not for the efforts of the doctors and nurses of the NHS in the UK, I would not be here today giving this presentation in a very literal sense. I'm hugely thankful to them and to the healthcare system in the UK that means that a year later, I am alive and able to give this talk on this stage. However, I'm not telling you this story to elicit sympathy towards me but instead to tell you about something slightly odd that happened at the RailsConf program last year.
00:01:30.330
We had this problem where Rails Camp last year was the conference where Rails 5 was due to be announced. That meant that there were a bunch of talks in the program about how to get various things working with Rails 5. It's a big major version, and stuff will break. That is bad! I was going to give the talk about how to get RSpec working with Rails 5, and there wasn't really a backup speaker that had a talk that looked like that. That was a fairly large gap in the commitment to the program. So, myself and the rest of the program committee decided that the correct thing to do would be to call up Justin Soles, who is giving the keynote tomorrow. I don't know if he's in the room; he's probably not. He's probably writing his keynote.
00:02:03.890
This is funny because the program committee rejected the talk Justin submitted outright, but we were like, 'Nah, that doesn't seem like a talk we'd like to have.' And then, here we are, Justin is morally obliged to give the RSpec talk because Sam is dying, and he took that with nothing but grace. If you've seen the talk, you know that he spent the back half of it basically trolling me. But in all seriousness, I've said this before, and this will be the last time I mention this on a Ruby Central stage: Justin, I am hugely thankful that you did that, even if you're not in the room right now. I think everyone should use the meme hashtag: 'Sam Phippen is Justin Soles confirmed.' Tweet him and let him know that you're all thankful!
00:02:56.040
Some of you will probably need RSpec to work with Rails 5.1, and that compatibility is coming, but with a couple of caveats. We do not yet have any support for ActionCable's test case nor ActionDispatch's belief system test case. Please have some faith in us. In all seriousness, Rails 5.1 has not been released yet, and that means we can't actually give out compatibility with it. But we have a branch ready that should just work, meaning that there will be no changes required by you. As soon as RSpec 3.6.0 is released, you should just be able to upgrade, and everything will work. We don't have integration with system tests because I haven't had enough spare time to do that yet, but I would absolutely welcome a pull request from anyone who would like to integrate Rails 5.1 system tests with RSpec. So that's sort of the front matter.
00:03:57.110
Now, let's get into the actual talk about the integrations between RSpec and Rails and how we got there. Before I go too far, I think it's really important to discuss the high-level architecture of the RSpec framework. One of the things that I think not a lot of people realize is that RSpec isn't a single monolithic gem but instead a series of testing components that are designed to work together really well. So when you specify the gem RSpec in your Gemfile, the first thing Bundler is going to do is go fetch the RSpec gem. However, the RSpec gem actually doesn't do anything on its own; it doesn't have any code inside it of any consequence. Instead, what the RSpec gem does is depend on the rest of the components of RSpec that perform together to make the testing framework you're so used to using today.
00:04:29.970
One of the dependencies is called RSpec Core, and that provides 'describe' and 'it' and tagging, and the runner, and all of the tooling that you use to actually build and execute your test suite. RSpec Expectations provides the 'expect' keyword to all of the matchers that you're used to using and the powerful composed matcher system. RSpec Mocks provides the mocking and stubbing capabilities of RSpec—doubles, spies, and all that good stuff. So these are the direct dependencies of RSpec, but they're independent gems. You can pull each part of them on its own without having to have any of the others. All three of these gems depend on what’s called RSpec Support, which most people don't know about. RSpec Support is our internal shared code between the RSpec gems, and we use it for various things like pulling methods of objects and determining method signatures.
00:05:27.780
Nothing in this gem is marked as part of the public API, so you should never call code in RSpec Support directly, but it is there nonetheless. RSpec Rails is an entirely different beast. It functions kind of on its own, packaging the rest of the RSpec ecosystem and also providing a bunch of its own code. It links directly into the Rails version that you're using. In this manner, RSpec Rails is able to provide the entirety of the aspects that you've already used, combined with Rails-specific stuff in order to make testing your Rails app really easy. So if you see 'gem RSpec' and 'gem RSpec Rails' in your Gemfile, this is actually kind of a smell. You're specifying your dependency on the gem twice, and instead, you can just pull RSpec Rails, and everything will be fine.
00:06:06.300
Let's talk about Rails. I think this is a reasonable statement: complexity also comes with the fact that RSpec is really permissive about which parts of Rails you can use at which versions. Typically, RSpec supports all Rails versions greater than or equal to Rails 3.0, which means we support 3.1, 3.2, 4.0, 4.1, 4.2, and soon 5.1. That's a lot of Rails versions, and it leads to a lot of code inside the gem that conditionally switches on what Rails version is being used. This leads our Cucumber examples to be tied with different Rails version support and contributes to documentation that contains Ruby code that switches on Rails versions. It was a lot of work to get RSpec compatible with Rails 5.
00:07:02.800
In the core of this talk, I'm going to be going through some of the issues that I encountered, how I thought about debugging them, where external contributors were involved, and how I helped them make their issues better, or how I got help from other necessary maintainers to fix everything up. This is broken into a series of lessons, and the first one is that major versions mean people can break things. At the keynote, not last year, but the year before, it was announced that controller tests were going to get soft deprecated. What that actually meant was that 'assigns' and 'assert_template' were going to be removed. 'Assigns' is the thing that lets you test the instance variables your controller assigns, and 'template' is the thing that lets you test which template is going to get rendered.
00:08:10.920
I'm like, 'Please no! Stop! There are thousands and thousands of controller tests in the world that desperately need these things to function. You can't do this to us!' I was overreacting because it turns out that the Rails team had a plan, which was this gem called Rails Controller Testing. All the Rails Controller Testing gem does is provide 'assigns' and 'assert_template' again. I'm like, 'Great! This is perfect! I'll just integrate this back into RSpec, we'll run our tests, and everything will be fine.' So let's add it, make sure it works, and oh no, the entire test suite exploded.
00:09:02.470
Looking at this a little bit closer, we discovered that actually, it's just view specs that have stopped working. I'm like, 'Okay, that's less bad, because mostly nobody writes view specs.' And wow, maybe you do! Alright, these specs, and I'm just terrible. No, but specifically a thing that was broken is path helpers, things like 'url_for' and 'gadgets_path,' etc. You know those things? I'm like, 'Alright, let's work out what's going on here.' So we take one of our automatically generated view tests, we stick a pry in it, we look for the gadgets_path, and we get undefined local method for gadgets_path. We switch back to Rails 4.2, we call the same method, and it works.
00:09:52.990
So we know that somewhere between Rails 4.2 and when this gem was extracted, something has broken. I'm like, 'Alright, well what could possibly have changed?' It's pretty likely that gadgets_path is not provided directly on the test but instead one of the modules that gets included in the controller. So I’m like, 'Alright controller, tell me everything that you’re made up of.' And you get a huge list of nonsense back. But just to explain this, the top line there is the singleton class of the object. The second line is the class of the object itself. Then we have Action Dispatch Test and three random anonymous modules including Action Controller Base. And then, a bunch of stuff. Actually, controller base is where we can approximately stop looking because that's the thing we're used to using.
00:10:32.780
You switch back to Rails 5, and it looks materially different, and specifically those three random anonymous modules have just disappeared out of this ancestor chain. And I'm like, 'Anonymous modules are the hardest thing to debug ever because they don't have a name and they don't have a source location. How on earth am I supposed to find this?' So I stick a pry in there, I rip my hair out, I get mad, and eventually, it points me at this line of code. I look at it, and it's like, 'New module including that module into another module,' and then doing some crazy hooks, and I'm like, 'Nope! No pad! Enough!' I mean, I know RSpec Rails like the back of my hand, but I am not an expert at the deep, deep tunnels of the Rails framework, and I'm willing to admit that.
00:11:29.120
So, I hit up Sean like, 'Sean, this is nuts! What's going on here?' and we stopped into a Skype call. We ended up spending something like three or four hours trying to debug this ourselves. Sean, would you say that's about fair? Yeah, yeah, so four hours of my life gone. Then eventually, Sean was like, 'This appears to be a load inclusion order kind of bug.' He wrote this essay of a commit message. This is the first pro tip of this talk: write really long commit messages that explain what’s going on because anything that takes four hours of a Ruby maintainer's time to work out is complicated, non-obvious, and scary.
00:12:07.930
The diff looks like this, and I won’t make you squint, but basically, it’s changing the order that the controller testing gem is including stuff in, and everything works. We went from a broken gem to a functional gem, and I think this is a huge win for collaboration. When I look at this change, I was like a slow low maintainer, scared, lost, and confused, and I’m like, 'Hey, Ruby ecosystem, I need some help!' And it helped me! It was great! I love it when we get together and fix that stuff, it makes me really mad that I had to debug anonymous modules, but I think this is a great win.
00:12:48.480
And then we were basically ready to release the gem, and kids and users started filing issues. Oh my god, so many! It turns out that the test suite that you have usually doesn't cover all the cases, and your users are doing things you can't possibly imagine. So the remainder of the issues that we’ll be looking at today were issues filed by actual RSpec users as opposed to stuff I encountered while trying to do the upgrade. Lesson two: it is actually possible for Rails to have bugs. This may seem like a controversial statement, but I hope you believe me.
00:13:28.120
We’re talking about an RSpec Rails issue, issue 16:58; you can look this up. The user was basically like signed cookies are not available in the controller aspect, and ask their Rails. I’m like, 'What the hell is a signed cookie? I've literally never heard of that Rails feature before.' I say, 'Alright, I’m sure that's supposed to work. Let's find out.' So I label it 'triage,' and this is a pretty common thing for a maintainer to have. It's a personal state machine for how they move issues.
00:14:16.460
When I label an issue as triage, it basically means, 'You filed this, and I haven't yet had enough time to work out whether there's enough information to reproduce the issue.' I let it lie for a little while, and I come back to it a few days later. I come to the conclusion that the original issue didn't have enough information in it. So, I ask this question of the original submitter, which basically says, 'Thank you for filing this bug. I wasn’t able to reproduce it with the information you gave me. Please give me a Rails app that I can clone so that I can just run bundle exec RSpec and see what your bug is.' Then I apply the label 'needs reproduction case.' I think this is an incredibly common maneuver for maintainers these days.
00:15:16.840
Even though filing an issue represents having a knowledge point that something is wrong, it can be really hard for us to work out what exactly is broken. Rails is a complicated environment. There are many gems, and all of your configurations cause this bug to appear. And frankly, I can't necessarily reproduce it just with a few lines of RSpec in an issue. If the user comes back immediately, am I like, 'Yes! Done! Clone this! Bundle exec, fantastic! You’re good!' I'm like, 'Yes! Go team! We did it!' This is how open source is supposed to work.
00:16:11.980
I confirmed that the issue is real, and I move the issue from 'needs reproduction case' to 'has reproduction case.' This state machine, this triage needs, is something I do with nearly all RSpec issues. It helps me know what I do and do not have to fix. Now that I have this green issue, I can actually begin debugging the problem. So we clone the Rails app that the user has provided, and we also clone a fresh version of Rails into its own repository. I do this so that I can move the commit that it’s on backwards and forwards.
00:17:03.500
Then, I point the Rails app the user has provided at my fresh Rails clone. I check that the tests are still failing and I get a 'bad' from Git. I check out 4.2 Beta 4, and there’s a very specific reason it’s this version. This is the last version of Rails 4.2 on the master branch before 4.2 was released. They have a 4.2 stable branch that doesn’t track master, so I can’t simply bisect between 4.2 and 5 on that branch. I have to use master. So, I check out this specific commit, and I run my tests. They pass, and I get a 'bisect good.' Git tells me I have 6,794 revisions to check in order to determine what’s wrong. Oh god, this is my life now.
00:17:58.960
So I basically run the test suite backwards and forwards. Git bisect bad good bad good bad good, and eventually, it’s commit pops out—commit number A29—whatever. I leave myself a little note as to exactly what the problem is, but not necessarily a fix. This is definitely one where the bug exists in Rails, not in our Specs interaction with it, and so I leave it for a while.
00:18:35.180
Life happens; I get a job where I have to move to New York, and that’s a thing. Then, after a while, I'm at RubyConf, having a discussion with somebody. I say, 'So I bisected six files and revisions in Rails, found the specific problem, and I'm not really sure what to do now.' Sean turns around and goes, 'You have a breaking commit—just show me! I’ll fix it right now.' It was like, 'Alright!' So I go onto GitHub, and I literally just paging around, and lo and behold, immediately this issue is not a bug in RSpec; it can be reduced with Rails alone.
00:19:22.330
Sean opens a blog on Rails which provides a Rails reproduction script. He specifically states this can be replicated purely with Rails using the public API. In open source, we have a twisty, turny maze of dependencies with your Rails app that uses RSpec, causing Capybara and a dipping down and a CoffeeScript compiler for some reason in 2017. If you've got rid of that, that’s an unfair accusation. So, before filing an issue on a specific gem, it’s worth noting that the bug could be anywhere in the dependency tree of that gem. It’s well worth powering down and seeing if you can reproduce it with a subset of the things that are available.
00:20:07.160
Specifically, Rails has a guide for how to provide reproduction scripts, and before filing bugs on RSpec Rails, I thoroughly encourage you to check that it's not a bug in Rails first. Lesson three: sometimes you can’t just call Sean to fix the problem. I love you, Sean. So this issue, RSpec Rails 1662, can no longer set host bangs in before blocks for request specs. That's not immediately obvious, so let me show you what this means.
00:21:05.360
Basically, if you have an RSpec configuration that globally sets a host in a before block, this stops working. And there are good reasons to do this. Before blocks are useful, and this host bang can happen if you have router configurations that cause you to have subdomains in your app that are also being responded to. I’m like, 'This should work.' I feel like I touched this code when I was upgrading RSpec to work with Rails 5.
00:22:02.670
So, other RSpec maintainer, please don’t close this issue! I want to take a look at it. I asked the submitter for a reproduction case exactly as I did in the other issue, and they came back to me with not just a reproduction case but with a short screencast of how to use this app to debug it. I’m like, 'Oh my god! Yes! That’s the best thing ever!' That’s so much better than just cloning this and running bundle exec.
00:22:51.050
Like a spiky, he walks me through, shows me where the bug occurs, shows me what he tried changing. So good! I pull down the reproduction app, I do the same bisect shimmy and shake thing, and the commit message spits out. I go have a look, and it is basically in the change to Rails 5.
00:23:26.170
This got committed, which changed the application initialization logic to happen lazily instead of happening before setup blocks. Before setup blocks are equivalent to before each in RSpec and so will always happen after that before all the user has specified. That's our bug. So I’m like, 'This one is simple enough,' but I can fix it myself. I open a pull request on Rails explaining what changed, why it broke the thing, and my proposed fix.
00:24:23.390
Maintainters love it when you drop context on them. The more words you write about your change increase the likelihood of it getting accepted because it just makes it easier for us to understand what’s going on. I also did that as part of not just the pull request message but the actual commit as well. The reason for that is that if any Rails maintainer in the future is ever blamed, they can look at that commit message and see what changed, why, and who did it.
00:25:01.390
They can come at me on the internet if I broke everything, but I don’t think I did. After some discussions and sort of round trips with Aaron, I think Rafael got involved for some of it, Sean got involved for some of it, I got merged, and then I completely forgot about it and didn’t close the issue on RSpec Rails for multiple months.
00:25:58.800
So someone just comes along and it’s like, 'Hey! You fixed the bug in Rails; you should close this issue.' Alright, you’re right! I totally, totally should! I’m a terrible, fallible human, and the truth is, if I fix something and you just ping me on GitHub and are like, 'Hey, you fixed this; do you still need this issue to be open?' I can be like, 'No,' and then I can close an issue.
00:26:52.040
You did a simple one-minute action that gets rid of an issue on my issue tracker. As an external contributor, that makes me super happy! To summarize, RSpec definitely has bugs, but the surface area of our integration with Rails is not all that big. If you have a Rails-specific bug in your RSpec suite, I have seen it take its place in both, but it can be because of Rails, and literally every fix that I showed today was in a gem that is owned by the Rails project, not by RSpec.
00:27:23.640
I know that if you're in this room you're more likely to be a fan of RSpec than a fan of MiniTest, but if you can use the Rails reproduction script guide—which asks you to test it in MiniTest—just to make sure that’s where the bug lies. Rails has a really good guide for doing that. When you open an issue that has more than the most basic form of complexity, I'm likely to ask you to send me an app where I can reproduce exactly what's going on.
00:28:01.790
The reason for that is not that I value my time more than yours; it’s that five minutes of you doing that extra work of packaging it up can save me literally hours of debugging in my framework, and that’s really important. You can always just call Sean to get Rails bugs fixed.
00:28:23.180
Ok, real talk though. I think working on open-source is a really hard thing to do as a sort of career maintainer; it absorbs your evenings and weekends, and sometimes stuff goes out into the world, and everyone's test suite is on fire, and you are the only person in the world who can fix it. I have a friend who once tweeted: 'retweet if participating in open-source has made you cry.' That has happened to me a hundred percent.
00:28:53.210
I can tell you that for sure. The work represented in this talk took more than 40 hours of maintainer time—that's a full work week. My time, other RSpec maintainers, Sean, and the Rails core team, all of you are wonderful people. Give maintainers hugs; they really deserve it. More importantly than that, I think open source is more approachable than it has ever been, and it will continue to do so.
00:29:22.440
The previous talk was about how to contribute, to get started contributing to Rails. The tour before that was a deep dive into a new Rails feature. If your company materially depends on the existence of Rails, RSpec, Bundler, Ruby gems in order to exist, it seems natural to me that you should find some time to work on that.
00:30:06.490
And with that, you saw a giant red slide at the beginning of this talk—I'm a fan of communism. Maybe we should find ways to ensure that maintainers can get paid for that work. Just a thought! So just one random chair.
00:30:42.220
Ah! In the meantime though, please buy me drinks at the bar if you use RSpec; that seems like a fair trade-off, right? Alright, quick wrap-up: I work for a company called Digital Ocean. We are an infrastructure as a service provider, meaning we do servers, block storage, networking—basically all the primitive components for you to build amazing applications.
00:31:08.440
We are hiring! I like working there a lot; it’s a lot of fun. Our products are cool! I have swag! Come find me! Let’s talk about servers! That’s all I’ve got.
00:31:22.710
Thank you very much for listening! I'm Sam Phippen on Twitter, Sam Phippen on GitHub. If you would like to email me, I'm [email protected]. Thank you! I will take any and all questions.
00:31:37.630
Are the anonymous modules still there? Yes, to the best of my recollection, it’s like an ActiveSupport concern that does something with route sets. Listen, I’ll send you the commit.
00:31:58.950
Computers are extremely bad! One thing that could be done there to make it easier is to have those modules defined self-name with something sensible, but I don’t know what the performance implications of that are or if it’s a hot path or whatever.
00:32:22.530
And there's a whole section of Rails maintainers just over here. So maybe just ask them! Anyone? I literally can't see. Shout! Be loud!
00:32:34.650
Alright, thank you very much!