00:00:03.679
Thank you! All right, hey everybody. You know, if it were a different time of day, you might be more likely to see me at a different one of Asheville's venues, making idle chitchat on the floor of a concert hall, waiting for that moment when the house lights turn out, the fog machine rolls in, the stage lights turn on, and the band takes the stage.
00:00:07.359
We happen to be here first thing in the morning, so we don't have that, but I figured we got the crowd all together here, so let's have a show! In fact, let's help stage an entire concert tour with the help of Ruby's Coverage Module. My name is Kevin Murphy, and I work at a company called Pub Mark. We help you find your next ebook or audiobook deal. Now, if you've heard of coverage—and it's fine if you haven't—it's probably been in the context of a conversation about test coverage, which is a metric to evaluate how much of your application code runs when your test suite runs. Typically, this is provided as a percentage. This is not a talk about test coverage; this is a talk about the tools you can use to measure things like test coverage with Ruby.
00:00:38.480
Ruby's Coverage Module has four different modes: lines mode, oneshot lines mode, branches mode, and methods mode. We're going to talk about each of them today. So we have our set—let's start the show! We're going to start helping out the band’s guitarist, or more specifically, the guitar tech. We're getting ready to go out on tour, and we need a lot of equipment, and one of the things we need a ton of are guitar strings. You see, when you play the guitar, you pluck the strings, and sometimes they just break. If they don’t break, then the tension loosens a little bit, and you play the note you wanted to, and that’s all great. But what makes a string break? It breaks when the tension isn’t set correctly, so we can predict that—that’s great—but it also just happens sometimes. There’s some randomness in the universe that we can’t predict.
00:01:14.560
When a string breaks, it marks itself as broken and plays a sound that is not the one you wanted to play. So we're trying to figure out how many of these strings we need in case we need to re-string an instrument during the show, and the truth is we just don’t know. So we're going to guess for the first ten shows of the tour and hope that we're right. We'll play the shows, keep track of how many strings we're breaking, and use that to place an order for the rest of the tour. So you might think we can just stick in a logging statement in the string class or something, just a quick one-liner to keep track of how many strings we're breaking, but we’re just out to get out on tour. The band has their instruments set up exactly the way they want; we cannot touch that instrument at all—the code is absolutely frozen; we can’t get anywhere near it.
00:02:01.000
So instead, we're going to play the first ten shows with the help of coverage. Coverage is part of Ruby's standard library, but if you want to use it, you need to require it, and you tell coverage to start running with the aptly named start method. Here, we're going to start coverage in lines mode, which will tell you how many times each line of code was executed while coverage is running. We're going to use this to find out how many times we’re breaking strings. We’ll load up the first ten shows of the tour and find out what our coverage is by calling the result method on coverage, and what we get back is a big old hash. Let's talk about what's actually in this hash. Each of these elements is keyed by a string representing the name of a file executed while coverage is running. The value of which is another hash keyed on the mode of coverage.
00:02:59.159
Here, we asked for lines mode, so coverage very nicely gave us lines mode back. The value of this hash in lines mode is an array of mostly numbers, which tell us how many times each line of code was executed. The order in this array is important. The first number is how many times line one was run. Lines one through six were each run one time; line seven is a special case. It doesn’t mean it wasn’t run; it would be zero in that scenario. This is coverage telling us 'Don’t worry about it.' If we go to line seven in the source code, it’s an empty line, so coverage avoids the philosophical discussion of what it means to have run or not run an empty line by instead saying, 'I don’t care, and neither should you.' It does this for comments; it does this for empty lines; it does this for similar lines like that.
00:03:43.599
So we're trying to figure out how many strings we've broken, so we need our lines coverage for our strings class. A string is broken when it sets that broken instance variable to true, and that’s on line fifty-four. So we're going to get into that array and ask for array index fifty-three because arrays in Ruby are zero-indexed to represent line fifty-four. We see that we broke sixteen strings over the first ten shows, and now we have a scientifically significant answer for a rock and roll tour for how many strings to order for the rest of the tour. So the guitarist can confidently go up on stage knowing that if they break a string, we have enough in the back to help them out. We got there thanks to lines coverage, which told us how many times each line of code was executed.
00:04:38.159
That’s not all coverage can tell us. Next, we'll help the band get ready for one very specific show. They've been booked to play a festival, and this is great news! They get introduced to an audience that otherwise might not come see them. They play a shorter-than-usual set, so less work, and the money's pretty good. It does have its share of challenges, though; they need to get on and off the stage really quickly. There are a lot of bands and a very strict schedule, so they need to make sure they stay on track and don’t ruin the experience for everyone. There's also only so much room in the back with all of these artists and bands for all of the trucks and equipment. So the crew is trying to figure out if there's some way we can reduce the amount of stuff we have on stage.
00:05:39.600
One of the things the band uses a lot of are synthesizers. Every member of the band plays the synth. Even if you're the bassist, you play the synth. If you're mostly a synth player, you're probably playing more than one. When I said everyone plays the synth, I meant it! Here's a picture featuring the drummer. Notice they're not playing a drum kit; they are playing the synthesizer. They do use live drums, but in this instance, it was more important that they play the synth. So they have a ton of these instruments strewn about the stage, and the crew is thinking if we can just get rid of one or two of these, it would save us a lot of time.
00:06:01.919
We’re trying to figure out if there's some way we can reduce the number of synths we have on stage. The way a synthesizer works is that it synthesizes different sounds. These sounds are the result of spinning different knobs and turning different dials and moving different faders and pushing different buttons to generate different sound profiles. On a digital synth, if you want to save these to recall them quickly, you can save them in something called a patch. It's essential when you play the synthesizer that you set the right patch before you play the right note because even if you play the right note musically, you've done the correct thing, but if you use the wrong patch, if someone’s familiar with the song, it's going to sound weird, if not just straight up wrong.
00:06:59.920
Now, each of these instruments store their patches differently, but let's look at one of these instruments. This happens to be from a manufacturer right here in Asheville; they have their factory just down the street. They have a storefront; I was there yesterday—go check it out if you have some time. This happens to be an older instrument. It only has four bays of patch memory. They use a case statement to access all of their patches, right? So when a user hits one of the four buttons, it just accesses the different patches out of memory. We're trying to figure out how many of these patches do we actually use. Remember, the band's playing a shortened set, so they're not playing some of the songs they regularly do, and there might be some opportunity to consolidate.
00:08:01.480
Now, luckily, with this being a different set, the band also needs to practice it. When they practice, we're going to set up the stage exactly as they would for a regular show, with all of their synths and all of their patches that they regularly get, and the band's going to practice the show. While they're doing that, we are going to once again use Ruby's Coverage Module. This time, we're going to start coverage in oneshot lines mode. Onesho lines mode can’t tell you how many times a line is run, but it will tell you if it is run, and that’s all we need here. Whether a patch is used one time or a million times during the concert, we still need it on stage, so this is sufficient to meet our needs.
00:08:58.320
The band's going to practice for the festival, and we'll see how many of these patches we’re using. What comes back looks really similar to what we saw with lines coverage, but let's explain all of the elements again. We still get a hash keyed on the name of a file executed while coverage is running. The value of which is a hash keyed on the mode of coverage. For oneshot lines mode, the value of that hash is an array of numbers. What these numbers are is different than lines mode; these are the actual line numbers that were executed because recall, oneshot lines doesn’t tell you how many times. So if we want to find out which lines of our patch memory were used, we get back a bunch of integers.
00:09:33.200
We want to know which of these patches we use, and we're going to keep track of the line numbers inside the case statement to say this is how we prove that we accessed these different patches. We'll build some incredibly fancy output here to keep track of that and give us a nice little output. We see that for this instrument, we’re using three of the four patches, which isn't great news in our consolidation effort. We definitely need this on stage, but if we perform this analysis for every other instrument, we might find one that we’re only using one of its patches, and maybe we can find a similar enough sound on that first instrument that will allow us not to set up that second one at all.
00:10:14.480
This would save a lot of time and space, allowing the band to still play the show and have it sound the way they want it to for the audience. A little more difficult for them because things aren’t set up how they want, but allowing us to have the festival stay on track, we got there with the help of oneshot lines mode, which kept track of whether a line of code was covered to find out if we were using a patch. We're on to the second half of our show! We're going to talk about methods coverage. We’re going to help out the lighting team; they have themselves a bit of a problem here. There’s a spotlight on the lead singer, and sometimes during the show, it just goes out — and not when it’s supposed to!
00:11:06.640
This is a problem because the lighting team takes a lot of pride in their work, but also the lead singer is a bit of a perfectionist. So let’s just say they’re well incentivized to figure out what this problem is. Now, similar to the band's reliance on synthesizers, they also work with a lot of lights—tons and tons of lights! They work with lighting companies and special effect groups to design lights that didn’t even exist before, just for their tours. So they have a lot of small systems all working together literally in concert to produce this giant output. We have one very specific small behavior that's not working exactly how they want in this giant monolith. Does that sound familiar to anybody?
00:11:54.679
So we have a very complex debugging problem on our hands. The way that lighting works at a concert is that the lighting team are performers just like the band on stage except they get none of the credit, but for every single note of every single show of every single song throughout the entire tour, they’re making sure that the right lights are turned off, the right lights are turned on, and the lights that are turned on are facing the right way, have the right color, are using the right effect, and are moving the right way so that the visual aesthetic of the stage looks exactly the way the band wants it. So the band can focus on playing the music.
00:12:12.640
Now for our lighting system, we have a lot of different types of lights, but whether we’re using a projector, a c-light, a moving light, or a problematic spotlight, all of them respond to this method called trigger. It's how you turn a light on or off. What those trigger methods do is radically different, but they all have the same named method. One of the hypotheses for why our light is turning out is that there’s not enough power at the venue, and it’s taking it out on our spotlight here. Now, you might think that’s impossible; we have a contract, we have a tour rider with the venue that specifies exactly how much power we need, and they signed it. You’re correct; you’ve got me there. But I will say that sometimes honest mistakes happen, and it’s just a hypothesis—we don’t really know. Also, as the tour's gone on, the band started playing different songs, which means they've stopped playing other songs, and there may be some lights that we only use for one song that we’re not even playing anymore.
00:13:02.240
But we can't keep track of it because there's just so much going on and so many lights. The next time we play a show, we're going to set up the stage with all of the lights and all the rigging we regularly have. We'll turn everything on, have the band play the show, and then tear everything back down again, moving it into the trucks and vans and heading off to the next city. But we're going to take advantage of this opportunity yet again to use Ruby's coverage module to figure out which lights we're using, to see if we can turn some of them off. This time, we're going to start coverage in methods mode to keep track of how many times methods are run, and we’re going to keep track of how many times we're calling these trigger methods on these different classes.
00:13:57.560
The band is going to play the show, and we’ll see what comes back. What we get looks different from what we've seen so far. We still get a hash, and that hash is still keyed on the name of a file executed while coverage is running. The value of which is a hash keyed on the mode of coverage. For methods coverage, what you get for a value there is another hash keyed on a bunch of information identifying the method, and the value of that innermost hash being the count for the number of times this method in this file was executed while coverage is running. But I skipped over a bunch of stuff; we’re going to talk about it now. Don’t worry! We already talked about the count of this in the value of this innermost hash; it’s the count for the number of times this method was executed.
00:14:51.760
Now for the keys going left to right, we have a constant representing the name of the class, a symbol for the name of the method, and then we get back four numbers. These numbers tell us the starting line number, the starting column number, the ending line number, and the ending column number for where we can find this method in this file in the source code if we need to. Now here, we’re just looking to see if there are any lights that we’re not using, and it turns out we’ve set up a projector for every one of our shows, and we don't actually use it anymore. So the next time we go to light the lights, we can say, 'Hey, not so fast there, projector! Thank you for your service, but we don’t need you anymore.' Hopefully, that allows the rigging to have enough power to light the lights that the band actually needs for their show.
00:15:57.120
And even if not, we’ve made the system a little less complex by taking the projector out of the equation. We were able to figure out that we can remove it with the help of methods coverage, which told us how many times we were executing each method while coverage was running. We have one last song together here today for our last song. We're going to get ready for the last show of the tour. We happen to be in a town that has another band in it that they're friendly with, and they’ve invited the lead singer up on stage and said, 'Hey, come play a song with us.' Let us know what you want to play, and we’ll do it. So they figure out what song they’re going to play, and it’s one of the band's songs, but it’s not a song they’ve been playing on this tour.
00:16:51.440
So, when the show’s leading up to it, they’ve been using soundcheck to practice so they’re not playing it for the first time in front of an audience. For soundcheck, the band knows what song they’re going to play, and it’s like mid to late afternoon, and the band ambles onto the stage—which might as well be first thing in the morning for a rock tour. All of us are gathered here today; we all made it in the morning! There’s some light stretching and talking about what they’ve been up to, and finally, they get to playing the song. It sounds all right, but something's a little bit off, and the band can’t quite figure it out. They’re a relatively tech-savvy group, so they come to us in the crew, and they say, 'Hey, we know you’ve used this coverage thing to help us out a couple of times. Can you figure out what's going on with this song?' We’re the crew; it’s our job to say yes, so we say yes. But now we have a bit of a problem on our hands.
00:18:04.920
Previously, we’ve been using coverage to answer a very specific question, but right now the question is this: 'This sounds kind of weird,' which isn’t even a question! But we told them we would give it a try, and so we don’t really know if coverage can even help us out here. So we’re going to start coverage in a special way; we’re going to pass it the all symbol, and that tells coverage to run everything and tell me everything you got. I’ll figure it out later, right? So the band’s going to practice the show at soundcheck again, and we’ll see what comes back. Unsurprisingly, what we get is a ton of information, but I want you to take from this a further appreciation for the data structure.
00:18:56.520
Previously, we’ve been talking about how coverage gives you back the mode of coverage, and that sounded repetitive, right? Of course, we started in lines mode and got lines mode back. But you can run coverage in multiple modes and access this data in a consistent way, whether you're running one mode or all of them at the same time. So, we ask coverage to start in every mode, and what we got back was lines mode, methods mode, and branches mode. But we spent an entire section helping out getting ready for a festival with oneshot lines, so what happened here? It turns out there's an implementation detail in the actual C code of how coverage is written. Lines mode and oneshot lines mode can't be run at the same time, and it will raise an exception. Rather than just raising an exception every time you ask it to run all the modes, Coverage instead chooses to use lines mode over oneshot lines. The reason for that is because lines mode can tell you the same information as oneshot lines; lines mode will tell you if a line of code was covered and how many times.
00:20:35.040
Now, enough procrastinating—time to actually figure out if coverage can help us with the weird sound of the song. So, we ask if we're using every line of code in the song, and it turns out we are. So, no help there. If we’re using every line of code, we’re probably calling every method, but we had the data, so we had to check. We ask for branches mode, and we see a zero here, which maybe piques our intuition. But we haven't actually talked about branches mode yet. So once more, for nostalgia purposes, let’s all work together to talk about what comes back when you ask coverage for its result: you get back a hash keyed on the name of a file executed while coverage is running.
00:21:27.920
The value of which is another hash keyed on the mode of coverage. For branches mode, the value of which is another hash keyed on each conditional in the file. The values of that hash are another hash keyed on each of the branches in the conditional in the file that were executed while coverage is running. Let’s look at one of these branches and describe all the data in here. You may have guessed already that the value on the right-hand side is the count for the number of times this branch in this conditional was run while coverage was running. Now, going left to right through the keys, this first symbol identifies the type of branch; this is the first part of an if-else statement, denoted with the 'then' keyword. Even though we don’t typically use the 'then' keyword when writing an if statement, this first number is an identifier. All branches and conditionals get an auto-incrementing unique identifier in branches mode; I don’t know why, but they do. It’s neat!
00:22:19.520
Then these next four numbers are the same numbers from methods coverage with the starting line number, the starting column number, the ending line number, and the ending column number for where we can find this branch in the source code. Now, here, we’re just looking to see if there are any lights that we’re not using, and it turns out we set up a projector for every one of our shows, and we don’t actually use it anymore. So the next time we go to light the lights, you can say, 'Hey, not so fast there, projector! Thank you for your service, but we don’t need you anymore.' Hopefully, that allows the rigging to have enough power to light the lights that the band actually needs for their show. And even if not, we’ve made the system a little less complex by taking the projector out of the equation.
00:23:17.800
We were able to figure out that we can remove it with the help of methods coverage, which told us how many times we were executing each method while coverage was running. We have one last song together here today for our last song. We're going to get ready for the last show of the tour. We happen to be in a town that has another band in it that they're friendly with, and they’ve invited the lead singer up on stage. They said, 'Hey, come play a song with us.' Let us know what you want to play, and we’ll do it. So they figure out what song they’re going to play, and it’s one of the band's songs, but it’s not a song they’ve been playing on this tour.
00:24:45.000
When the show’s leading up to it, they’ve been using soundcheck to practice so they’re not playing it for the first time in front of an audience. For soundcheck, the band knows what song they’re going to play, and it’s like mid to late afternoon, and the band ambles onto the stage—which might as well be first thing in the morning for a rock tour. All of us are gathered here today; we all made it in the morning! There’s some light stretching and talking about what they’ve been up to, and finally, they get to playing the song. It sounds all right, but something's a little bit off, and the band can’t quite figure it out. They’re a relatively tech-savvy group, so they come to us in the crew, and they say, 'Hey, we know you’ve used this coverage thing to help us out a couple of times. Can you figure out what's going on with this song?' We’re the crew; it’s our job to say yes, so we say yes. But now we have a bit of a problem on our hands.
00:25:57.679
Previously, we’ve been using coverage to answer a very specific question, but right now the question is this: 'This sounds kind of weird!' Which isn’t even a question! But we told them we would give it a try, and so we don’t really know if coverage can help us out here. So we’re going to start coverage in a special way; we’re going to pass it the all symbol, and that tells coverage to run everything and tell me everything you got. I’ll figure it out later, right? So the band’s going to practice the show at soundcheck again, and we’ll see what comes back. Unsurprisingly, what we get is a ton of information, but I want you to take from this the appreciation for the data structure.
00:26:56.679
Previously, we've been talking about how coverage gives you back the mode of coverage, and that sounded repetitive, right? Of course, we started in lines mode and got lines mode back. We don’t have to run coverage in one mode; you can run a lot of them, and this gives a consistent way to access all this data, whether you’re running one mode or all of them at the same time. We asked coverage to start in every mode, and what we got back was lines mode, methods mode, and branches mode. But we spent an entire section helping out getting ready for a festival with oneshot lines, so what happened? It turns out there's an implementation detail in the actual C code about how coverage is written—lines mode and oneshot lines mode can't be run at the same time.
00:27:48.960
It will raise an exception. Rather than coverage just raising an exception every time you ask it to run all the modes, it instead chooses to use lines mode over oneshot lines. The reason is that lines mode can tell you the same information as oneshot lines; lines mode will tell you if a line of code was covered and how many times. So, enough procrastinating! Time to actually figure out if coverage can help us with the weird sound of the song. So we ask if we’re using every line of code in the song, and it turns out we are—so no help there. And if we're using every line of code, we're probably calling every method.
00:28:42.080
But we had the data, so we had to check, and we ask for the branches mode! We see a zero here, which maybe piques our intuition, but we haven't actually talked about branches mode yet. So once more, for nostalgia purposes, let's work together to talk about what comes back when you ask coverage for its result. You get back a hash keyed on the name of a file executed while coverage is running. The value of which is another hash keyed on the mode of coverage. For branches mode, the value of which is another hash keyed on each conditional in the file. The values of that hash are another hash keyed on each of the branches in the conditional in the file that were executed while coverage is running.
00:29:37.080
So let’s look at one of these branches and describe all the data in here. You may have guessed that the value on the right-hand side is the count for the number of times this branch in this conditional was run while coverage is running. Now, going left to right through the keys, this first symbol identifies the type of branch; this is the first part of an if-else statement, noted with the 'then' keyword. Even though we don’t typically use the 'then' keyword when writing an if statement, this first number is an identifier—auto-incrementing unique identifier for all branches and conditionals in branches mode. I don’t know why it exists, but it does! It’s neat. These next four numbers are the same numbers from methods coverage with the starting line number, the starting column number, the ending line number, and the ending column number for where we can find this branch in the source code.
00:30:43.440
Now, we’re going to use that here because you might see this branch starts and ends on the same line. In fact, the entire conditional starts and ends on the same line—it’s a ternary statement! Even if you’re willing to follow what the 'then' part of that means, you’d probably be more confused when you just see this question mark-colon syntax. So we’re going to use the columns to say, 'Hey, this ten never gets returned,' and that could be totally fine. Now we actually have to figure out whether that’s a problem. So let’s take a step back and talk about what this code is doing. This sets up a vocal effect on the lead singer's microphone for all of the choruses that are supposed to wax and wane in intensity as the different choruses go on. But it’s never waxing; it’s only waning.
00:31:39.120
The reason is that when we start our song, we pass certain numbers into the chorus, and the numbers we pass in don't meet both conditions of the conditional, so it always goes to the else case, and that’s not what we want here. So we found an actual bug in our code; we can change that. It's kind of an off-by-one error; we pass different numbers in, and we have the band rehearse the song one more time at soundcheck. We execute both sides of the branches, and after making that code change, our song sounds the way it’s supposed to, and they can confidently go up on stage with their friend for the last show of the tour—all thanks to branches coverage, which told us how many times each branch of code was executed while coverage was running.
00:32:43.440
Branches mode gave us different information than we saw from the other modes. Right? Lines mode told us every line was covered, and that was true, but we had one line of code that had multiple code paths inside of it, and one of those code paths wasn’t executed when it should have been. So, what have we learned about coverage here today? Coverage has four different modes that can answer different questions for you. If you want to know how many times each line of code was covered, you can use the default mode, which is lines mode. If you don’t care about how many times it was run but just if it was run, you can start oneshot lines mode. The benefit here is that oneshot lines is a bit more performant than lines mode because it doesn’t need to keep track of a count of how many times; it just needs to know that it was run.
00:34:21.760
If you don’t want to mess with line numbers at all but want to know if a particular method was executed or how many times, you can use methods mode. Lastly, we just talked about branches coverage to tell us how many times each branch of code is executed. Now, with all of this newfound information about coverage, I’m willing to bet that it’s unlikely—though conceivable—that you’ll use coverage directly. But that doesn’t mean that all this information is lost; you can use this to tune the tools available to you that do use Ruby’s coverage module. I started by saying this wasn’t a talk about test coverage, and hopefully, that’s borne out to be true. But there are tools that use coverage that can evaluate your test coverage, like the simple COV gem.
00:35:54.000
By default, simple COV runs in lines mode, but you can tweak it and ask it to run in branches mode to get your test coverage of your branches, which might tell you different information than lines coverage does. There’s also a gem called cover band, which runs in production, and if you want to use cover band to identify dead code—code that hasn’t been used, and therefore can safely be removed—then you don’t care about how many times it was run, just that it was run. You can start cover band in oneshot lines mode, and now remember, oneshot lines is more performant, and cover band runs in production, so anything you can do to minimize the overhead in all of your requests is of benefit to your users and also to you and your systems.
00:36:49.920
Now you can take this information and apply it to your tools that use coverage to better meet your needs. If you'd like a copy of these slides, if you'd like a narrative description of coverage, if you'd like to know how to get in or stay in touch with me, you can get all of that information at Kevin J Murphy dot com slash coverage. If you’re interested in trying out Chirp and getting your first audiobook for 20% off, you can use code bluebridge ruby. I’ll be honest; I don’t know how long that code is valid for. That’s not meant in an 'act now' way; it’s just that I legitimately don’t know. I’m happy to take any questions or talk to anybody. I’ll be just off the side of the stage over here, and we’ll chat individually. I want to thank all the organizers for bringing us all together here today, the volunteers for making this all happen. I want to thank all of you for being here. I hope you’re as excited as I am to sit in those seats and listen to the rest of the talks coming afterwards. We have a great lineup here, so I hope you are super jazzed to be sitting where you are, because I’m super excited to be joining you for the next two days. But nonetheless, I hope so far you’ve enjoyed the show. Thank you!
