Enough Coverage To Beat The Band

Testing

Kevin Murphy

@kevin-j-m

#code-coverage

#ruby

#software-development

#testing

#developer-tools

#rubygems

#performance-optimization

Enough Coverage To Beat The Band

by Kevin Murphy

In the video titled "Enough Coverage To Beat The Band" presented at RubyConf 2020 by software developer Kevin Murphy, the concept of code coverage in software development is vividly illustrated using a concert metaphor. Kevin draws parallels between preparing for live music performances and the importance of code coverage in software applications, emphasizing how proper coverage can boost confidence in deployment and ensure code quality.

Key Points Discussed:
- Introduction to Coverage: Kevin introduces himself and sets the stage by describing the excitement of a concert as an analogy for the coding process. He explains the need for thorough preparation in both scenarios and how coverage serves as a crucial tool for developers.
- Types of Coverage:
- Lines Coverage: Demonstrates how to track the execution of each line of code. He uses an example of a band's practice set, where the execution of certain lines (or strings broken) is counted during performances.
- One-Shot Coverage: Focuses on specific instances, such as optimizing equipment usage for a particular show, illustrating how this mode aids developers in identifying unutilized parts of their code.
- Methods Coverage: Highlights tracking how often specific methods are executed rather than individual lines. This is exemplified through the lighting team managing numerous stage lights to determine which were actively used.
- Branches Coverage: Discusses the exploration of conditionals in the code, comparing it to transitioning between songs in a setlist, highlighting potential bugs like an off-by-one error that could affect performance.
- Practical Applications: Each coverage type offers insights that allow developers to refine their code, minimize dead code, and streamline performance, akin to a band's preparation for diverse concert settings.
- Conclusion and Recommendations: Kevin summarizes the benefits of utilizing Ruby's coverage tools, suggesting gems like 'simplecov' and 'coverband' for assessing test and production code coverage respectively. He encourages audience members to utilize these tools to enhance their coding practices.
- Engagement: Kevin invites attendees to participate in a live Q&A in the associated Slack channel for further discussion on coverage and share personal concert experiences.

The video serves as both an educational session on Ruby's coverage module and a creative reminder of the parallels between the meticulous preparation of a concert and software development, emphasizing the importance of such practices in ensuring high-quality code delivery.

00:00:05.600 Well, hello everybody! Here we are again for the next set of sessions. If you have not been paying attention or have not followed the channel associated with this talk, I recommend that you do so. You can find it by browsing the channels in the open chat, which will give you the opportunity to ask questions or follow up as necessary.

00:00:27.519 Looks like we have an echo, so let's see if we can figure out where that's coming from. Let's try and start this again and see if that helps. If it doesn't, we'll cut it short. Now, regarding the echo, it seems to be isolated, which is good. Let me introduce Kevin Murphy. He's going to go through our next live talk, which is really exciting to me. Don't forget to follow the Slack channel so that you can ask questions in the follow-up, and we will go over them there. Alright, without further ado, Kevin, it's all yours!

00:01:29.119 There was a time, not that long ago, when people could safely gather in crowds and wait for that wonderful moment when the house lights would turn out, the fog machine would kick in, the stage lights would turn on, and the band would take the stage. Concerts are one of the many things that I’m missing in 2020, but I figured, you know what? I've got a captive audience here. Let’s have a show! In fact, we’re going to help stage an entire tour of concerts using Ruby’s coverage module.

00:02:51.760 As Adam mentioned, my name is Kevin Murphy, and I’m a software developer at The Gnar Company, a software consultancy in Boston, Massachusetts. More accurately, these days, I’m working from my couch, dining room tables, and makeshift offices in the greater Boston area. We call our method the aptly named 'start method', and here we’re going to explicitly ask it to start in lines mode. We’ll plan the band to play the first ten shows of the tour and see what we get back.

00:03:37.440 What we get out of coverage is a large hash. I’ve already condensed the output here, but let’s dig in further into just one of the elements of this hash and explain what we get back as results. Each of the keys in this hash are the names of the files that are executed while coverage is running. The value of each key is another hash, where the key is the name of the mode. Here, we explicitly asked coverage to start in lines mode, so thankfully we got lines back.

00:04:17.760 For a value here in our inner array, we have an array that represents how many times each line in this file was executed while coverage was running. Order is important in this array. The first item at index zero is how many times line one of this file was run. In fact, lines one through six were all executed one time over the course of the first ten shows. Line seven is a special case: the nil here doesn’t mean that it wasn’t executed; it would be zero in that case. Coverage is telling us that there’s something special about this line, which it considers irrelevant.

00:05:17.360 Let’s open up this string file and see what we’re talking about. Line seven is an empty line, so if you looked at the results of coverage and saw a zero, you might take that to mean there’s a problem or at least something to investigate. Maybe you have some dead code you can remove or perhaps you have a bug in your code where you aren’t running certain lines that you should be. But really, this is just an empty line, and what does it even mean to run an empty line, a comment, or an end statement? I don’t know, and coverage doesn’t take a position.

00:06:17.360 Instead, it provides semantic meaning to nil to tell you that you don’t have to worry about this line. If we go back to our problem statement: we’re trying to determine how many guitar strings we broke so we can figure out the number of times each line in this file was executed. If we go back to our implementation, we’ll consider a string broken when it reports itself as broken, which is on line 54 of this file.

00:06:41.680 But remember, arrays are zero-indexed, so to get the 54th line, we actually need to ask for index 53. As a result, we find that we broke 16 strings over the course of the first ten shows. If you believe in the statistical significance of that, we can then place an order for the rest of the tour, and the band can confidently play the remaining shows, knowing that they have the equipment they need.

00:07:41.960 Next, we’re going to move into one-shot coverage. For one-shot, we’re going to help the band prepare for one very specific show. They’ve been booked to play a festival, which is great news because they’re being introduced to a larger audience that they regularly do not play to. The money’s pretty good, and they actually play a shorter set, so they make more money for less work. However, it does come with a unique set of constraints.

00:08:06.720 There’s a very tight schedule, with many bands on the bill, so they need to make sure they get on and off the stage quickly. They’re trying to evaluate some ways to pare down the amount of equipment on stage to reduce the time it takes to get on and off. One of the items the band uses a lot are synthesizers. Multiple members of the band play synthesizers. Even if you're playing another instrument, you’re probably responsible for also playing the synthesizer. Band members need to stretch out across the stage to reach all the synthesizers they need to play at one time.

00:09:05.680 Here’s the drummer, not playing the drums, but instead playing the synthesizer. These instruments are scattered across the stage, and they’re trying to determine if they can get away with not using so many of them for this one particular show. As the name implies, a synthesizer synthesizes different sounds, and you can store those sounds in patches, which act as presets that you can recall quickly.

00:09:53.360 Having those patches being able to recall really quickly is crucial because when you play a note on the synthesizer, you want to make sure you have the right patch set beforehand. If you have the wrong patch set while playing the correct note musically, it won’t sound like the song at all. Each synthesizer implements how they store these patches differently.

00:10:35.680 Let’s look at one example manufacturer. When you want to read the patch memory where these patches are stored, they use a case statement. This synthesizer has four buttons for presets. When you press each button, it accesses different positions in memory. To be sure of which patches are in use for this one show, the thought is: if we can figure out which patches we are using and not using, maybe we can consolidate.

00:11:05.360 We’re going to take advantage of the fact that the band has to practice this set because, as mentioned, they’re playing a shorter show, so they’re cutting certain songs out. They want to ensure that the transitions sound right. When they practice that show, we’ll set up the stage just like we do normally, where each synthesizer will have all of the patches we use. We’ll have the band practice the show.

00:11:51.360 As you might imagine, we’re going to do this using coverage. We’re going to explicitly ask coverage to start in one-shot lines mode, and the band will play the setlist to see what we get back. The output looks very similar to lines coverage, but let’s dig into the details. We receive a hash, whose keys are again the names of files executed while coverage is running. The value is another hash where the key is the mode of coverage, followed by an array that tells us whether or not each line was executed.

00:12:59.080 In line coverage, this array tells us how many times each line was executed, with a specific order. In one-shot lines mode, it only tells us if a line was executed or not and doesn’t provide how many times, so we can find all the lines and their numbers. In our implementation of reading memory addresses, we have this case statement, and we want to identify which patches we accessed.

00:14:03.760 If we keep track of these line numbers, we can take the output from coverage and check which line numbers are included in that array. As a result, we find out that we’re using three of the four patches on the synthesizer. This might either be good news or bad news; it’s not a great candidate for consolidation, but we’re using it a lot, so we probably need it.

00:14:59.760 If we perform this exercise on every synthesizer on stage, we might find one that we’re only using one of the patches for, and as long as we don’t need to use both of those instruments simultaneously, we may get away with moving the patch onto another synthesizer, allowing us to avoid bringing that synth on stage altogether.

00:15:55.600 I recognize that if you're a synthesizer person, you might be screaming that two of the same patches will sound radically different on different synthesizers. While I appreciate that, the band is simply looking for a reasonable facsimile for this one show. Thanks to one-shot coverage being able to tell us which lines we accessed or not, we were able to consolidate the amount of equipment we needed, getting everything on stage that we did not need while ensuring the band could still perform the festival setlist.

00:16:53.680 We now move on to the second half of our set. We will talk about methods coverage and help the lighting team, which has a bit of a problem. They have a spotlight right on the lead singer, and for the last couple of shows, that spotlight has been going out when it shouldn’t be. That would be a problem for any light, but it’s particularly troublesome when it’s front and center on the stage with a perfectionist lead singer.

00:17:50.400 It’s in their best interest to figure out the problem quickly. If you thought that the band used a lot of synthesizers, just wait until you see their light show; they are well-known for the lighting rigs they bring on tour. They contract with special effects firms and lighting teams to design lights that didn’t exist before just for their concerts.

00:18:35.840 They have so many lights that they can’t even keep track of them, and they might be using a light for only one or two songs. As the tour progresses and songs change, there may be lights they’re not even utilizing, but they don’t realize it. One of the theories on the team is that perhaps the venues just don’t have enough power to support all the lights. If we could find some way to unplug some of them, maybe there’d be enough power for the lights we do need.

00:19:43.680 The lighting team operates like another set of performers on stage, although they don’t get credit. For every note of every song, they're managing the visual aesthetic and ensuring the stage's appearance matches the performance. They turn lights on and off, adjusting colors and effects to fit the required atmosphere.

00:20:52.320 To investigate the lighting issues for our concerts, we will conduct an experiment during the next show. We will set up everything normally—lighting all the lights, plugging everything in, having the band play the show, and then we’ll tear everything down. However, this time we will use our friend coverage to help out. We’ll use methods coverage because we want to know how often particular methods are executed, not individual lines.

00:21:49.920 We don't need to know how the details of what happens within the trigger method of the can light, moving light, or spotlight; we just need to ensure it’s called at all. We’ll have the band perform the show, and we’ll see what we get back. The output will look a little different from what we’ve seen, but let’s do the same previous exercise and break down one element in the hash we receive. The outer hash key will be the name of the file executed under coverage.

00:22:49.440 The value will be a hash where the key is the mode of coverage being reported, and the value is another hash containing identifying information about the method. The key of this innermost hash is the count of the number of times that the method ran while coverage was active. If we zoom in on this innermost hash, we can explore each element to understand better how often methods are being executed.

00:23:37.680 The first identifier describes what kind of branch this is. In branches coverage, each conditional gets a unique identifier. For example, here we have an if conditional with then and else, and this represents the then part of the conditional. The next numbers represent the starting and ending line and column numbers to help identify where in the file this method is located.

00:24:54.320 We’re checking for lights not being used, and we see that our projector was set up, but we never used it. The next time we roll into a town to light things up, we can simply keep the projector unplugged and only power the lights that we actually use. By using methods coverage, we can focus on whether an overall method was executed instead of worrying about which specific line numbers were called.

00:26:55.520 Now, let’s wrap this up with the closing number. For our last song together, we’re helping the band prepare for the final show of the tour where one of their friends, also a singer, has been invited on stage to play a song he chose from their repertoire. Since it’s not one they’ve been playing on this tour, they’ve been using soundcheck as a chance to practice so they don’t make fools of themselves.

00:27:52.720 By soundchecking in the early afternoon, they plug in their instruments, tune up, and play the song. During this practice, something doesn’t sound right, and they can’t quite figure out what’s wrong. They realize they need to analyze what's going on, and they decide to leverage the coverage module again.

00:28:27.840 Since they’re unsure what problem to solve, they opt to throw the kitchen sink at it. By passing the all symbol to coverage.start, they instruct coverage to give them everything it has. They will rehearse the song again and see what results come back from this comprehensive coverage.

00:29:25.360 Predictably, they get back a multitude of information. However, it’s crucial to appreciate why the data returned from coverage is structured this way. In previous examples, we started coverage with one mode, but you can run multiple coverage modes simultaneously. This is why a single file can have multiple keys representing different modes.

00:30:07.840 Looking for the types of coverage run on our song, we see lines coverage, methods coverage, branches coverage, but we’re missing one: one-shot lines. It turns out there's a technical limit in how coverage is implemented, meaning lines coverage and one-shot lines coverage cannot run at the same time. Lines coverage can tell you if a line was executed; you also know how many times it was executed.

00:30:55.680 So, faced with the option of using one of the other coverage results, we resolve to use lines coverage in this case. We want to check if there are any lines in our song that we’re not executing adequately and ensure we’re using all the relevant lines.

00:31:50.720 Since we confirm that we’re executing every relevant line, we can conclude that we’re calling every method as well. However, we capture the necessary information to ask. When exploring branches coverage, however, we encounter a zero, so we decide to delve into coverage results one last time.

00:32:49.440 We return to the hash structure. The outer hash's key will still be the file name under coverage, with values being other hashes that include coverage info. Delving deeper, we could see that the value is the count of how often a specific branch was executed while coverage ran. Following the data trail reveals that the identifier points to a particular kind of branch, revealing that we’ve never returned a zero back out from this if test.

00:33:45.680 This indicates an actual bug in our code, stemming from an off-by-one error. Our choruses have been zero-indexed when they should have been one-indexed. Once we change the number we input into the chorus method, running coverage again shows we now execute both the main branches of this conditional.

00:34:01.440 As a result of this adjustment, the song sounds the way it should! The band can confidently take the stage for the final show of the tour, inviting their friend on stage to perform a song that aligns perfectly with their set.

00:34:48.480 Despite all the challenges, the audience remains unaware of the intense effort that went into troubleshooting the sound. Thanks to branches coverage, they were able to identify that while lines coverage confirmed they were executing every line, it was branches coverage that highlighted the existence of critical paths within those lines that were missed.

00:35:06.720 This emphasizes how understanding coverage results can lead to crucial insights about your code. Let's take a break now before we move on to our final discussions and wrap up.

00:35:21.280 We’ve been discussing Ruby’s coverage module, which has four modes that help answer important questions about your code. If you want to determine how many times each line was executed, you can use lines coverage.

00:35:46.079 If you don’t care about how often something was executed, but just that it was executed, you can use one-shot lines coverage, which provides a performance benefit over standard line’s coverage. If you want to check on method execution rather than individual lines, methods coverage will help. If your aim is to explore your conditionals, you can use branches mode.

00:36:24.880 If you're interested in using coverage in your application but want some assistance, I can recommend a couple of gems. If you want to evaluate your test coverage, check out 'simplecov', and if you're interested in analyzing your production code, consider the 'coverband' gem. Both utilize Ruby's coverage module under the hood.

00:37:14.640 A detailed narrative description of coverage, as illustrated in this talk, is available on a GitHub page, with links to slide copies and a repository containing all code examples. You can also discover more about The Gnar Company if interested.

00:37:54.320 I’m also active on Twitter @kevin_j_m if you’d like to connect and discuss any questions you might have. Right after this, I’ll jump into the Slack channel for this talk: 'RC Talk: Enough Coverage to Beat the Band', and I invite any questions there.

00:38:37.440 We can talk about what concerts you've seen, which ones you had tickets for this year but couldn’t attend, or discuss coverage. Thank you all for being here; you’ve been a great audience. I appreciate your time, and I hope you enjoyed the show!

00:39:18.880 Thank you so much, Kevin. Much appreciated. As Kevin said, please head on over to the Slack channel and ask any questions if you have them, giving them some well-deserved kudos. Join them on stage, virtually speaking, of course!

00:39:35.040 Before you go, we’re still looking for a handful of folks to participate in sharing their stories or experiences. Don’t forget; we’re still accepting submissions. Otherwise, thank you again, and we’ll see you in the Slack channel.

RubyConf 2020