Dispelling the dark magic: Inside a Ruby debugger

by Daniel Azuma

In the talk "Dispelling the Dark Magic: Inside a Ruby Debugger," Daniel Azuma addresses the perceived complexity of using debuggers in programming, particularly Ruby. He likens his initial fear of debuggers to the awe experienced during a total eclipse, emphasizing that a lack of understanding can lead to intimidation. Azuma has primarily relied on simpler methods of debugging, such as using 'puts' statements, but aims to demystify debuggers for others who feel similarly.

The presentation covers several key points about debugging in Ruby:

Understanding Debuggers: Azuma introduces the concept of a debugger, reassuring the audience that it operates on fundamental principles rather than mystical mechanisms. He emphasizes that implementing a basic debugger is straightforward and not as complicated as it may seem.
Implementing a Simple Debugger: Throughout the talk, Azuma demonstrates how to create a simple Ruby debugger using the TracePoint API, which has been part of Ruby since version 2.0. He outlines how this API allows developers to listen for events happening at the Ruby VM level, such as method calls or line execution.
Key Features: The implementation focuses on key debugging functionalities:
- Breakpoints: Setting breakpoints in the program to pause execution and inspect states.
- Using IRB: Incorporating Interactive Ruby (IRB) to allow inspection and manipulation of program state when hitting breakpoints.
- Stepping and Step Over: Implementing commands for executing code line by line, allowing for detailed observation of program behavior.
Advanced Debugging Techniques: Azuma shares insights into production-level debuggers like Stackdriver Debugger, which need to operate without halting user interactions. He discusses challenges such as managing multiple threads and performance optimization.
Addressing Side Effects: It’s crucial to avoid altering the program state while debugging. The approach should detect and prevent unintended changes through a side-effect detector. This includes being cautious of Ruby bytecode operations that might affect object states.

In conclusion, Azuma’s talk reassures developers that debugging does not have to be an intimidating experience. With the right understanding and tools, it can become an integral and manageable part of programming. He encourages experimenting with existing debugging tools in Ruby, such as the default Rails debugger and Stackdriver Debugger, available on GitHub. The presentation highlights that debugging is a skill that can be learned and perfected, ultimately reinforcing the idea that it isn’t dark magic, but a logical and systematic process.

Overall, the session empowers attendees to feel more confident in utilizing debuggers in their Ruby development efforts.

00:00:10.630 All right, so this is 'Dispelling the Dark Magic: Inside a Ruby Debugger.' My name is Daniel, and I watched the eclipse this summer. Who here traveled to see the total eclipse? Okay, fair amount of hands. I really felt like it was worth it. It's a great experience! I packed up my family and we drove a few hours down south from where we live in Seattle to Central Oregon. We converged on a small town in Oregon called Madras, Oregon, along with pretty much the entire West Coast. I think it was an epic traffic jam getting out of there, but the eclipse was really awesome. This was a photo that I took right at the beginning of totality. It was a surreal and amazing experience. I remember looking up and thinking, it's a weird feeling—it's like your body is telling you that it's not supposed to look like that; like there's something wrong. You're seeing a special effect or something, and it's just this eerie experience.

00:00:41.390 I remember thinking that I was starting to understand the reaction of our pre-scientific, pre-industrial ancestors. They might have concluded that oh man, the world is about to end or that some sort of dark magic is involved here. The only thing that kept me from having that same reaction was the knowledge of what was going on—a little understanding of the inner workings of what it was that I was seeing. I realized that as a developer, I often have a similar response to some of my tools, particularly debuggers. I've been developing professionally since the late nineties, and throughout that time, I've been very hesitant to use debuggers. They felt kind of eerie and spooky. These tools were doing something bizarre to my program, stopping it from running and causing the machine to do something else. It felt intimidating and uncomfortable.

00:01:21.020 I spent pretty much my entire career debugging and troubleshooting using 'puts' or 'printf,' or whatever the call happened to be in the languages I was using. I was a 'puts' debugger. I'm not alone in that; there was an article written by one of our Ruby luminaries, a member of the core team, called 'Puts Debugger.' It turns out debuggers really aren't as spooky as they may seem at first. To prove that, we're going to spend this hour looking under the hood at a debugger. We'll see how it works, what it does, and we'll discover that it's not just dark magic; it's just Ruby under the hood, and it's actually not that complicated.

00:02:11.900 In fact, we're going to implement a fully featured debugger right here! Implementing a debugger is not that hard; it just takes a few minutes. Afterwards, if we have time, we'll look at a real-life production debugger, some of the techniques that we use to implement it, and explore some of the debugging facilities of the Ruby VM. It'll be fun! Before I get started, a little bit about myself: as I said, my name is Daniel. I've been developing since the late nineties, started with Ruby around 2005, right as Rails was starting to enter the picture. I joined a series of Rails startups, and about four and a half years ago, I switched gears and joined a small company.

00:03:01.950 I became part of the cloud platform team, working on things to help Ruby developers use the cloud platform. It's been a really good gig, and I've really enjoyed it. However, as I work at a small company, there is a bit of small company bureaucracy that I have to get through. So, all the code you see here during the live coding sessions is copyrighted by the small company, and it's Apache 2 licensed. Now, with that out of the way, let's write a debugger! We’ll create something very straightforward. We'll create a library that allows you to set breakpoints, open a command shell debugger, inspect the program states, and step through the execution of your program—kind of the basic functionality you'd expect in a debugger.

00:03:45.650 So, let me switch over to my editor here and take a quick look at a sample toy program that we’ll use to demo our debugger features. This is a simple program that prints out a hello message to the sender and the recipients, which we pass on the command line. We set an instance variable for the recipients, and there’s a method that prints out a hello message, as well as another method that prints it out twice—because why not? Here’s how that actually works. Hello, world from RubyConf! That was fun. Now, let's go write a debugger! We're going to start by managing breakpoints, so we’ll add breakpoints to a program and store them in a struct, with each breakpoint having a name and the file and line number it's in. All the methods in this class will just be class methods for simplicity's sake.

00:05:58.140 We'll create a method to add a breakpoint and store all the breakpoints in an array. We need an initialization method to initialize that to an array of breakpoints. We'll make sure that this initialization gets called when we require our file. Now we have a simple library that lets you add breakpoints to an array. Let's go back to our toy program and require that file, and we'll set a breakpoint. We’ll give it a name, and we’ll set the breakpoints in this file, placing it on line 11. All right, we’ve added a breakpoint to our program, so let's see what happens. Nothing happens because we added the breakpoints to an array, but we didn’t actually do anything with it. We didn’t have any code that breaks at that breakpoint, so how do you interrupt a running Ruby program to implement a breakpoint?

00:07:36.960 One thing you can do is use a powerful class in the Ruby core library called TracePoint. TracePoints have been part of Ruby since version 2.0. It's a Ruby class that lets you listen for events happening at the Ruby Virtual Machine level. You can register callbacks that will be called whenever those events take place. For example, this method listens for method call events. Whenever a method is called, it executes this block that prints out a little message. TracePoint also passes objects to the block, which includes information about the event that just happened, such as the name of the method. A number of events are supported, including method calls, returns from methods, exceptions being raised, and threads starting and ending. There’s even an event for moving to the next line in your Ruby application, and it’s these events that we’ll use to implement breakpoints.

00:09:12.890 Each time we move to a new line in the program, we’ll register this callback that searches through our breakpoints. So let’s go back to our code and implement that. We’ll create a breakpoint via TracePoint. I often forget that when I practice this. We’re going to trace the line events. Each time we move to a new line, we'll search for breakpoints that match the file and line number. If we find a matching breakpoint, we'll print out a message indicating that we've found the breakpoint, providing the name of the breakpoint as well as additional information available in the TracePoint object about the event.

00:09:56.550 Let’s check if that works. We remember we set our breakpoints at line 11, and it looks like we hit that breakpoint. We hit 'my breakpoint' in the method hello on line 11. Now that we can detect breakpoints, once we know we’ve hit a breakpoint, what do we do next? Many debuggers provide a command-line interface that lets you interact with the program at that point. You can query the program to see its state and get information about how the program is operating. If we want to add a command line to our debugger, an easy way is to use IRB (Interactive Ruby). Many of you are probably familiar with IRB as the Ruby REPL; you can invoke it from your shell, run Ruby code, and see the output.

00:11:39.630 IRB is also part of the Ruby standard library, and you can call it from within your Ruby program. We're going to use another method on the TracePoint’s binding, which returns a binding object that provides a bunch of context information about where you are in your program and what the state is. When you require IRB, it adds an IRB method to the binding object, opening an IRB shell using that binding as the context. That's what we’ll do when we hit a breakpoint. Let's try that—looking back at our program after hitting the breakpoint, we get our IRB shell.

00:11:58.430 Here's where things get interesting: we have an IRB shell open with the current context on line 11. We can access the local variable 'recipient,' which is in scope, and see that its value is 'world.' Since we are in the context of our object, we also have access to member variables like 'sender,' and we can see its value too. We can call methods, for example, calling 'hello' to see what it does. Moreover, we can change the variable values, altering the program state. Let’s change 'recipient' to 'your lines' and also change 'sender.' After that, when we continue the execution of the program by just exiting out of our IRB shell, the program will continue with that modified state. We now have new output for the program.

00:13:03.520 That's pretty much it; those are the basics of our debugger. Real production debuggers will often include additional features, and since we have some time, let's try implementing a few of those. We noticed that we continued program execution using the IRB exit command, so let’s enhance that. Normally, a Ruby debugger shell would use a command like 'continue' to resume execution. So, if we wanted to implement that as a command in IRB, we need to add methods to a module that IRB extends. Any methods present in this module are added to IRB as commands. We’ll create a command called 'continue.'

00:14:48.960 To implement 'continue,' we’ll defer this back to our mini debug module and pass in an object called IRB context. This object will be available to the IRB commands and will let us interact with the IRB session. Once we go back to the mini debug, we implement 'continue' by making the IRB context exit. Now, when we return to our debugger, we should be able to use the 'continue' command to resume the program execution—and there we go! What else can we do? Let's implement stepping. Stepping means that after we hit a breakpoint, we can execute one line of the program and then return to our shell.

00:15:32.740 In our example, we break at line 11; stepping allows us to execute that line, and then we can check the effects it had. This process can be repeated, allowing us to navigate through our program line by line and observe how its state evolves over time. We already have something that lets us do this line by line: our TracePoint. Let’s modify this to implement stepping. Instead of opening the IRB shell only when we hit a breakpoint, we’ll also open that shell if we're in stepping mode. We’ll create a 'stepping' mode and initialize it to 'false.' When we hit a breakpoint, we can set it to 'true.' As long as we're in stepping mode, we’ll open our debugger shell each time we hit a line.

00:17:03.490 Now that 'continue' means you want to resume your program and stop stepping, we will turn off the stepping mode there. We've added stepping mode functionality. Next, we need to create an IRB command for it and implement it. Stepping just involves exiting out of IRB, but unlike 'continue,' stepping does not turn stepping mode off—so we leave it on so the next time we hit our line, the TracePoints will open the debugger shell again. Let’s try that out. Once we break on line 11, we can now step, execute a line, and see we are on line 7, which indicates that we called a method in the process. If we step again, we're now at line 12.

00:19:04.240 Using 'step,' we continue executing lines sequentially and eventually we reach the end of our program. Notice that when we stepped the first time, we entered the implementation of the 'hello' method—that's called a step in. Stepping into calls can be useful, but sometimes you want a variation called 'step over.' Stepping over means that if you make a method call, you’ll step over the call and be brought to the next line of the current method. For instance, from line 11, if you step in, you call 'puts,' which takes you to line 7. If you step over, you'll be right back at line 12 instead.

00:20:26.640 To implement this, we need to keep track of method calls and returns so we know when we're back on our original method. The TracePoints are again the key here. We'll keep track of our current stack depth by incrementing it each time we hit a method call and decrementing it when we return from that method. So, whenever we need to implement 'step over,' we first check our current stack depth before deciding whether to open the shell again. Let’s try this out! We broke on line 11; now let’s step over, and we are successfully taken to line 12. Now that we have stepping and step over functioning, we could also implement 'step out.' This would mean that you don’t break into your debugger shell until you've exited the current method and returned one stack level beyond where you started.

00:21:55.540 Implementing 'step out' would leverage the same mechanisms we used to track our stack depth, but since we don't have time to cover it today, I’ll just mention it as another useful feature. Overall, we now have a working debugger: we can set breakpoints, open a command shell, and interactively examine our program while stepping through execution. It’s a usable debugger, and it’s roughly only 60 lines of code or so, which is quite manageable. Now, let's go a little deeper. My team at the small company implemented a Ruby version of a product we have called Stackdriver. Stackdriver Debugger operates differently from the one we just built; it’s designed for live web applications, particularly when users are involved. If there's a user waiting for a response, you can't set a breakpoint and halt everything.

00:23:54.700 You must be careful not to break the user experience. However, it would be beneficial to set breakpoints, interrogate program states, and analyze behaviors without the need for redeploying. That is the goal of Stackdriver Debugger; it’s been running an internal version for some time and was recently released for cloud customers. As a production debugger, it employs some advanced techniques to challenge encountered in typical web applications. One significant issue is thread management. In a Ruby web app, there may be multiple threads active, each handling separate requests asynchronously. So when in a debugging session, you often want to isolate your analysis to one thread, to avoid interference from other requests.

00:25:53.480 Unfortunately, the Ruby TracePoint API applies globally across all threads. To solve this, we have to drop down into C, where a C API exists for TracePoints that does support thread scopes. However, you can't yet access this from pure Ruby, necessitating a C extension for this purpose. Most of our trace points in the Stackdriver Ruby Debugger operate under this technique. This scenario presents an example of several advanced Ruby VM features that are exposed in C but not yet in Ruby. Given how beneficial this feature can be for debuggers, it might be an idea to expose it at the Ruby API level. The second issue is performance. In our mini debugger, we used line tracing to detect breakpoints, which means we're executing extra Ruby code for every line of code in our program; this could lead to significant performance hits.

00:27:28.310 To avoid harming performance while monitoring your program in a debug environment, we've incorporated multiple optimizations. One such optimization involves selectively turning line tracing on and off based on how close we are to breakpoints. For instance, we can listen to particular events that indicate file or method transitions and then trigger checks querying whether a breakpoint exists nearby before enabling our line tracer. While this may come off as a bit of a hack from an ideal perspective, we'd want native breakpoint support directly in the Ruby VM. In fact, Ruby has an experimental C API for line tracing that illustrates this logic, but the way we manage our breakpoints did not fit perfectly in that design, hence we resorted to using line trace points in our implementation.

00:29:11.550 Designing a quality API for breakpoints is challenging, considering that numerous different use cases will define a robust debugging experience. Ideas of side effects arise when using Stackdriver for observing live web applications. It's vitally important to prevent any changes to state while the application is being analyzed. As a protective measure, we developed a side-effect detector as part of the debugger. It's not perfect but serves to safeguard against modifications while debugging. Side effects in Ruby can typically be categorized into two groups: object changes (like instance variable value changes) and external side effects, such as writing to a database or sending data over a network connection.

00:31:07.820 We can detect these side effects by examining Ruby bytecode. Whenever we execute Ruby code at a breakpoint, we compile and analyze the bytecode. If we encounter any operations that change state—like assigning an instance variable—we trigger an exception to prevent executing that code. Methods, especially compiled C methods, are another common source of side effects. We whitelist C methods known to be safe for execution, while blocking others. This approach is inherently conservative, carefully monitoring for side effects. The need to prevent side effects may sound niche or odd, but it can be applied conceptually to the concept of immutability, prevalent in functional programming languages.

00:32:53.800 In the context of debugging and Ruby, while traditional object-oriented approaches recommend mutability, immutability presents a clear rationale in safety and performance benefits. Applying a method of immutability could help avoid issues during debugging sessions to safeguard against unintentional modifications that compromise user integrity and performance metrics. As we wind down, let's recap what we've covered today. At a basic level, debuggers are straightforward: you set breakpoints and observe behaviors, and in some cases, you can alter aspects before stepping through execution to see how a program's state evolves as it runs. We implemented a simple but fully featured Ruby debugger during this session, demonstrating how achievable it is.

00:34:43.930 We also discussed Stackdriver Debugger and some techniques used in its implementation, touching on challenges faced along the way. Ruby offers some support for debugging capabilities via useful APIs optimal for debuggers, yet possibilities remain for enhancements. If you'd like to experiment with debuggers yourself, I recommend checking out the default Rails debugger, as well as the Stackdriver Debugger. Both are open-source on GitHub, with core components undergoing similar principle designs based on TracePoints, as we explored throughout our time here together. Hopefully, all this insight enables you to feel less intimidated by debugging and dispels any myths around the complexity of debugging in Ruby. It's not dark magic; it's simply Ruby.

00:36:34.660 Thank you for coming!

00:36:51.619 I don't know if we have time for questions, but I have a couple of minutes if there are any.

00:37:04.069 The question is: when we stepped into the hello method and stepped over the 'puts' method, why didn't it step into 'puts?' As I understand, 'puts' is actually a C method, and you'd need a different trace point—the C call trace point— to detect that, rather than line trace points. Debugging C methods is challenging, so we did not implement that functionality here.

00:37:31.509 Thank you!