Software Engineering
Getting Unstuck: Using the Scientific Method for Debugging
Getting Unstuck: Using the Scientific Method for Debugging
by Caroline TaymorIn her talk at RubyConf 2017, Caroline Taymor discusses how to effectively debug using the scientific method. The presentation emphasizes transforming the debugging process into a scientific experiment to alleviate frustration when facing bugs. Key points include:
Introduction to the Scientific Method: Caroline explains the structure of the scientific method, which includes gathering knowledge, asking questions, forming hypotheses, designing experiments, and drawing conclusions. This method creates a systematic approach to debugging software-related issues.
Steps to Apply the Scientific Method in Debugging: The process starts with documenting everything related to the bug, brainstorming the existing knowledge, and identifying user-facing impacts. By encouraging documentation, the method helps clarify thoughts and phrases that may otherwise seem overwhelming.
Formulate Questions and Hypotheses: After gathering information, practitioners are advised to note uncertainties and form hypotheses about the bug’s potential causes. The importance of selecting a focused question for investigation is highlighted to streamline efforts.
Experimental Design: Caroline emphasizes the necessity of documenting the experiment thoroughly and the expected outcomes. By understanding what to observe during experiments, developers can better track their observations.
Iterative Learning: Following each experiment, developers should analyze the results, which may confirm or refute their hypotheses. This iterative learning helps narrow down potential causes and refine further investigations.
Sharing Findings: Caroline advocates for sharing findings to foster team collaboration and help others learn from past debugging experiences. Utilizing shared notes contributes to clearer team communications and promotes onboarding for new team members.
Personal Anecdote: Caroline shares a personal story about debugging a Node.js application that hung due to large data sets. By employing the scientific method, she was able to isolate the issue and implement a solution involving the correct use of the I/O library.
Conclusion: The scientific method is presented as a valuable tool for debugging; it provides a structured framework to regain composure and systematically approach problems. Caroline encourages the audience to adopt this method to enhance their debugging practices and emphasizes documenting, knowledge sharing, and iterative questioning as key components in this process.
00:00:10.660
Hi, I'm Caroline Taymor, and I'm here to talk to you about getting unstuck using the scientific method for debugging.
00:00:16.730
I'm a software engineer at Pivotal Cloud Foundry, and I want to share with you my favorite tools for debugging.
00:00:23.900
So, who here has ever gotten really stuck while debugging? I see a majority of hands, which is great because I think it’s a pretty universal experience.
00:00:29.119
If you haven't yet, you probably will someday—I know I certainly have.
00:00:35.629
Today, I want to share with you my favorite tool for dealing with those moments when you're really, really stuck.
00:00:41.150
It's the scientific method, which turns your debugging process into a scientific experiment.
00:00:48.670
Today, we're going to discuss the what, how, why, and when of the scientific method.
00:01:00.620
What is the scientific method? How do you use it for debugging? Why is it such a valuable tool? And when do you know to pull it out of your toolbox?
00:01:08.510
The scientific method is just a formal term for the process of doing science, which is really cool.
00:01:15.080
It starts with gathering knowledge: what is already known about your topic?
00:01:21.260
If you are researching stars or frogs, what research have other scientists already done?
00:01:27.170
Next, you start asking questions, which are about what we still don't know about the topic.
00:01:32.330
What is there still to learn? What did previous research find that is interesting and worth exploring?
00:01:39.830
Then, you formulate a hypothesis—an educated guess—about what you think might be the answer to your question.
00:01:45.850
In scientific research, you typically write your hypothesis in terms of a null hypothesis, suggesting there is no correlation between two variables.
00:01:53.390
Your goal is to disprove it and provide evidence that there is some statistical correlation.
00:02:00.620
Once you have your hypothesis, you design an experiment.
00:02:07.000
You need to decide what information is required to disprove your hypothesis and what data you can collect.
00:02:13.180
When designing your experiment, it is crucial to be very detailed; your research only holds weight if it can be replicated by others.
00:02:20.060
After designing your experiment, you run it, following your procedure, and you take thorough notes.
00:02:26.500
Taking good notes is where the scientific method shines as a tool for debugging.
00:02:32.769
After running your experiment and taking notes, you reach a conclusion.
00:02:39.230
Did what you observed support or disprove your hypothesis? Maybe it didn’t disprove it, which could lend some support for its validity.
00:02:44.980
Or perhaps you learned that your question was irrelevant or that your hypothesis was off target.
00:02:50.530
All of these outcomes are useful because they lead you back to the knowledge-gathering step.
00:02:56.170
Before you started, you knew less than you do now, even if that knowledge is simply realizing your research question wasn’t valid.
00:03:04.060
Also, you need to share your findings; what good is scientific research if it only remains in your lab or living room?
00:03:09.310
When conducting scientific research, this sharing often involves publishing results in a peer-reviewed journal.
00:03:16.750
Now, let’s talk about how to debug using the scientific method.
00:03:26.260
One of the first steps in this process is to write everything down.
00:03:31.450
The way you write it down can vary—sometimes I use a notebook, and sometimes I prefer to take notes in my text editor.
00:03:37.930
Using stickers on my notebook makes it fun, and a whiteboard is great for collaboration.
00:03:43.840
If I'm pair programming with someone who is skeptical about the scientific method, I might use sticky notes to map out my process.
00:03:50.080
My favorite way to jot down notes is directly in my text editor—it’s efficient and integrates organically with my debugging.
00:03:55.480
It makes it easy to copy information from the code or logs into my notes, and then share important pieces later.
00:04:02.410
While writing it down is essential, it’s more crucial that you find a way to externalize the knowledge floating in your head.
00:04:09.310
You might record your thoughts into a tape recorder and articulate your thoughts aloud.
00:04:16.750
Next, you begin with gathering existing knowledge.
00:04:22.260
When debugging software, this looks like brainstorming everything you know about the bug.
00:04:26.260
What is the user-facing impact of this bug? How did you notice it?
00:04:31.540
For example, perhaps a customer encountered a 500 error page and sent an email with insufficient information.
00:04:38.350
In every process, you’ll discover useful pieces. I often forget this step when I start debugging.
00:04:44.170
After working on a bug for an hour or even days, I might forget what I've already learned.
00:04:49.479
However, it is vital to write down everything you remember about the issue, including odd log entries or related strange behavior.
00:04:55.690
This process might feel like stream-of-consciousness, but writing it all down can really help.
00:05:02.620
As you begin documenting, questions will naturally emerge.
00:05:08.259
Initially, you will note what you know is happening, and then uncertainties will surface.
00:05:13.419
For example, you might think, “Is my service spamming the database, causing everything to crash?”
00:05:18.880
Questions like these are important to document.
00:05:23.979
You may have two or three questions or even a whole list of them.
00:05:30.289
Once you have a few questions, hopefully, one will stand out as interesting.
00:05:38.289
If you feel overwhelmed by many questions, just pick one to focus on.
00:05:44.909
From there, you can refine your approach.
00:05:50.050
Ask questions like, “How do I know what I believe is true? Are those statements actually true?”
00:05:57.230
The most valuable question you can ask may very well be, “Is what I think true actually true?”
00:06:05.950
Once you've selected a single question, form a hypothesis about it.
00:06:11.720
My approach to using the scientific method for debugging flexes a bit; I use it as a general framework rather than a rigid rule.
00:06:18.120
Sometimes, I might frame my hypothesis as a null hypothesis, but most of the time, I simply state what I think is happening.
00:06:24.890
It’s helpful to have more than one idea; don’t hesitate to pick a hypothesis arbitrarily.
00:06:31.620
Once you have a statement to test against, you can proceed to design your experiment.
00:06:38.600
It’s important to be detailed again—you may not need to submit this to a journal, but you will need to refer back to it later.
00:06:45.150
When designing your experiment, consider what information is needed to disprove your hypothesis.
00:06:51.050
You don’t have to fix the bug; just find evidence that invalidates your supposed cause.
00:06:57.710
As you design your experiment, document what you expect to see.
00:07:04.300
What behavior or log entries would confirm your hypothesis?
00:07:10.390
Then, you run the experiment, taking thorough notes on what you observe.
00:07:18.080
It’s important to track if you saw the expected outcomes and to note anything unexpected.
00:07:25.510
Documenting the surprises, especially the ones you didn’t think your software could do, is crucial.
00:07:32.460
I often discover unrelated bugs while following this process, and it’s a good practice to note them down for later.
00:07:39.860
Write out your findings in your bug tracker so you can avoid getting distracted.
00:07:45.420
One effective method during this note-taking phase is to grab annotated log snippets.
00:07:51.450
You may want to piece together logs with timestamps that show timing issues in the system.
00:07:57.390
For example, if a process times out, record the time it takes from the start to finish of a segment.
00:08:03.440
This helps track down irregularities and can remind you of why certain logs were interesting.
00:08:09.200
Returning to your findings later will help jog your memory about the issues encountered.
00:08:17.370
After these steps, you will come to a conclusion: was your hypothesis discredited?
00:08:23.600
Do you feel closer to understanding the cause of your bug, or are you still clueless?
00:08:31.350
If you have managed to deduce the problem, fantastic! Implement the fix.
00:08:38.780
But if you're still puzzled, that’s ok too—it means you've discarded many possibilities.
00:08:45.260
Earlier, you may have had thousands of potential cause, and now you're down to a manageable few.
00:08:52.600
If you must circle back to the knowledge-gathering phase, you might have new insights or questions.
00:08:59.390
You can refer back to your earlier set-aside questions and see which one may now seem more relevant.
00:09:06.840
And don't forget the sharing phase of what you learned during debugging.
00:09:12.960
There's a wealth of information that you can gather and pass along to your team.
00:09:19.500
Certainly, all those new bugs you’ve found, if recorded properly, can help others in the future.
00:09:26.130
If you realize the issue originated from a misconfiguration elsewhere, indicating it to your team can prevent future headaches.
00:09:34.570
Another valuable item to share is your experimental procedure, which can serve as a useful onboarding tool.
00:09:41.240
This can help newcomers understand how to troubleshoot bugs, especially if they are new to the codebase.
00:09:48.200
Now, I’d like to share a personal story about how this process has helped me with some interesting bugs.
00:09:54.710
I was working on a project where we processed customer input and utilized a Node.js code sandbox.
00:10:01.810
While our application was designed for quick responses, a customer reported a process that hung for nearly five hours.
00:10:10.260
I had a vague idea of where the issue may lie, but I couldn’t determine if it was related to the JavaScript running in the sandbox.
00:10:17.340
Questions arose about possible issues in our logging since we couldn't pinpoint where the hang occurred.
00:10:24.890
I proposed a procedure where we could replicate the environment as closely to the customer's as possible.
00:10:30.960
After adding log lines to track progress through the code's lifecycle, I restarted the server.
00:10:37.080
To my surprise, the JavaScript code didn't hang—meaning my initial hypothesis was incorrect.
00:10:44.130
This discovery shifted my focus to whether the data might be too large.
00:10:51.960
If our data set was much larger than our testing data, perhaps it was causing the hang.
00:10:59.000
We decided to test it with a smaller data set, which confirmed my suspicion: it worked.
00:11:05.260
From there, we returned to gathering knowledge and found that Node.js had a buffer limit that our customer exceeded.
00:11:12.510
We resolved the issue by properly utilizing the I/O library to handle data flow.
00:11:18.600
This example demonstrates how your first question might lead you towards understanding through unexpected avenues.
00:11:25.890
The scientific method is particularly beneficial for debugging because it fosters collaboration and helps you get unstuck.
00:11:34.700
Once you’re unstuck, it enables you to keep progressing.
00:11:41.000
Collaboration occurs when you and your teammates are on the same page, particularly during frustrating times.
00:11:48.080
If you engage with teammates about a bug, your notes act as an invaluable resource for onboarding them on the issue.
00:11:54.500
They can quickly access your relevant findings without needing to understand the entire history of that bug.
00:12:01.400
This method also helps shift focus during discussions of cause and effect, simplifying communication.
00:12:08.150
By agreeing on a single hypothesis, you facilitate sound communication and investigation.
00:12:14.840
In situations where disagreements occur, focusing on a single question can enhance collaboration.
00:12:22.080
Writing down your thought process also accelerates the un-sticking mechanism.
00:12:28.990
By documenting your thoughts, you can mitigate feelings of being overwhelmed and regain focus.
00:12:36.600
When your thoughts swirl without structure, they can be daunting—writing clarifies them.
00:12:43.500
Through externalization, you gain enough emotional distance to discern which thoughts are truly impactful.
00:12:50.130
You may even find that thoughts you hadn't considered become pivotal to your debugging.
00:12:57.110
Additionally, questioning how to disprove your hypothesis can provide actionable steps amid chaos.
00:13:06.840
Flipping the overwhelming bug into a more focused question allows you to identify clearer actions.
00:13:13.740
Indeed, moving forward can happen quicker once you regain momentum after feeling stuck.
00:13:21.040
Once you have a direction after finding a solution, you can persist in seeking answers.
00:13:27.840
The scientific method helps sidestep the repeated questioning of assumptions, streamlining your investigation process.
00:13:34.950
When asking for help, your notes will clarify what has already been tried.
00:13:41.400
Everyone seeks clarity on the most recent developments and precious context you honor through note-taking.
00:13:48.070
Your thorough notes from prior exploration serve as a roadmap to where you're currently navigating.
00:13:54.160
It's common to stumble into previous approaches when faced with complex bugs—having a record helps prevent wasted effort.
00:14:01.690
Your notes also reveal patterns, making it easier to recognize and avoid repeating the same attempts.
00:14:08.580
Going through multiple iterations with a structured framework provides a sense of how much you've progressed.
00:14:15.090
While you may still be wrestling with several options, you’ll notice that you're systematically narrowing them down.
00:14:21.210
So, when should you use the scientific method for debugging?
00:14:30.060
Let me share an experience that crystallized this approach for me.
00:14:36.490
I was involved with a project that required packaging a Ruby on Rails app as a VM image for different infrastructures.
00:14:43.960
Each had to be configured differently to suit its respective environment.
00:14:50.290
We originally used a standard Ubuntu image but intended to swap it for one with extensive security hardening.
00:14:58.240
Our general procedure was to simply change the URL in our packaging tool and build it.
00:15:05.350
However, it turned out that it didn’t work as we hoped, and we spent a month troubleshooting.
00:15:11.780
The whole team felt incredibly stuck and frustrated.
00:15:17.530
By documenting our frustrations and experimenting, we focused better on finding solutions.
00:15:25.050
I realized that there are two types of ‘stuckness’—one being overwhelmed by too much information.
00:15:32.610
The second is the opposite, where you feel stalled with too little knowledge.
00:15:38.420
The scientific method has tools that can aid you through both of these states.
00:15:46.370
When you feel you have too little information, writing everything down can help concretize your understanding.
00:15:54.480
When there's too much information, framing a single question allows you to direct your focus.
00:16:01.090
Thus, the scientific method serves you throughout the debugging process.
00:16:08.070
We’ve talked about what the process of doing science is, how to apply it to debugging, and why it’s beneficial.
00:16:14.550
I hope the next time you find yourself stuck on a bug, you engage with the scientific method.
00:16:21.420
Remember to gather knowledge, ask probing questions, create hypotheses, and run experiments.
00:16:28.090
Take meticulous notes, draw conclusions, revisit your knowledge, and communicate with your colleagues.
00:16:34.530
I also have a reference guide on my website for debugging using this method.
00:16:40.800
I’m not a huge fan of slides, as they may lack fidelity; this can be a better reference for you.
00:16:47.950
I would love to hear from you about how the scientific method has helped your debugging process.
00:16:57.800
Thank you! I believe we have a few minutes for questions.
00:17:07.180
So, how do you determine when to stop working on a bug?
00:17:12.590
That’s a tough question, and it involves judgment calls.
00:17:21.000
I wouldn’t advise spending too long before consulting your colleagues, especially after several hours.
00:17:27.730
At my team, we usually have check-ins if a bug takes longer than two days.
00:17:34.250
How about estimating how long it’ll take to resolve a bug?
00:17:40.700
I usually only develop a working framework for our conversations about priorities.
00:17:46.130
If a bug seems to stretch longer, it’s key to pull in teammates for support.
00:17:53.030
But this is an evolving conversation depending on team culture.
00:17:56.880
If there are no further questions, thank you very much!
