00:00:10.660
Hi, I'm Caroline Taymor, and I'm here to talk to you about getting unstuck using the scientific method for debugging.
00:00:16.730
I'm a software engineer at Pivotal Cloud Foundry, and I want to share with you my favorite tools for debugging.
00:00:23.900
So, who here has ever gotten really stuck while debugging? I see a majority of hands, which is great because I think it’s a pretty universal experience.
00:00:29.119
If you haven't yet, you probably will someday—I know I certainly have.
00:00:35.629
Today, I want to share with you my favorite tool for dealing with those moments when you're really, really stuck.
00:00:41.150
It's the scientific method, which turns your debugging process into a scientific experiment.
00:00:48.670
Today, we're going to discuss the what, how, why, and when of the scientific method.
00:01:00.620
What is the scientific method? How do you use it for debugging? Why is it such a valuable tool? And when do you know to pull it out of your toolbox?
00:01:08.510
The scientific method is just a formal term for the process of doing science, which is really cool.
00:01:15.080
It starts with gathering knowledge: what is already known about your topic?
00:01:21.260
If you are researching stars or frogs, what research have other scientists already done?
00:01:27.170
Next, you start asking questions, which are about what we still don't know about the topic.
00:01:32.330
What is there still to learn? What did previous research find that is interesting and worth exploring?
00:01:39.830
Then, you formulate a hypothesis—an educated guess—about what you think might be the answer to your question.
00:01:45.850
In scientific research, you typically write your hypothesis in terms of a null hypothesis, suggesting there is no correlation between two variables.
00:01:53.390
Your goal is to disprove it and provide evidence that there is some statistical correlation.
00:02:00.620
Once you have your hypothesis, you design an experiment.
00:02:07.000
You need to decide what information is required to disprove your hypothesis and what data you can collect.
00:02:13.180
When designing your experiment, it is crucial to be very detailed; your research only holds weight if it can be replicated by others.
00:02:20.060
After designing your experiment, you run it, following your procedure, and you take thorough notes.
00:02:26.500
Taking good notes is where the scientific method shines as a tool for debugging.
00:02:32.769
After running your experiment and taking notes, you reach a conclusion.
00:02:39.230
Did what you observed support or disprove your hypothesis? Maybe it didn’t disprove it, which could lend some support for its validity.
00:02:44.980
Or perhaps you learned that your question was irrelevant or that your hypothesis was off target.
00:02:50.530
All of these outcomes are useful because they lead you back to the knowledge-gathering step.
00:02:56.170
Before you started, you knew less than you do now, even if that knowledge is simply realizing your research question wasn’t valid.
00:03:04.060
Also, you need to share your findings; what good is scientific research if it only remains in your lab or living room?
00:03:09.310
When conducting scientific research, this sharing often involves publishing results in a peer-reviewed journal.
00:03:16.750
Now, let’s talk about how to debug using the scientific method.
00:03:26.260
One of the first steps in this process is to write everything down.
00:03:31.450
The way you write it down can vary—sometimes I use a notebook, and sometimes I prefer to take notes in my text editor.
00:03:37.930
Using stickers on my notebook makes it fun, and a whiteboard is great for collaboration.
00:03:43.840
If I'm pair programming with someone who is skeptical about the scientific method, I might use sticky notes to map out my process.
00:03:50.080
My favorite way to jot down notes is directly in my text editor—it’s efficient and integrates organically with my debugging.
00:03:55.480
It makes it easy to copy information from the code or logs into my notes, and then share important pieces later.
00:04:02.410
While writing it down is essential, it’s more crucial that you find a way to externalize the knowledge floating in your head.
00:04:09.310
You might record your thoughts into a tape recorder and articulate your thoughts aloud.
00:04:16.750
Next, you begin with gathering existing knowledge.
00:04:22.260
When debugging software, this looks like brainstorming everything you know about the bug.
00:04:26.260
What is the user-facing impact of this bug? How did you notice it?
00:04:31.540
For example, perhaps a customer encountered a 500 error page and sent an email with insufficient information.
00:04:38.350
In every process, you’ll discover useful pieces. I often forget this step when I start debugging.
00:04:44.170
After working on a bug for an hour or even days, I might forget what I've already learned.
00:04:49.479
However, it is vital to write down everything you remember about the issue, including odd log entries or related strange behavior.
00:04:55.690
This process might feel like stream-of-consciousness, but writing it all down can really help.
00:05:02.620
As you begin documenting, questions will naturally emerge.
00:05:08.259
Initially, you will note what you know is happening, and then uncertainties will surface.
00:05:13.419
For example, you might think, “Is my service spamming the database, causing everything to crash?”
00:05:18.880
Questions like these are important to document.
00:05:23.979
You may have two or three questions or even a whole list of them.
00:05:30.289
Once you have a few questions, hopefully, one will stand out as interesting.
00:05:38.289
If you feel overwhelmed by many questions, just pick one to focus on.
00:05:44.909
From there, you can refine your approach.
00:05:50.050
Ask questions like, “How do I know what I believe is true? Are those statements actually true?”
00:05:57.230
The most valuable question you can ask may very well be, “Is what I think true actually true?”
00:06:05.950
Once you've selected a single question, form a hypothesis about it.
00:06:11.720
My approach to using the scientific method for debugging flexes a bit; I use it as a general framework rather than a rigid rule.
00:06:18.120
Sometimes, I might frame my hypothesis as a null hypothesis, but most of the time, I simply state what I think is happening.
00:06:24.890
It’s helpful to have more than one idea; don’t hesitate to pick a hypothesis arbitrarily.
00:06:31.620
Once you have a statement to test against, you can proceed to design your experiment.
00:06:38.600
It’s important to be detailed again—you may not need to submit this to a journal, but you will need to refer back to it later.
00:06:45.150
When designing your experiment, consider what information is needed to disprove your hypothesis.
00:06:51.050
You don’t have to fix the bug; just find evidence that invalidates your supposed cause.
00:06:57.710
As you design your experiment, document what you expect to see.
00:07:04.300
What behavior or log entries would confirm your hypothesis?
00:07:10.390
Then, you run the experiment, taking thorough notes on what you observe.
00:07:18.080
It’s important to track if you saw the expected outcomes and to note anything unexpected.
00:07:25.510
Documenting the surprises, especially the ones you didn’t think your software could do, is crucial.
00:07:32.460
I often discover unrelated bugs while following this process, and it’s a good practice to note them down for later.
00:07:39.860
Write out your findings in your bug tracker so you can avoid getting distracted.
00:07:45.420
One effective method during this note-taking phase is to grab annotated log snippets.
00:07:51.450
You may want to piece together logs with timestamps that show timing issues in the system.
00:07:57.390
For example, if a process times out, record the time it takes from the start to finish of a segment.
00:08:03.440
This helps track down irregularities and can remind you of why certain logs were interesting.
00:08:09.200
Returning to your findings later will help jog your memory about the issues encountered.
00:08:17.370
After these steps, you will come to a conclusion: was your hypothesis discredited?
00:08:23.600
Do you feel closer to understanding the cause of your bug, or are you still clueless?
00:08:31.350
If you have managed to deduce the problem, fantastic! Implement the fix.
00:08:38.780
But if you're still puzzled, that’s ok too—it means you've discarded many possibilities.
00:08:45.260
Earlier, you may have had thousands of potential cause, and now you're down to a manageable few.
00:08:52.600
If you must circle back to the knowledge-gathering phase, you might have new insights or questions.
00:08:59.390
You can refer back to your earlier set-aside questions and see which one may now seem more relevant.
00:09:06.840
And don't forget the sharing phase of what you learned during debugging.
00:09:12.960
There's a wealth of information that you can gather and pass along to your team.
00:09:19.500
Certainly, all those new bugs you’ve found, if recorded properly, can help others in the future.
00:09:26.130
If you realize the issue originated from a misconfiguration elsewhere, indicating it to your team can prevent future headaches.
00:09:34.570
Another valuable item to share is your experimental procedure, which can serve as a useful onboarding tool.
00:09:41.240
This can help newcomers understand how to troubleshoot bugs, especially if they are new to the codebase.
00:09:48.200
Now, I’d like to share a personal story about how this process has helped me with some interesting bugs.
00:09:54.710
I was working on a project where we processed customer input and utilized a Node.js code sandbox.
00:10:01.810
While our application was designed for quick responses, a customer reported a process that hung for nearly five hours.
00:10:10.260
I had a vague idea of where the issue may lie, but I couldn’t determine if it was related to the JavaScript running in the sandbox.
00:10:17.340
Questions arose about possible issues in our logging since we couldn't pinpoint where the hang occurred.
00:10:24.890
I proposed a procedure where we could replicate the environment as closely to the customer's as possible.
00:10:30.960
After adding log lines to track progress through the code's lifecycle, I restarted the server.
00:10:37.080
To my surprise, the JavaScript code didn't hang—meaning my initial hypothesis was incorrect.
00:10:44.130
This discovery shifted my focus to whether the data might be too large.
00:10:51.960
If our data set was much larger than our testing data, perhaps it was causing the hang.
00:10:59.000
We decided to test it with a smaller data set, which confirmed my suspicion: it worked.
00:11:05.260
From there, we returned to gathering knowledge and found that Node.js had a buffer limit that our customer exceeded.
00:11:12.510
We resolved the issue by properly utilizing the I/O library to handle data flow.
00:11:18.600
This example demonstrates how your first question might lead you towards understanding through unexpected avenues.
00:11:25.890
The scientific method is particularly beneficial for debugging because it fosters collaboration and helps you get unstuck.
00:11:34.700
Once you’re unstuck, it enables you to keep progressing.
00:11:41.000
Collaboration occurs when you and your teammates are on the same page, particularly during frustrating times.
00:11:48.080
If you engage with teammates about a bug, your notes act as an invaluable resource for onboarding them on the issue.
00:11:54.500
They can quickly access your relevant findings without needing to understand the entire history of that bug.
00:12:01.400
This method also helps shift focus during discussions of cause and effect, simplifying communication.
00:12:08.150
By agreeing on a single hypothesis, you facilitate sound communication and investigation.
00:12:14.840
In situations where disagreements occur, focusing on a single question can enhance collaboration.
00:12:22.080
Writing down your thought process also accelerates the un-sticking mechanism.
00:12:28.990
By documenting your thoughts, you can mitigate feelings of being overwhelmed and regain focus.
00:12:36.600
When your thoughts swirl without structure, they can be daunting—writing clarifies them.
00:12:43.500
Through externalization, you gain enough emotional distance to discern which thoughts are truly impactful.
00:12:50.130
You may even find that thoughts you hadn't considered become pivotal to your debugging.
00:12:57.110
Additionally, questioning how to disprove your hypothesis can provide actionable steps amid chaos.
00:13:06.840
Flipping the overwhelming bug into a more focused question allows you to identify clearer actions.
00:13:13.740
Indeed, moving forward can happen quicker once you regain momentum after feeling stuck.
00:13:21.040
Once you have a direction after finding a solution, you can persist in seeking answers.
00:13:27.840
The scientific method helps sidestep the repeated questioning of assumptions, streamlining your investigation process.
00:13:34.950
When asking for help, your notes will clarify what has already been tried.
00:13:41.400
Everyone seeks clarity on the most recent developments and precious context you honor through note-taking.
00:13:48.070
Your thorough notes from prior exploration serve as a roadmap to where you're currently navigating.
00:13:54.160
It's common to stumble into previous approaches when faced with complex bugs—having a record helps prevent wasted effort.
00:14:01.690
Your notes also reveal patterns, making it easier to recognize and avoid repeating the same attempts.
00:14:08.580
Going through multiple iterations with a structured framework provides a sense of how much you've progressed.
00:14:15.090
While you may still be wrestling with several options, you’ll notice that you're systematically narrowing them down.
00:14:21.210
So, when should you use the scientific method for debugging?
00:14:30.060
Let me share an experience that crystallized this approach for me.
00:14:36.490
I was involved with a project that required packaging a Ruby on Rails app as a VM image for different infrastructures.
00:14:43.960
Each had to be configured differently to suit its respective environment.
00:14:50.290
We originally used a standard Ubuntu image but intended to swap it for one with extensive security hardening.
00:14:58.240
Our general procedure was to simply change the URL in our packaging tool and build it.
00:15:05.350
However, it turned out that it didn’t work as we hoped, and we spent a month troubleshooting.
00:15:11.780
The whole team felt incredibly stuck and frustrated.
00:15:17.530
By documenting our frustrations and experimenting, we focused better on finding solutions.
00:15:25.050
I realized that there are two types of ‘stuckness’—one being overwhelmed by too much information.
00:15:32.610
The second is the opposite, where you feel stalled with too little knowledge.
00:15:38.420
The scientific method has tools that can aid you through both of these states.
00:15:46.370
When you feel you have too little information, writing everything down can help concretize your understanding.
00:15:54.480
When there's too much information, framing a single question allows you to direct your focus.
00:16:01.090
Thus, the scientific method serves you throughout the debugging process.
00:16:08.070
We’ve talked about what the process of doing science is, how to apply it to debugging, and why it’s beneficial.
00:16:14.550
I hope the next time you find yourself stuck on a bug, you engage with the scientific method.
00:16:21.420
Remember to gather knowledge, ask probing questions, create hypotheses, and run experiments.
00:16:28.090
Take meticulous notes, draw conclusions, revisit your knowledge, and communicate with your colleagues.
00:16:34.530
I also have a reference guide on my website for debugging using this method.
00:16:40.800
I’m not a huge fan of slides, as they may lack fidelity; this can be a better reference for you.
00:16:47.950
I would love to hear from you about how the scientific method has helped your debugging process.
00:16:57.800
Thank you! I believe we have a few minutes for questions.
00:17:07.180
So, how do you determine when to stop working on a bug?
00:17:12.590
That’s a tough question, and it involves judgment calls.
00:17:21.000
I wouldn’t advise spending too long before consulting your colleagues, especially after several hours.
00:17:27.730
At my team, we usually have check-ins if a bug takes longer than two days.
00:17:34.250
How about estimating how long it’ll take to resolve a bug?
00:17:40.700
I usually only develop a working framework for our conversations about priorities.
00:17:46.130
If a bug seems to stretch longer, it’s key to pull in teammates for support.
00:17:53.030
But this is an evolving conversation depending on team culture.
00:17:56.880
If there are no further questions, thank you very much!
