Talks

Summarized using AI

Crescent Wrenches and Debuggers

Kerri Miller • June 16, 2017 • Earth

In her presentation titled "Crescent Wrenches and Debuggers" at RubyNation 2017, Kerri Miller discusses the intricacies of software failure and the importance of building a toolkit for rational inquiry. She emphasizes the perpetual state of failure that software exists in, impacted by various factors such as user errors, intrusions, and technical limitations. To navigate the complexities of troubleshooting, Kerri introduces a structured approach that mirrors the scientific method, advocating for asking relevant questions, conducting research, forming hypotheses, and testing them.

Key points covered in her talk include:

  • The Three C's: Kerri introduces the concepts of Challenge, Consider, and Conclude, which form a method for critically analyzing strange ideas or problems, akin to the scientific process.
  • Simple Diagnostic Approaches: By using the analogy of diagnosing engine problems, she identifies four critical components (fuel, air, spark, pressure) that can lead to engine failure, promoting a systematic approach to problem-solving in software.
  • Software Engineering Analogies: She draws parallels between diagnosing engine issues and troubleshooting software, highlighting the need for structured inquiry to pinpoint faults that can compromise performance.
  • Case Study in Learning: Kerri shares a personal anecdote about a motorcycle repair, illustrating how incorrect assumptions during troubleshooting can lead to unnecessary complications.
  • Addressing External Pressures: She outlines how external stresses, such as time constraints, can influence decision-making in software development, sometimes leading professionals to make poor choices under duress.
  • Functional Fixedness: The concept of functional fixedness is discussed, demonstrating how we often limit the potential uses of our tools and perspectives, with a historical example of the Titanic tragedy to underscore this point.

Kerri concludes her presentation with actionable takeaways for software engineers:
- Take apart and put together systems as a means to understand and solve problems.
- Seek expert insight, leveraging resources effectively.
- Maintain documentation to track problem-solving approaches.
- Recognize personal and systemic limits to decision-making.
She reinforces the notion that navigating challenges in software development requires both skill and collaboration, reminding her audience of the creativity and innovation inherent in their work as engineers.

Crescent Wrenches and Debuggers
Kerri Miller • June 16, 2017 • Earth

Crescent Wrenches and Debuggers: Building Your Own Toolkit For Rational Inquiry by Kerri Miller

Software exists in a constant state of failure, facing pressure on many fronts - malicious intruders, hapless users, accidental features, and our own limits of imagination all conspire to bring our system to a screeching halt. Untangle even the most tangled of Gordian Knots by building your own toolkit for inquiry, by relying on the simplest technique of all: asking “why?”

This presentation will discuss the challenges and potential solutions for refreshing multiple application environments (Development/Staging/UAT/etc.) with data from a Production database, while keeping some amount of table data intact from the prior database after the Production restore.

RubyNation 2017

00:00:25.250 Hey, good morning, everybody. I am, in fact, Kerri Miller. I am a staff engineer at Cork, and I am also a motorcycle enthusiast. Right now, before getting on the plane at the airport, I almost said yes to every motorcycle loan offer I received. So, I am excited to get home, no offense to Seattle, Washington. This is what it looks like every single day for about the five minutes when it's not raining.
00:00:40.680 This is a picture of my dog, whose name is TJ Higgins. His full AKC registered name is 'Princess Doll Loves Sampson's TJ Higgins.' They also have to include both the mother’s and the father’s names in the registration. You might ask what 'JSD' stands for. I'm glad you asked! JSD is a Doctor of Jurisprudence; it’s a law degree. So, he's not just my dog; he is, in fact, my lawyer. He even acts as my financial advisor, although his current advice is to invest in sticks! You know why? Because in 2006, this had a very high rate of return!
00:01:45.890 Thank you! In my mind's eye, he makes a lot of really strange claims. For example, he has never said that he is poor or abused; he just doesn't get to sleep on the couch half as much as he would like. Whenever he possesses some of these weird ideas, I fall back on the three C's: challenge, consider, and conclude. This is a triplet that comes up a lot in the skeptics' community, of which I'm not really a part. I love conspiracy theories; I absolutely love them. I think that they are magical, wonderful things, and I love thinking that the world might be stranger than we suppose. However, when someone presents us with an odd fact, like we should invest in tennis balls or sticks, we want to challenge that. We want to look at that idea and ask ourselves—Is this even true?
00:02:40.620 Then we consider what it would mean if that idea was true. If that strange idea were true, what would that lead to? How can we prove it or disprove it? Finally, we conclude by using something like Occam's razor to decide whether a fantastical claim is true or not. We use this same process consistently with most ideas we encounter. It acts as a variant of the scientific method, helping us separate fact from fiction in a very straightforward way.
00:03:10.470 I learned this process at Goddard College, which is, in fact, one of those colleges that emphasizes self-directed education. I like to say that I have a PhD—my 'Petit Haven Degree'—which stands for 'Daughters of Dewey School.' It's based on the principles of democratic education, emphasizing hands-on work through discussions about particular readings, performances, or pieces of material. As a group, we study these things, each person bringing something different based on their experiences, whether it's a novel or clay. We share those experiences and apply techniques of skepticism and inquiry to communicate with each other and find common ground regarding our narratives but also search for the core truth of the matter.
00:05:09.480 One of the declared values there is to 'get it right.' As alumni, we challenge ourselves and each other to embrace uncertainty, experiment, and imagine unexpected outcomes. Goddard really taught me that I could do anything I wanted to, so about 15 years ago, I decided I wanted to learn about small engine repair. I bought my first scooter, my first real motorcycle. My dad was ecstatic. As an old motorcycle guy, he grew up on a farm working on engines. Finally, there was a topic we could bond over! He would give me advice and teach me about engines over the phone or via Skype. This is my daily ride right now, used for getting around the city. I have other motorcycles that I don't want to ride on a daily basis; they are more for the weekend.
00:07:43.210 My dad taught me that there are really only four things that can ever go wrong with an engine. If the engine doesn't start or you have a problem with it, it's one of these four things: fuel, air, spark, and pressure. If your car doesn't start in the morning, it's usually one of these four things; nothing else can really go wrong with it. This forms a very simple search tree for diagnosing problems: Do you have fuel? Yes or no. Do you have air? Yes or no. Is there spark? If it’s an electrical issue, is the battery flat or dead? Finally, is there pressure? Because an internal combustion engine operates like a controlled explosion, capturing the energy of that explosion is key to making it work.
00:09:14.330 This simple mnemonic provides a search algorithm for diagnosing problems. Similarly, we can create search trees for our systems. We can ask, 'Why isn't our metrics tool working?' For instance, it might talk to Redis, so we check whether Redis is operational. Is Redis actually saving data? Oh, the server isn’t even online! This is often why, when you call Comcast because your cable modem isn't working, the first thing they say is to turn it off and back on again, as this frequently resolves the problem.
00:10:46.800 This process resembles the scientific method, which is a structured and repeatable way of conducting inquiries. The first step is to ask a question: What do we want to find out? Why is Redis behaving the way it is? We conduct background research to gather ideas and may stumble upon the notion that Redis is not accepting connections. Next, we devise ideas to test this hypothesis—to see if there’s a heartbeat or health check API that I can access. I gather results and interpret them to prove or disprove my initial idea about why the system is working or not. Finally, I communicate those results, integrating everything I have investigated and pushing that information out to the rest of the world.
00:11:52.620 This really outlines the process scientists utilize, formalizing the inquiry process that we all use in various situations. Of course, the purpose of the scientific method is to ensure that we don’t fool ourselves into thinking something is true when it really isn’t. One of the first times I took apart an engine—beyond just changing the oil—was when I cracked the engine on my scooter in half. I laid all the parts out on a clean blanket, visually displaying an exploded view of my engine. I put it back together because I wanted to upgrade the fuel jets to boost performance. After removing unnecessary weight from the bike—like the battery, which I figured was an extra five pounds—I reopened everything, hoping to get better performance.
00:12:59.580 After reassembly, I started it up and rode down the road; it ran smoothly. However, it started to experience odd issues: it began to stutter and shudder, and I witnessed it backfire. This was frightening! I ended up cruising to a gas station just as it started to rain in Seattle, increasing the drama of the situation significantly. At that point, I feared I had installed something incorrectly, and I spent about thirty minutes with my tools trying to diagnose what had gone wrong. I assumed it was the fuel jets because I had cleaned them with a special cleaner, but good thing I was at a gas station—I'd run out of gas!
00:14:51.120 It was just five ounces of fuel; the gas gauge was off! This illustrates the importance of having a structured method for making inquiries about your system—whether you sub in the scientific method or any other structured approach. In that moment, if I had immediately conducted a simple examination instead of assuming I had put something back together incorrectly, I would have saved myself a lot of time and unwanted rain!
00:16:20.389 It's common in software, too—if there’s an outage on a website, people often panic. Especially if there's no experienced support team, everyone tends to blame the code that was recently pushed. Yet this assumption can lead to misguided troubleshooting efforts; often, the code we just pushed causes stress elsewhere in the system. The idea of software existing in a perpetual state of failure is very real. You may find your software running smoothly in most situations until—BAM!—the pressure you've put on the system causes a fault.
00:18:46.780 We need structured inquiry processes to help us locate these invisible blind spots and biases we carry into any situation. It can be tempting to simplify this by saying the last code change is the only reason for the failure—a heuristic that simplifies explanations. Understanding our individual circumstances also plays into how our systems fail. Iko Karo in the poker world acknowledged this trend with his ‘threshold of misery.’ It serves as a reminder that, while we can sustain some pain, exceeding that threshold leads to problem acknowledgment or avoidance.
00:20:44.520 In software engineering, we hit these thresholds frequently. You may find yourself under time pressure—if you miss a crucial meeting and your whole day is ruined, you might make poor choices that compound the issue, like adding complications to the code when you really need to refactor it properly. External pressures and demands play critical roles in the decisions we make about code and software. We sometimes make choices not out of incompetency, but rather as calculated risks in challenging environments.
00:22:25.160 As engineers, we occasionally joke about the scalability of tire fires. I reflect on how our bad decisions stem from uncertain futures or lack of comprehensive understanding. Historically, companies have invested in areas that didn’t yield returns, and we learn from these situations. Our industry is consistently persisting out of outdated models or assumptions, and this impacts our effectiveness as professionals.
00:24:09.200 Let me now talk about my favorite tool: the crescent wrench. I love crescent wrenches; they are versatile for countless applications. In my early years as a stagehand before becoming a software developer, I had to use tools in creative ways. I’ve wedged them to prop open doors on semi-trucks, and I once used it to fix a jammed brake on my scooter while it was raining. However, I did not once use it as a substitute for a fuse! They say necessity breeds creativity, but tools like my crescent wrench should be used wisely and in their intended way.
00:25:08.330 We also discuss functional fixedness—how we limit our perceptions of a tool’s uses. A common example relates to the Titanic. Many believe that the tragedy occurred because of a lack of lifeboats, correlating it directly to the idea that a lifeboat could've potentially saved those people. In reality, the lesson here teaches us to stay open to possibilities beyond our common perception of tools and situations. Much like how a wrench should not be our sole creative process, a fresh perspective is often key in software development.
00:26:54.669 To wrap up, I’d like to leave you with a few actionable takeaways. This is my dad’s guide to fixing things: take it apart and put it back together. Sometimes we get frustrated, letting obstacles deviate us from better solutions. It's easy to feel like everything needs an immediate fix, but taking a breath can reduce the panic and help clarify the situation. Consult experts—there’s abundant information and resources available, and you can get there simply by asking. Be sure to document your process and maintain breadcrumbs of all the experiments you've conducted. Lastly, know when to pull the parachute—there are limits to what we can achieve given our current understanding, and recognizing this is crucial.
00:29:09.340 Thank you for your time! Remember that what we do is fundamentally challenging yet rewarding work. Each of us performs modern miracles in dealing with complex systems. Embrace the difficulties, know your tools, and be kind to each other.
Explore all talks recorded at RubyNation 2017
+3