Grasping Complexity with Both Hands

by Glenn Vanderburg

In the talk titled "Grasping Complexity with Both Hands," Glenn Vanderburg discusses the challenges programmers face when dealing with complexity in software design and problem-solving. He emphasizes the tendency to seek simplicity, often leading to oversimplification of complex issues, especially in the context of programming and broader societal issues like politics.

Key Points Discussed:
- Preference for Simplicity: Programmers often gravitate towards simpler solutions, finding straightforward code easier to read and maintain. However, simplicity can sometimes lead to overlooking the nuanced complexities of problems.
- Binary Thinking: The programming mindset can promote black-and-white thinking, which can be detrimental when faced with more ambiguous situations that require nuanced understanding.
- The Complexity of Solutions: Contrary to the desire for complete solutions, Vanderburg explores the idea that many problems are inherently complex and resist simple answers. He illustrates this through humor, such as the joke about the zoologist, physicist, mathematician, and programmer, which critiques how people react to observations based on their disciplinary biases.
- Impacts of Oversimplification: The speaker notes how oversimplifying can lead to poor decision-making, particularly highlighted in political discussions that often rely on reductive arguments.
- Ignoring Second-Order Effects: Vanderburg provides examples from programming, such as the effects of patent application systems and garbage collection in Java, to illustrate the consequences of ignoring the broader effects of certain decisions.
- Wrong Questions: The importance of asking the right questions is highlighted. An example involving organizational practices shows how preconceptions can hinder understanding of valid processes.
- Iterative Engineering Approach: Vanderburg urges the programming community to look to engineering disciplines that share commonalities with software engineering, particularly the importance of iteration and experimentation.
- Managing Complexity: He suggests that techniques to deal with complexity include focusing on incomplete solutions and accepting a failure rate, rather than seeking perfection at the outset.
- Understanding Emergent Phenomena: The speaker advocates for a focus on understanding the fundamental drivers behind complex situations rather than trying to control every aspect, which can lead to paralysis.
- Wicked Problems: The concept of 'wicked problems' is introduced as issues that defy simple solutions and require multi-faceted approaches.

Conclusions and Takeaways: Vanderburg concludes by urging programmers to embrace complexity and refrain from oversimplification. He emphasizes the need for tools and strategies to make informed decisions in the face of complexity, encouraging an openness to engage with wicked problems and adopt approaches that allow for nuanced understanding. The overall message champions a balanced view of simplicity and complexity in problem-solving in software engineering and beyond.

00:00:08.480 I am really happy to be here. I've been trying to see a talk by Glenn Vanderburg for years, and I always seem to miss him. I figured the only way I would ever get to see him was to get him here at my own conference. So, I wrote him an email and invited him to come speak. He initially replied that he wasn't speaking anywhere this year because he had too much on his plate. But then he said, 'It's GoGaRuCo; how can I say no?' So, he came here because of all of you. Thank you for getting him out here! Glenn has been doing Ruby since around 2000; he is part of the old guard and has worked at InfoEther with people like Rich Kilmer and Chad Fowler. I'm delighted to have him here today to talk to you. He is from Dallas, Texas, so please give him a warm San Francisco welcome.

00:01:00.320 I currently work for LivingSocial, and as you might have noticed, we're hiring. Thanks, Josh, for allowing me to present the penultimate talk of the conference. I want to start with a disclaimer: I'm here to talk about something I am not an expert in. I chose this topic because I wanted an excuse to study these concepts, learn about them, and improve my understanding. I don’t claim to have all the answers; this is part of my learning process, and I hope you find something valuable in it. I would love to receive your feedback or insights at [email protected] or on Twitter @glv if you want to engage further or teach me something.

00:01:20.320 As programmers, we have certain biases and habits of thought that can serve us well. We tend to love simplicity; we seek code that is as straightforward as possible to accomplish our tasks. Throughout my career, I've emphasized the importance of keeping things simple whenever possible. Simpler code, simpler design, and simpler solutions are easier to read, understand, debug, and modify. While it is often more challenging to write simple code and to devise simple designs, the rewards become apparent in the long run.

00:02:06.320 Because of the nature of our work, we also tend to think in binary terms. Our world appears black and white; answers to questions are true or false, and solutions are either right or wrong. Our experiences in coding foster this mindset, which can be beneficial, but it also seeps into areas of our thinking where things might not be so clear-cut. To be fair, this tendency to oversimplify is not limited to programmers; it is common across various fields. Everyone tends to seek straightforward answers when faced with complexities, often leading us to overlook the nuances.

00:03:00.959 We programmers enjoy solutions that cover every edge case and handle boundary conditions thoroughly. The holy grail for us is discovering algorithms or designs that are both simple and comprehensive, effectively addressing all nuances without resorting to convoluted special cases in the code. To illustrate this tendency, let me share a joke involving a zoologist, a physicist, a mathematician, and a programmer on safari. As they observe a herd of zebras, the zoologist exclaims, 'My God! That zebra is spotted, not striped! I’ll be famous for discovering spotted zebras!' The physicist responds, 'Wait, all we know is that at least one zebra is spotted.' The mathematician then critiques that view, saying, 'No, what the evidence truly reveals is that at least one side of a zebra is spotted.' Meanwhile, the programmer, standing at the front of the vehicle, exclaims, 'Oh no, a special case!'

00:04:05.439 When I first heard that joke, my reaction was that it was probably made by someone who wasn’t a skilled programmer, as a good programmer would have had a different reaction. A proficient programmer might recognize that spots are generalizations of stripes and might think about how to write algorithms that simulate both. This is the ideal solution: a single algorithm that accommodates all the various special cases we encounter in reality. So we appreciate simplicity, yet sometimes, we also embrace complexity, as long as it seems manageable; we enjoy the challenge it represents. This is evident in our enthusiasm for languages like Scala and participation in coding contests that often glorify obfuscation.

00:05:40.000 However, there are instances of complexity that feel boundless, where problems appear to lack clean solutions. These can be problems that aren’t even computable, or they may be context-dependent in such a way that our accumulated context is insufficient for an adequate solution. Often, the most favorable solutions we can conceive come with significant downsides, leaving us paralyzed as we choose among flawed alternatives. It’s easy to shy away from such complex problems, and when confronted with them, if we can’t completely disregard them, we may instead resort to oversimplifying. We act as though these issues are simple, arguing and making decisions as if they were uncomplicated.

00:06:57.600 I want to be transparent about the motivation for this talk; it's an election year, and I find myself frustrated by the oversimplified and often angry rhetoric that dominates our political culture. It seems that as a society, we struggle to grapple with the full complexity of the difficult challenges we face. This phenomenon isn't unique to politics; programmers exhibit a similar tendency to oversimplify. I wish to discuss the many ways we painfully simplify matters and how this leads to poor decision-making because we fail to confront the actual complexities that lie before us. I also want to highlight mental habits and techniques we can cultivate to engage with complex issues constructively and productively, eventually driving toward sensible solutions amid the complexities faced by programmers today.

00:09:41.960 We often oversimplify by focusing too much on primary effects while neglecting second-order effects that arise indirectly from our solutions. For instance, a friend of mine once sought my help to develop a software system for him and his business partner. His partner's acquaintance was the leading patent holder at Dell Computers, and they had streamlined the patent application process, making it easier to navigate. My initial objection was that the world does not need more patents. Yet, my primary concern was about the nature of the system he was proposing—it was not mechanical but a human process.

00:10:19.760 The patent office is comprised of human examiners and managers, who are already overworked and undercompensated. Therefore, if a product were to expedite and funnel more patent applications into the system, it would undoubtedly alter the current process, exemplifying a failure to account for second-order effects and assuming the system would not react to these changes. Another example comes from my early experiences with Java; many people expressed resentment towards garbage collection because of its perceived slowness. Critics focus on delays caused by garbage collection, failing to consider the necessity of bookkeeping that occurs in manual memory management. They neglect the fact that if it is handled by a garbage collector, it can lead to a much more efficient overall system.

00:11:27.119 We often neglect secondary benefits as well. An excellent illustration of this is found in test-driven development (TDD) or unit testing, where I've witnessed a tendency to zero in on one perceived benefit while overlooking others. Many people who learn TDD tend to fixate on the advantage that resonates most with their experiences, ignoring the rest. Intriguingly, if you only evaluate TDD based on a single benefit, it may not appear cost-effective. However, when you consider all the advantages together, the case for TDD becomes compelling.

00:12:17.040 Sometimes we're simply asking the wrong questions. Sandy mentioned a fitting example during her talk: she pointed out how people often ask, 'What object should know this?' which inherently assumes that one object must bear the knowledge. The response that 'an object that hasn't been created yet' isn’t likely to come to mind. Instead, she suggested rephrasing the question to 'What message should I send?' This reframing can clarify the need for new classes to receive that message.

00:12:54.960 A few years ago, I worked with a stark organization that was trying to modernize. For those of you familiar with AS400s, you’ll appreciate the challenges we faced in a large development group where many of the standard practices like source control and testing automation were novel. I discovered that their policy required us to provide a WAR file of the source code along with a script detailing how to build the system, almost an archaic process ensuring they could always provide the source code for everything running in production.

00:14:00.160 This seemed absurd to me, as it essentially guaranteed that we could never deploy the actual tested code. After expressing my frustration, I realized I was asking the wrong questions. Instead of assuming that the operations team was illogical, I should have started from the premise that they had valid reasons for their methods. This led me to uncover that the operations group was under obligation due to disaster recovery initiatives to produce the source code for everything running in production. If the development team used reliable source control practices and automated deployment, this could have been a collaborative process.

00:15:09.760 Let me share an example that I've been pondering for the last five years. Early in my career, I often heard the sentiment, 'Why can't programming be more like engineering?' Many believed we needed to adopt formal methods and a more disciplined approach akin to engineering. For some time, I embraced that notion. Eventually, I realized that this inquiry was misguided; a more productive question would be to ask, 'Which type of engineering resembles our work in programming?' In fact, many questioners were picturing large-scale civil engineering, where iteration and experimentation are costly. Conversely, other branches of engineering—like electrical, chemical, and industrial engineering—are much more similar to software due to their empirical and iterative processes.

00:16:47.440 We fall into the trap of oversimplifying by viewing issues in black and white, as my friend Neil Ford terms it, the 'sucks/rocks dichotomy.' The dynamic between static and dynamic typing and arguments surrounding object-oriented versus functional programming often exhibit this tendency. During a past conference, we discussed Google’s V8 JavaScript VM, and when a colleague commented that it had a bug, it illustrated how readily we sometimes dismiss new ideas or technologies without considering their potential value. Finding a single flaw can lead us to outright rejection, but systems do not need to be flawless to be beneficial.

00:18:12.560 Similarly, individuals may reject Test-Driven Development (TDD) due to misconceptions about its efficacy, demanding it provide mathematical certainty, which it doesn't. TDD serves to enhance our confidence in the correctness of systems but does not equate to proof. During debates, we often adopt rigid positions for rhetorical reasons, only to forget these positions are intended to serve as discussions and not definitive answers. In the early stages of Extreme Programming (XP), proponents took strong stances saying you had to do all twelve practices or risk failing. Many knew there were flexibilities, yet this dogmatic stance temporarily resulted in breakthroughs, leading some who initially rejected practices to reconsider and even adopt them.

00:19:19.920 In the realm of software engineering, we frequently disregard context. For example, at LivingSocial, we deploy software solutions with frequency, while my colleagues at Square mentioned they take a more conservative approach with financial systems due to the additional risks involved. This demonstrates how easy it is to jump to conclusions based on our experiences without recognizing or respecting the unique challenges faced in different contexts. Sometimes, the problems we encounter cannot simply be simplified, and we end up throwing our hands up in defeat, resorting to 'magic solutions.' Often, when someone says, 'He just doesn't get it,' it signals a lack of effort to understand the reasoning behind differing perspectives.

00:20:52.640 This inability to comprehend alternative viewpoints often leads to arguments grounded in aesthetics or ephemeral qualities like beauty and elegance without critical examination. I aim to refrain from painting a fatalistic picture of our thinking processes; rather, I wish to highlight strategies for managing complexity that can lead us toward more intelligent decisions even amidst uncertainty. For example, pursuing incomplete solutions can yield results; I have worked with individuals who tackle 'impossible' problems by targeting an 80% solution with 20% effort.

00:22:00.480 One entrepreneur I encountered established a methodology for weight loss encapsulated in three simple rules: never eat white foods, never eat in front of the TV, and always park far from your destination. These may not be optimal but they are memorable and manageable for most people. While we often prefer completeness in our solutions, there is merit in smaller or less-than-perfect solutions — they can drive action and results. We should also consider heuristics or probabilistic answers that help target more accurate solutions in many cases.

00:23:11.760 For instance, Git relies on the uniqueness of sha hashes—though theoretically, different pieces of text may yield the same hash, the probability of this occurrence is so exceedingly low that we can trust it. My friend Stuart Holloway referred to GitHub functioning as 'a globally distributed attack on SHA-1'. The first system I encountered using a similar concept was the 'Venti' network file server from Plan 9, which utilized a block-addressable system, tying file access to SHA hashes of content. It allowed for efficient backups while minimizing storage growth over time, demonstrating that adopting workable probability approaches can lead to practical solutions.

00:24:05.760 We should learn to accept a manageable failure rate. In programming, we often encounter power law distributions where a small subset of cases dominate results, known as the Pareto principle. Many may observe normal distributions and assume this is the default pattern; however, power law distributions are evident in our code bases and user experiences, dictating how we distribute our efforts. Finding solutions for more common cases whilst permitting imperfections in less frequent scenarios can be a strategy to effectively manage complexity.

00:25:09.840 When confronting complex situations, focus on assessing costs, benefits, and risks rather than labeling actions as right or wrong. Recently, I engaged in a conversation on technical debt, where someone posited whether technical debt was hindering us—arguably, it does not prevent us from completing tasks, but it can substantially raise the cost and time involved in executing them. Embracing this mindset fosters better decision-making by consistently considering future costs, potential risks, and the implications of our current choices.

00:26:15.040 We should also analyze emergent phenomena—when we restrict our focus to controlling every facet of a situation, it can lead to paralysis in the face of complexity. Understanding simple rules that yield broad impacts can lead to your environment's stability. For instance, Sandy's presentation yesterday highlighted smaller, localized decisions leading to better software design. Instead of being fixated on avoiding all issues, we should allocate our time effectively to ensure projects progress meaningfully.

00:27:06.720 Finally, sometimes deeper analysis may yield insights from problems we thought beyond reasoning. By categorizing entities as costs, risks, and relationships, we can uncover hidden simplicity within complexity. David Thomas once remarked that a piece of software mirroring all elements of extreme programming would likely encounter serious validation challenges. This led me to realize the merit in flawed systems working cohesively to support each other, promoting an understanding that mistakes occurring on smaller scales can be rectified through larger systemic practices.

00:28:12.960 I encourage you not to resign too quickly. When faced with confusion about why something seems wrong or unappealing, take the time to articulate your intuition. In Steven's talk, he explored how experts sometimes forget the reasoning behind their understandings. As experienced programmers, we might rely heavily on intuition, but it is vital to break down those gut feelings to enable clearer communication and more informed decisions.

00:29:06.960 In conclusion, take the time to explore what are deemed 'wicked problems.' These intricate problems lack definitive solutions and require multifaceted approaches. Initially identified by Rittel and Webber in the 1970s, wicked problems are now acknowledged as prevalent in areas like political planning and health management. To develop your skills in handling complexity, delve into techniques that tackle wicked problems and learn from those experiences to wield a better approach.

00:30:03.680 To summarize: Yes, seek simplicity; it is beneficial. Avoid over-complicating simple things, but when faced with real complexity, grasp it firmly. Do not oversimplify problems, as that will exacerbate the situation. Equip yourself with knowledge and techniques that aid in making informed decisions amidst complexity. Thank you very much.

00:30:30.880 You.