Craft, Engineering, and the Essence of Programming

by Glenn Vanderburg

The video titled "Craft, Engineering, and the Essence of Programming" features Glenn Vanderburg's insightful talk from RailsConf 2011, where he explores the nature of software engineering, challenging the traditional perceptions of engineering in relation to software development. Vanderburg outlines the evolution of software engineering from its inception during the software crisis of the 1960s to the emergence of the Agile movement, which shifted ownership and methods within the industry. He highlights the misinterpretation of engineering, where many equate it solely with rigorous mathematical modeling and formal documentation, particularly influenced by civil engineering ideals.

Key Points Discussed:
- Historical Context: The software engineering field emerged from the software crisis that emerged in the 1960s, leading to the first software engineering conference in 1968 and the subsequent dominance of traditional engineering methods for about 30 years.
- Shift to Agile: The late 1990s saw the rise of the Agile movement, empowering programmers to take ownership of the processes involved in software delivery, improving project outcomes.
- Crisis of Identity: Vanderburg references Tom DeMarco's view on the failed perception of programming as an engineering discipline, alongside Bruce Williams' assertion that those in software roles are not engineers, underscoring a misunderstanding of engineering principles.
- Nature of Engineering: He distinguishes between various engineering disciplines, illustrating that while civil engineering relies heavily on documentation, other fields utilize iterative processes and design thinking more prominently.
- Examples and Anecdotes: Vanderburg shares Eugene Ferguson's story about pre-rotating tires on a Douglas DC-3, showcasing how engineering solutions often have multiple approaches that are context-driven rather than strictly mathematical.
- Reframing Software Engineering: The talk argues that programming represents a unique blend of engineering and craftsmanship, suggesting that the dual role of designing and building in software allows for a deeper engagement with the work.

Conclusions:
- Vanderburg encourages software professionals to embrace their identity as both engineers and craftsmen, highlighting the importance of iterative development and prototyping rather than traditional upfront analysis.
- He stresses that understanding this balance enriches the role of software engineers, ensuring they focus on the significance of quality, structure, and long-term implications of their designs.
Overall, this video provides a thought-provoking examination of the essence of programming, advocating for a broader view of engineering that encompasses the creative and iterative nature of software development.

00:00:05.440 You look just like they said. My boys will be immensely pleased by that introduction. Thanks.

00:00:10.639 Am I on the screen? In one of the many pearls of wisdom in Gerald Weinberg's great book, 'Secrets of Consulting,' is one that he calls Ronda's First Revelation.

00:00:19.400 Ronda's First Revelation says, 'It may look like a crisis, but it's only the ending of an illusion.'

00:00:26.240 The field of software engineering originated in the middle of what was called the software crisis in the 1960s. As software projects got bigger, more complicated, and more ambitious by the standards of the time, they also became more difficult to control, manage, and predict.

00:00:38.680 They started failing in various ways, and this became common. So, NATO of all things called people from industry and academia, practitioners and theorists, a whole big group of people to Germany for the first software engineering conference in 1968.

00:00:53.120 It actually got off to a kind of promising start, but then for various reasons that I won't go into in detail, it went wrong. The result was that we labored under traditional software engineering ideas for about 30 years.

00:01:10.600 That's a little unfair because the software engineering folks did realize some of the limitations of the waterfall model, and they tried other methods. However, in my mind, all of the traditional software engineering processes suffer from the same fundamental limitations that waterfall did.

00:01:35.840 Things were pretty unpleasant in large parts of the software industry for about 30 years until the late '90s when the Agile movement came along. Practitioners and programmers began taking ownership of the processes and methods they used to deliver software.

00:02:00.200 Things got a lot better, but the failure of the traditional software engineering ideas has given rise to another crisis. A couple of years ago, Tom DeMarco, who in some ways was one of the fathers of software engineering, published an article where he suggested that the idea of programming as an engineering discipline was a failed idea that should be abandoned.

00:02:17.519 Closer to home, my colleague Bruce Williams tweeted, 'I am not an engineer, and neither are you. I doubt you're an architect either.' Both crises—the original software crisis and the crisis of identity we are witnessing now—are driven by the same illusion: a thorough misunderstanding of engineering and what it is.

00:02:37.400 This was my response to Bruce. It was a little more curt than maybe he deserved, but you know, I was busy. Let's talk about engineering for a bit and what it is. When you discuss engineering with people who lack an engineering training and background, they tend to think of engineering in terms of modeling, mathematical modeling, formal analysis, and a heavy emphasis on documentation.

00:03:07.360 The goal of this rigorous approach is to ensure that the end product is safe, reliable, and predictable in terms of budget, time, and quality. Additionally, people often think about engineering as related to civil engineering, particularly large-scale structural engineering like bridges and highways. This perspective is natural because those structures are tangible and easy to understand.

00:03:44.599 If we take that engineering checklist and apply it to civil engineering, we find that modeling is indeed a significant part of the process. Mathematical modeling, formal analysis, and extensive documentation are vital elements. We trust the safety of bridges and civil engineering projects; however, failures do occur every few years despite these precautions.

00:04:29.360 While these projects are largely safe, it's not guaranteed, and predictability can be just as illusory. Anyone involved in local or state government sees this firsthand, often observing significant cost and time overruns. Sometimes the predictable parts of civil engineering come down to construction processes rather than the engineering itself, which adds to the illusion that the engineering processes are truly predictable.

00:05:37.120 It’s important to note that there are many kinds of engineering—civil, structural, aerospace, mechanical, among others. These fields differ in terms of scale, complexity, and levels of abstraction, but as you go down the list from large-scale concrete works to smaller, more abstract kinds of engineering, the rules change substantially. In fields like chemical, industrial, and electrical engineering, costs are less often dominated by materials and labor and more by design processes.

00:06:13.320 For instance, when we look at industrial engineering, which focuses on designing and optimizing processes (often factory processes), the approach differs. Industrial designers collaborate with mechanical engineers to create solutions for factory machinery. A prime example would be the automatic donut machine in a Krispy Kreme shop. Its design was not predicated on mathematical modeling and formal analysis; it's likely the result of iterative tinkering—trying different configurations until finding one that worked effectively.

00:07:55.919 This iterative approach is common in engineering, and it's often about balancing against documentation and mathematical rigor. Furthermore, as you work toward smaller and more abstract types of engineering, the propensity for prototyping increases, indicating a shift away from classical engineering models which are often criticized for being overly rigorous. We have an idealized view of engineering as a very sophisticated and planned endeavor; however, it can also simply consist of common sense and iterative processes.

00:08:57.920 The aerospace engineer Eugene Ferguson tells the story of his first engineering invention as a boy when he observed a Douglas DC-3 landing in 1937. He noted how tires that weren't rotating yet made contact with the runway, creating smoke and possibly harming the integrity of the tires under stress. This inspired him to sketch a design for a device to pre-rotate tires before landing, which he later submitted to the Douglas Aircraft Corporation.

00:09:41.120 The response he received illustrated how often technical problems have multiple solutions. The corporation informed him that this issue, known as pre-rotation, had already been recognized, and while there were multiple proposals, they ultimately rejected those solutions that added excessive machinery and instead used the weight saved to improve tire tread thickness. Ferguson realized that every technical problem often has alternative solutions; the key is knowing the constraints and the impact of those solutions.

00:10:41.680 Fast forward to today, many in the engineering community still debate whether software engineering is a true discipline. My response to Bruce, who expressed a change of heart after our discussions, led him to ask about architecture. My reluctance shows a prevailing attitude towards the debate of whether software architecture counts as real architecture. Most misunderstandings stem from a lack of clarity on what engineers do and the true nature of engineering itself.

00:12:22.560 We tend to idealize engineering, viewing it through the lens of documentation and mathematical modeling. However, many important aspects of engineering go unrecognized in traditional discussions. Books like 'What Engineers Know and How They Know It' by Walter Vincenti and 'Engineering in the Mind's Eye' by Eugene Ferguson are essential for those wanting to understand the broader scope of engineering qualities and practices.

00:13:07.720 Programming fits into engineering on that continuum toward abstraction. However, it differs from other engineering disciplines in two critical ways. First, the conceptual model for software engineering, as derived from traditional engineering, assumes that engineers produce designs handed off to laborers who build the finished product, a notion that doesn't hold true in the realm of software.

00:14:16.440 Software engineers often find themselves creating ongoing modules and systems without a clear handoff process or distinct division of labor. This was critiqued by Jack Reeves in 1992 when he argued that software engineering processes needed to depart from traditional design documents. He suggested that the source code itself acts as the primary design, pointing out that while source code is crucial, it does not represent the finished artifact or what customers truly pay for.

00:15:40.800 He encouraged viewing source code not only as a working artifact but as an evolving design, with the key point being that running software expresses what customers want, fundamentally changing the economic model of software engineering. Unlike traditional engineering, where material and labor are significant costs, software programming often requires iteration, allowing engineers to avoid over-engineering.

00:16:38.360 The implication of this is important: the proper approach to software engineering isn't to front-load analysis and documentation but rather to be iterative, developing multiple prototypes that can be tested rapidly. This shifts the responsibility for engineering's accuracy away from rigid structures toward flexible, guided experimentation.

00:17:55.360 When considering whether there is a model in programming, source code serves as both the model and verification document. Mathematical principles exist in programming languages, although not always as formal specifications. The nature of the programming disciplines means we exploit both the documentation and execution of our code, yielding an advantage not readily available in traditional engineering realms.

00:19:11.799 There’s a distinct upside-down economics in software, where traditionally expensive approaches become cheaper, redefining our understanding of documentation. Engineering is more than structured documentation; it allows for the execution of specifications, providing a dual layer of verification compared to classical methods.

00:20:01.240 Additionally, many successful engineering disciplines embrace processes without questioning their fundamental validity. Electrical engineers never ponder if their field is real engineering; they simply embrace their practices. I propose we adapt the same attitude for software programming—it is time we stop questioning our place and embrace our practices.

00:20:35.600 In the current landscape, we face a dilemma: would we prefer to think of ourselves primarily as engineers, or would we rather recognize the craft within our work? Both engineering and craftsmanship play distinct roles in the software development process, prompting the need to consider how craft influences engineering outcomes.

00:21:07.520 The distinction between craftsmanship and engineering often appears when discussing documentation. An artisan may produce documents to help conceptualize their work but ultimately builds based on intuition. Engineers, however, must generate documents to convey their designs to builders because they are typically separate from the actual construction.

00:22:12.179 To understand this better, consider civil and structural engineering where documentation mainly serves as formal communication with builders. These engineering models may seem arbitrary but are crucial for conveying complex structures, while software's abstraction allows flexibility and allows it to feel more intimate versus separated.

00:23:08.360 As we explore engineering documentation from structural through to software engineering, we note that while early documentation resembles the tangible product being constructed, as we arrive at software engineering, the models appear completely abstract. However, programming languages are meticulously designed to maintain isomorphism with the constructs that they execute on the computer, rendering our code the ultimate blueprint of our final product.

00:24:49.760 This relationship implies that as software engineers, we experience our craft in unique ways that intertwine the roles of designer and builder; the act of coding applies direct pressure on the abstractions and constructs we create. Software engineering is the most abstract of engineering disciplines, yet our models have concrete relationships with our actions, enabling us to explore our work directly without a separation that might exist in traditional methods.

00:26:28.040 As engineering disciplines evolve, especially with the integration of technology, traditional models become less mathematical and more simulation-based, highlighting the need for engineers to adapt. In electrical engineering, circuit simulations offer efficiencies that parallel the advantages we’ve adopted in coding, where rapid prototyping becomes not only viable but often necessary to lead innovation.

00:27:58.720 In conclusion, the models we work with in software engineering provide a direct sense of working with the artifacts we create. These constructs possess an isomorphic relationship with what gets executed, becoming evident in our interactions with coding. This dual relationship, where we design and build simultaneously, fosters deeper engagement with our creations and emphasizes that as software professionals, we must embrace both the aspects of engineering and craftsmanship.

00:29:49.799 We are designers and makers, deeply engaged in our creations. Understanding this balance brings luxury and responsibility to our role, ensuring we never lose sight of the importance of structure, quality, and the long-term impact of our digital creations. Thank you very much.