00:00:17.920
Okay, welcome everyone. My name is Shaina, and I'm part of the RubyConf programming planning committee. Today, we're delighted to have a senior Microsoft software engineer from Seattle joining us.
00:00:24.000
Our speaker, a Ruby and functional programming enthusiast, has been in the programming world since 2008. He is passionate about learning new languages and exploring software engineering processes and organization. Let's warmly welcome Phil Crissman to the stage!
00:01:05.280
I made a lot of new Rails apps from the ground up; it was a great experience. In about 2013, I moved to Seattle and took a short contract at Amazon, working on a couple of Seattle-based startups. Then in 2015, I started at Microsoft. For anyone who’s too young to know, that was a real Microsoft logo at some point in the '80s—the most metal of all Microsoft logos.
00:01:22.000
Yes, and the part of Microsoft that I joined used to be a startup called Yammer. Yammer was built on top of Ruby on Rails and had a bunch of Java microservices, and they decided to keep that tech stack because it would have been crazy to rewrite it.
00:01:38.520
This talk is called 'How Programs Learn, and What Happens After They’re Built.' I got a little clicker, maybe I’ll try that. It’s not about machine learning, oh that goes backwards. Okay, I know how to do this.
00:01:58.920
Sorry, so let’s see. I’ll start with a little preliminary stuff. There’s a book called 'How Buildings Learn' that inspired the title of this talk. A lot of the content comes from a paper from the late '90s called 'The Big Ball of Mud'.
00:02:15.680
So that’s where some other material for the talk comes from. Then I’ll try and think about: given all this, now what? Oh yeah, there’s our source material.
00:02:29.000
Clicking is good. Who’s ever seen this acronym? What does that mean? Meantime to failure? Yes, we get that from manufacturing, and I think kind of DevOps and computing have adopted it a little bit. A little while ago, when I saw these, I thought about a metric I like to use for my own codebases; that is MTTA.
00:02:58.640
It started as a thought experiment. I imagined, okay, if we have a new person at work or maybe there’s someone visiting someone I really respect, and it’s my job to walk them through our codebase, how long will that walkthrough take before I find myself apologizing for something? An apology in my head is very broad—it could be anything from 'we’ve really been meaning to refactor this' all the way to 'I’m sorry you have to see this' and 'I’m sorry it exists.'
00:03:25.160
I realized that in a lot of codebases I’ve worked on, my MTTA would be effectively zero because I’d preemptively apologize for what I was about to show them. But I’m also Canadian, so that might have something to do with it.
00:03:41.200
When I started, I also had the good fortune to learn about agile software development processes, because even the Rails book at the time was about agile web development with Rails. I loved the idea; it made a lot of sense to me immediately. This is sort of how I imagined our software would grow.
00:03:59.680
It would grow incrementally and iteratively; parts of it would be different, but it would all fit together wonderfully. It was all going to be beautiful. However, how I really felt about the software usually felt a little more like this. I got gradually to work on larger systems, and I felt like this was the world that Kbac wants me to live in.
00:04:30.160
But again, the reality often felt a lot more like this. You think of a new feature or a new thing you need to do, and you can draw it; it’s pretty and small and simple. Chat GPT drew this, so some of the arrows don’t actually connect, but let's imagine it makes sense.
00:05:08.120
But again, the reality always felt a little bit short. Within a couple of weeks of doing the work, I would take a look at it and wonder how it even works. Do the tests pass? One thing I really liked about that first eight or so years I got to spend contracting is that I was the guy typing 'rails new' a lot.
00:05:44.720
I got to witness that exact moment when the code goes from something that’s small and beautiful and fits in your head to when it doesn’t anymore and someone will come up to you and ask you how this works and what it does. Well, I don’t think anyone really knows.
00:06:23.160
What is knowledge anyway? How do we know anything? Software architecture, as a metaphor for building software, has a long history in the software development and software engineering community. It’s by no means the only metaphor for building software, and it’s possibly not even the best one. But because of the source material I’m using, we often use the word architecture and architect for roles and for features of what we build.
00:06:54.640
Other metaphors exist; some people think of software as more organic, growing like a garden. There are other ways to think of it, but for better or worse, today we're going to think about it as it relates to buildings.
00:07:34.000
This is the book, 'How Buildings Learn' by Stewart Brand. You can see right on the cover kind of where he’s going with it. That’s a photograph of two buildings in New Orleans, about 30-40 years apart. You can see they’ve both grown, and they’ve both changed, but they haven’t changed in the same way.
00:08:05.680
That’s kind of an ongoing theme of this book. Stewart Brand is mostly known for the Whole Earth Catalog. He’s written a lot of books over the years. He always kind of had one foot in the Berkeley '60s counterculture and the other in technology and the development of new things.
00:08:30.000
The themes in his book deal with adaptations—how buildings do or don’t adapt easily. There’s the concept of time layering; different layers of architecture change at different rates. Ideally, your structure remains static for a long time, but what is inside that structure can change at different rates and with different levels of difficulty.
00:09:03.440
Almost no buildings adapt well; they’re designed not to adapt. This is a building known as the 'Salt Tower' in England. It was originally a water tower, but at some point, when the water tower was no longer needed, someone decided to convert it into a house.
00:09:20.000
It seems to have worked but at the cost of creating an unconventional house. Brand uses the term 'shearing layers,' which will come up again. He makes this statement because of the different rates of change among a building’s components: a building is always tearing itself apart.
00:09:47.760
The diagram is great because it uses the thickness of the line to show how hard it is to change something, while the arrows and the number of arrows indicate how frequently you might need or want to change that thing. Some parts of a building change very slowly, while other things change frequently.
00:10:06.640
Fortunately, most of those things that change frequently are also easier to change, like the stuff inside your building. Brand got this idea of layers from architect Frank Duffy, who said, 'the unit of analysis isn't the building; it's the use of the building through time.' Time is the essence of the real design problem.
00:10:30.000
There’s what we’re building today and what we’ll need in the future. Brand identifies a couple of different categories of architecture, not in the sense of Georgian, Victorian, modern, but in his concept of low-road architecture.
00:11:00.160
Low-road architecture refers to what he thinks of as office parks and industrial warehouse spaces; it’s cheaper to build and easier to change. This contrasts with high-road architecture, which is meticulously designed, very beautiful, but difficult and expensive to change.
00:11:25.000
Building 20 at MIT was considered a temporary structure, built in the '40s for radar research in World War II. It was briefly designed in one afternoon and built in six months. Yet it lasted for 55 years. The exposed pipes and wires made it easier for students and faculty to modify the building.
00:11:58.000
They considered it the best experimental building ever built because they could cut holes in the floor or walls without concern. The Tech Model Railroad Club at MIT, a precursor to early MIT computer hacker culture, was housed there because it was a place where they could build their things.
00:12:30.160
This library in Boston, known as the Boston Athenaeum, has been around since at least 1915. In the book, Brand includes photos showing one of the reading rooms, where tables and chairs remain unchanged over a span of 40 to 50 years.
00:12:59.040
It turns out that if you know exactly what you want and are never going to change your mind, big design upfront is fine. But the issue is, we rarely have the luxury of knowing that.
00:13:30.000
You can imagine if you had to repurpose this space, how would you even approach it? It is built to be a library and would be difficult to repurpose.
00:13:58.120
Here’s another building at MIT, designed by renowned architect I.M. Pei. Brand mentions that there were problems with planning between the faculty and the architect. Initially, the researchers were too vague, and eventually, they became too specific, wanting rooms designed exactly around their current research without thinking of future use.
00:14:40.000
As a result, there are many unusable spaces in the building that are not conducive to being occupied by people. The concept of piecemeal growth is another thing examined by Brand, which is also applicable to software.
00:15:24.000
Structures often grow without a plan, much like how a final vision of a software project might not align with its original conception. The burden of past choices often weighs us down. Brand notes that buildings are a prediction and that predictions are always wrong.
00:16:00.320
Interestingly, authors like Mark Richards and Neil Ford have made similar statements about software architecture: that it is also a prediction, and we are often wrong.
00:16:24.000
As Brand explores solutions to the issues he discusses, he asserts that buildings can be designed so that it doesn’t matter when they’re wrong, which piques my interest as a software engineer.
00:16:55.120
One idea is scenario planning, wherein various potential future needs are considered in the design process. This helps designers create buildings that can accommodate different needs.
00:17:23.360
The architectural profession often employs procedures to understand what the client needs, what they want, and what they can afford. This brings to mind the poignant rule: whatever you think will happen, won’t; whatever you're ready for won't happen, and whatever you're unprepared for, will.
00:18:00.120
There are also useful illustrations of scenario planning used by a film company, which looked at growth potential versus market stability. They aimed to avoid the worst-case scenario of a large structure in a volatile market.
00:18:37.680
Another concept he discusses is building with change in mind. You don't have to hide your services inside walls; instead, if you want to change services or pipes, it should be straightforward.
00:19:16.840
This reminds me of software: the term 'connaissance'—meaning if one element is connected to another, a change in one necessitates a change in the other. We want to minimize this.
00:19:59.000
Another influential figure in architecture is Christopher Alexander, who suggested buildings should be designed to last 300 years. This idea makes me think about how I design software.
00:20:26.440
In the late '90s, connected to these patterns, there were conferences called 'Pattern Languages of Programs.' In 1997, Brian Foote and Joseph Yoder submitted a paper there called 'The Big Ball of Mud,' which many of you may have heard of.
00:21:06.680
Initially, I thought the term was just a humorous reference to poorly designed software. However, they were serious about their point, highlighting that we have many smart engineers who care about design, yet we often see a 'big ball of mud.'
00:21:43.920
Why does this happen? They identify several forces at play in creating big balls of mud: time, cost, experience, skill, visibility, complexity, change, and scale.
00:22:16.560
Time and cost are probably the most familiar and significant forces impacting software development. Deadlines, whether self-imposed or set externally, often lead to sacrifices in design.
00:22:45.680
Cost may relate to time, but it can also simply mean that the time to design and build an ideal version exceeds the budget, leading to compromises.
00:22:57.920
They also discuss experience and skill, which encompass not only inexperience affecting code quality but also varying levels of skill among team members. Even a seasoned developer may face challenges in a new domain without prior experience.
00:23:26.600
This raises questions about whether we learn from mistakes or if it’s necessary to make mistakes to learn important lessons. If all of us have to learn the same lessons repeatedly, it would hamper the overall development of our industry.
00:23:58.720
In response, what we hope for is that the mistakes we make tomorrow are different from those we've made yesterday. Visibility refers to the ugly code that often goes unnoticed by senior leadership, product managers, and users; they only see that the code works.
00:24:27.440
Complexity has been widely discussed in software development, as we prefer code that fits well within our understanding. Complexity is necessary, but confusion is undesirable.
00:24:50.480
There’s a real comment in a codebase I worked on, and while we all believe in the necessity of good comments, by the time I read this one, it was 11 years old, and the person who wrote it was no longer available.
00:25:29.960
Change in scale arises when we find ourselves needing to scale and are unprepared or trying to scale prematurely. The agile process encourages us to embrace change and adapt accordingly.
00:25:57.600
Brian Foote, one of the authors of the paper, tells a story about passing a house under renovation. He realized that building work on solid foundations is ultimately simpler than many of the complex changes we make in software.
00:26:32.520
In the paper, they identify some patterns or anti-patterns associated with the big ball of mud. Throwaway code, familiar to many, is a quick fix that often becomes critical to infrastructure.
00:27:01.920
Likewise, piecemeal growth refers to the incremental, ad hoc manner in which software evolves, while hotfixes, which address urgent concerns, rarely lead to necessary refactorings.
00:27:35.520
Another concept they mention is 'shearing layers,' where layers of software change at different rates, suggesting that components changing at similar rates should be kept together.
00:28:08.760
Sweeping it under the rug is another problematic approach—when software is too fragile or complex, it can lead us to avoid necessary changes entirely, resulting in an even scarier situation.
00:28:48.040
The last pattern they mention is 'giving up and ripping it up to start again.' Paul Buchheit, known for leading the Gmail project, shared his realizations about what 'good enough' truly means, conveying how sometimes this mentality can lead us down a problematic path.
00:29:29.760
This reconnects to Richard Gabriel, who contrasts two software development ideologies: the 'right thing,' emphasizing correctness, and 'worse is better,' emphasizing simplicity over other attributes.
00:30:05.040
Overall, there's a tension between aiming for the best quality and the need to innovate quickly. Kent Beck's advice to make software work, make it right, and then make it fast creates a scenario where we often prioritize short-term needs over necessary improvements.
00:30:41.000
The question becomes whether big balls of mud are inevitable. While it may be necessary to create some disposable products to maintain progress, we should not let that philosophy become accepted.
00:31:22.520
To reduce the impact of these forces, we must hold conversations, perhaps by establishing book clubs that discuss these topics. If we neglect these discussions, we risk losing sight of challenges and progress.
00:31:45.720
There's an effectiveness formula noted in a book called 'Improving Software Development Productivity,' which suggests that effective communication and strong management are critical. Even with high skill levels, poor communication or management can drag down overall effectiveness.
00:32:27.760
Communication can be improved more readily than technical or management skills. As a speaker at this conference, I've had multiple opportunities to converse about these topics.
00:33:09.040
Rich Hickey once mentioned that all he wanted from his talk was for attendees to go back to their teams and discuss the ideas.
00:33:48.920
Pushing back can take the form of mentoring, ensuring more experienced members support the inexperienced, and making the ugliness in our code visible to others—whether through metrics or other means.
00:34:56.199
Scenario buffered strategies prompt discussions about the various possibilities, though they can become complex without clear, collective understanding.
00:35:28.999
Understanding our tools helps us design code that remains flexible under changes, much like ships have maintenance logs for ongoing conditions.
00:36:12.000
Chesterton’s fence illustrates the necessity of understanding the rationale behind existing systems before deciding to change or remove them.
00:36:52.000
When we write software, we often oscillate between viewing it as either temporary—ready to be disposed of—or as permanent. This perception often leads us to mismanage our progression with software.
00:37:43.200
Brand’s observations about buildings and their adaptability invoke relevant parallels to software systems. While his solutions focus on buildings, they resonate with many aspects of software design.
00:38:26.000
Ultimately, there’s no one-size-fits-all approach. The key takeaway is that accommodating change and understanding growth become pivotal as we navigate the landscape of software engineering.
00:39:09.600
Thank you for listening, everyone!