00:00:31.000
I want to start off with an observation that I think reflects something neat about our world and our industry, particularly this little Ruby community we have. I first gave this talk a few months ago in Stockholm, Sweden, and there are at least two people in this room today who were also in the room then. I find it kind of neat that I can travel to another continent to give this talk and come here to see some of the same people. I hope that we, as a community, continue to make and take opportunities to see each other and talk face-to-face. I think that's a good thing.
00:01:10.479
This talk is somewhat personal. It's about a strategy and a philosophy that has shaped my career and has served me really well. It's one that I've tried to teach to the programmers I've had in training classes or had the opportunity to mentor. Since it's a personal talk, there will be a lot of stories.
00:01:30.360
The first story happened in late 2005 when I was shifting from toying with Rails on the side to using it full-time and becoming a full-time Ruby programmer. I had a conversation on the phone with my friend Mike Clark, or my Ruby friend, Mike Clark, to be precise. Mike was telling me about his Rails project—he was a bit farther down that path than I was. He said, 'I really like this project; you should find yourself on a project like this. It's giving me a chance to deep dive and look inside Rails as an internals.' Something he said struck me immediately and I’ve never forgotten it. He said, 'I like to visit all the turtles.'
00:02:19.160
I knew immediately what he was referring to; he was alluding to the old joke about the man who insists that the Earth rests on the back of a giant turtle. When a skeptic asks, 'But what does the turtle stand on?' the old man replies, 'You're very clever, sonny. It's turtles all the way down.' This phrase has entered our vocabulary. We've even heard it once during this conference in a talk yesterday. In our field, 'turtles all the way down' refers to the layers and layers of abstractions that we sit on top of when we work today, each one building on top of the other.
00:02:58.640
Various kinds of purists, especially from the Smalltalk and Lisp communities, like to use that phrase to emphasize the consistency that all the layers in their platform share or that they are composed of the same kinds of things—until you reach the layer where they don't, of course. If that’s what you care about, you might do something as drastic as have the Y combinator tattooed on your forearm. This is one of the amusing results you get when you Google image search the term 'turtles all the way down.' However, most systems and platforms aren’t that consistent. You have layers built upon layers, and they’re all different; they’re made of different things and are aimed at providing different kinds of abstractions.
00:04:03.239
So, how much of this do you need to know as a programmer? Is it possible to be happy and productive as a programmer while only understanding the level of abstraction that you’ve chosen to work at? For most of us in this room, it’s probably Rails. How much do you need to know about what goes on underneath? That question takes careful thought because one answer might be 'nothing.' But I’ll explain why I don’t think that’s the right answer. It can’t be everything, either because there’s just too much complexity out there.
00:05:00.960
Eric Sink, a well-known Windows and .NET developer, wrote a blog post recently where he attempted to map out all the different layers of abstractions that programmers have to build on top of when they work. I don’t like putting things on slides that are false. He mentions that abstractions reduce performance, and while they undoubtedly always introduce bugs and increase overall complexity, sometimes abstractions, if well chosen and well implemented, can serve as loci for optimization that otherwise wouldn’t be possible and actually make things faster.
00:06:12.759
In any case, Eric Sink wrote this blog post about the net abstraction pile, and it kind of goes on for quite a while. You can’t learn all that stuff. When people discuss this, they often refer to a famous 2002 article by Joel Spolsky called 'The Law of Leaky Abstractions.' Unfortunately, most people remember only the term 'leaky abstraction.' We complain about abstractions that leak, and some abstractions leak more than others or in different ways—worse or better—and that’s mostly what we debate.
00:06:45.640
As an example, I came across an argument on a Stack Overflow post where a friendly coder was mildly impacted by the law of leaky abstractions. It was very hard to convince him to use any new framework or toolbox at all. He’s missing the point that anyone who’s ever read... oh, I missed that slide! This is what I get when I modify slides the day before. There’s a book by Charles Petzold called 'Code,' which starts at binary numbers and builds up. If you’ve seen that book, you’ll know that even machine code is an abstraction atop other elements.
00:07:19.560
The real point of Spolsky’s article was that all non-trivial abstractions are to some degree leaky. Just because an abstraction leaks doesn’t make it bad; it simply defines it as an abstraction. The question is: How do they leak? Do they leak in transparent ways or opaque ways? Typically, the worst way that abstractions leak is when they break, as Neil Stevenson pointed out. It’s often the case with technologies that you gain the best insight about how they work by observing them fail.
00:08:06.560
That insight can be fascinating when you’re trying to understand how something works inside. However, most of the time, that’s not what we are trying to do; we want to make something else work, and then it fails. You’re then greeted with a stack trace from the bowels of the system, employing terminology that bears no relevance to the level of abstraction at which you think you’re working. It’s akin to when a clock breaks, and you suddenly find yourself facing gears, cogs, springs, and other elements that have no intrinsic relation to the concept of time.
00:08:44.880
This situation is easy to understand because someone creating a library implementing an abstraction is typically more focused on making it function correctly than on ensuring it can fail gracefully. It’s difficult to predict all the potential ways people could misuse your framework, and when the implementor encounters a failure, it doesn’t strike them as jarring because they understand internally how it operates. Thus, the stack trace or whatever isn’t as impenetrable to them as it is to you; it doesn’t feel like such a big deal.
00:09:12.440
Good abstractions should aim to address this. Jim Wyrick recently noted that you can gauge the maturity of a framework by the quality of the error messages it produces. I think that's accurate; however, in practice, the quality of said error messages usually varies from appalling to merely bad. Spolsky's main argument was that, especially in 2002, it would be nice to teach someone about programming using visual studio wizards and all the code generation features. But if anything goes awry, they won’t have the faintest idea of what happened or how to fix it.
00:10:02.120
With that long introduction, this talk is about visiting all the turtles—why and how, the importance of knowing what occurs beneath the first level of abstraction, and how to approach that knowledge. It’s time for another story. My second full-time job after college was at the University of Texas at Dallas. Back then, university computing departments were excellent places to learn a wide array of information. Today, there’s really only one kind of computer, but at that time, there were many different types.
00:10:28.279
You’d get bizarre and quirky pieces of equipment donated by alumni who had gone to work for various computer companies. Since they were donations, it wasn’t an option to decline and say you didn’t know how to use them. So, you took these donations and made them work. I worked closely with a fellow named Billy Barron, and we quickly gained a reputation within our circle of programmers and system administrators as the go-to pair for resolving issues when everyone else was stuck. Not that the others were ineffective; there were many talented individuals among us.
00:11:13.560
However, Billy and I were often able to figure out problems quickly, so an enterprising student who worked for us was keen to learn our secret. He independently asked both of us how we learned all this information and how he could apply his knowledge similarly. Interestingly, both Billy and I related our experiences back to our very first mentor, David Lipkey. Even though we attended different schools and had never met before, we ended up at the same job thanks to shared mentorship.
00:12:02.600
David Lipkey was the Executive Vice President of Operations at AOL from around 1993 to 2003 and was the mastermind behind AOL Instant Messenger. What he taught us over and over again was to never be afraid of exploring what seems complicated or intimidating; it’s not as opaque as it appears. You’re capable of understanding it, so open it up, look inside, and figure out how it works. Around the same time while working at UT Dallas, I became one of the first outside of Sun Microsystems to get my hands on a copy of Java, which at the time was still known as Oak.
00:12:52.920
I learned Java quite early on, and compared to the alternatives, it looked appealing. As a result, I was tapped to write a book about it. For those who remember 1995, one major excitement surrounding Java was applets, which included a security model for untrusted code built into the Java Virtual Machine. My book outline proposed a chapter dedicated to the security model that was entirely undocumented. However, at that time, an almost unique advantage for Java programmers was that it came with a zip file containing the source code of its standard library.
00:14:07.360
So, I figured I could simply crack open the necessary elements of the Java source code, no sweat! However, to my surprise, I discovered that a small subset of Java library classes did not have the source code available, and the security manager was one of them. Now, alert viewers will probably ask, 'Why are we seeing an elephant and not a turtle?' The answer is that Rincewind and the elephant are standing on a giant turtle. I had committed to writing this piece on the Java security model, and discovering the absence of source code was a setback. As I pondered this, I decided to work on the rest of the book and save the security chapter for last.
00:14:55.680
Eventually, Java provided a disassembler for its class files. So, during the Super Bowl in 1996, I found myself on the couch, surrounded by printouts of the disassembled bytecodes of the Java security manager class, accompanied by the Java Virtual Machine specification. During that time, I essentially learned to read these bytecodes as if they were source code. While I wouldn't say that this skill has come in handy often, it has proved useful more times than you’d expect. On about five or six occasions throughout my career, the insights I gained from that Super Bowl session resolved critical issues with proprietary Java code, showcasing the value of not being afraid to explore.
00:15:38.799
Here's my first point: Don't be afraid. The second point is to make it easy to explore. When I first started with Rails, I struggled with the fact that Rails wasn’t just a single package; it was a bundle of different packages. I understood the rationale, but when I was uncertain about which part of the system was defined in Active Record, Action Pack, or Active Support, I found it challenging. Like everyone else at that time, I was using TextMate, which had this nifty projects feature. By defining my own project and including all the Rails gems, I could perform full-text searches across the entire Rails codebase, helping me uncover its implementation.
00:16:36.680
There was nothing magical about creating that setup; I was simply trying to reduce the barriers to exploring the source code. Many people didn't enjoy Rails because of its apparent magic, but after two months, I began to understand how things worked. What once felt like magic quickly became ordinary once I discovered its inner mechanisms. With a solid understanding of what goes on under the covers, I could write correct code when using those magical elements, thus saving time and effort in my coding.
00:18:07.280
Point three is to choose tools that make sense. Prior to working with Rails, I had extensive experience with an object-relational mapping library that I won't name. It attempted to create a perfect abstraction around the database to the point of inventing its own query language, completely obscuring the database. Unfortunately, that abstraction was so intricate that configuring it to recognize your schema could be challenging. Subsequently, while everything looked like objects after configuration, my friend, the author of that framework, was utterly disgusted when I began diving into Rails due to the database's transparency.
00:20:14.560
The lesson I extracted from this experience is that if you attempt to build a perfect abstraction that clogs up the details, as Joel's law suggests, it will inevitably leak in subtle and insidious ways, often catching you when you least expect it. Conversely, I found value in Active Record's intentional leaks because they are easy to comprehend, allowing me to refer to it as an 'onion skin API,' where lower-level functionalities are accessible, with higher-level abstractions built on top—all well documented and available to end users.
00:20:54.920
Can anyone recall another system with similar characteristics as Active Record? Unix is one! Yesterday, a systems programming talk demonstrated a deep dive beneath some standard layers of abstraction we’ve learned to accept. Even if you never do that, understanding how core components work, like Unicorn, is beneficial. Now can anyone think of another frequently used system with this onion-skin quality? Git is a notable example. In fact, components of Git are categorized into porcelain and plumbing, where the plumbing is still documented and accessible for those needing to explore.
00:22:29.960
Tools that exhibit such transparent qualities allow you to understand what’s occurring underneath without needing to disassemble bytecodes and decode them on your couch. There’s a vital point tied to this: reading documentation is essential for making sense of your tools. Raise your hand if you struggled with Git the first time you encountered it? Keep your hand raised if discovering the documentation that elucidated some of Git’s conceptual foundations helped you understand it better. Many of you found that once you learned about its foundations, Git became clearer and less daunting.
00:23:08.120
Numerous tutorials and documents aim to clarify Git by simplifying instructions for daily tasks. That approach is adequate until something unexpected arises and you find yourself in a panic with no idea how to recover. Many found it beneficial to delve into the documentation that explained the deeper concepts behind Git, which eased their learning. Analogous to learning Vim—a tool I personally favor—when I used it, I attempted to learn it through typical tutorials, which taught commands disconnectedly. This approach often fell short because advanced users recognize that Vim commands comprise a language.
00:24:01.760
True productivity with Vim comes from grasping its underlying grammar involving verbs, nouns, and modifiers. This resource provides clarity and structure, offering a conceptual framework for all Vim command sets, making it significantly easier to process. Therefore, I encourage choosing tools and documentation that promote understanding. Try building your own tools is a strategy for someone striving to become a master.
00:25:12.440
I recall years ago at a conference, Chad Fowler delivered a brilliant talk about Active Record titled 'Active Record: Easy Living on the Golden Path.' He began with an empty TextMate window and progressively built a model class, incorporating persistence operations directly into it. He gradually abstracted these elements to derive a class that resembled a typical Active Record model and summarized the simplified internals of Active Record. Although developing a completely new tool such as this may not be feasible for most of us, even constructing a simplified version can teach valuable lessons regarding our pre-existing tools.
00:25:51.360
A great example is a screencast by Jim Wyrick titled 'Source Control Made Easy.' He explains how to conceptually build Git from fundamental principles rather than the code level, systematically tackling each problem until resembling Git’s command set. Another noteworthy example is the tale of the creation of Rake. Jim told me he accidentally wrote Rake one afternoon, spurred by dissatisfaction with Make. In a lighthearted conversation, exploring what it would look like if written in Ruby, they attempted to develop an initial prototype. Within an hour, version one of Rake emerged effortlessly.
00:26:29.680
Thus, I recommend trying to build something yourself. The final point I’d like to address is the common sentiment I encounter when discussing this topic: 'My employer isn’t paying me to learn about what’s happening under the hood of these technologies or to explore the source code of Rails.' I strongly disagree; this is an inherent part of your job. A true professional comprehends their tools. Although your employer may not fully grasp why this is crucial, it is undoubtedly part of what you’re being compensated for.
00:27:04.320
Reflecting on my experience reading Java disassembled bytecodes as if they were actual source code, while I mentioned that it isn't a frequent necessity, when it has been useful, it made a significant impact. Many of you can likely pinpoint individuals within your organization who act as invaluable resources when someone encounters a confusing stack trace arising from deep down in the framework or from some dependency. These are the people who possess this knowledge, and they are treasures to your team.
00:27:40.320
A company dismissing this learning process is essentially trading future growth for minimal short-term productivity advantage. While exercising discipline is essential—it’s easy to waste time fixing code—it is also vital to understand how your tools work beneath the surface. You and your employer must accept that understanding these fundamentals is an integral part of your role. There will inevitably be days when exploring these aspects won't be enjoyable. Upon examining the tools you utilize, you may uncover structures so flawed it's astonishing they hold together. Some encounters will reveal code that looks like monstrous entities from a horror show. It’s crucial to persist through these discoveries because it contributes to your professional maturity.
00:30:06.720
In summary, this exploration leads to understanding and professional development, and ultimately contributes positively to your career. Thank you.