The World Runs On Bad Software

by Brandon Keepers

In the talk titled "The World Runs On Bad Software," Brandon Keepers addresses the pervasive issue of poorly written software in the tech industry. He begins by acknowledging the emotional toll that recognition of bad software takes on developers who pride themselves on craftsmanship. The discussion highlights how despite the existence of bad software, the world continues to function, which leads to a deeper exploration of the balance between pragmatism and perfection in coding practices.

Key Points Discussed:
- The Reality of Bad Software: Keepers emphasizes that bad software is ubiquitous, and despite its shortcomings, it often suffices for business needs. He questions what a world of perfect software would look like and considers the implications of waiting for high standards before launching products.
- Defining Good Software: He defines good software as that which meets an objective at a reasonable cost while managing risk, suggesting that adequacy rather than perfection is often sufficient.
- Causes of Bad Software: He identifies the concept of entropy in software development, explaining how software tends towards chaos due to complexity, malleability, and team dynamics. He mentions the immaturity of the software industry compared to more established fields and the inherent challenges of managing a large codebase.
- Building Better Software: To counteract bad software, Keepers proposes several strategies including user feedback, automated testing (viewed as an insurance policy), and the DRY principle (Don't Repeat Yourself). He advocates for incremental refactoring instead of rewriting code wholesale, which he believes can be more efficient and manageable.

Significant Examples:

- Keepers shares anecdotes about real-world projects he's been involved in, such as the challenges faced with "Words with Friends" regarding scaling issues and the impact of quick performance optimizations on code maintainability.
- He humorously presents certificates throughout his talk for various examples of 'bad software' to engage the audience and illustrate points about coding practices.

Conclusions:

- Keepers encourages developers to embrace a pragmatic approach that balances the need for quality with the realities of software development. He stresses that writing 'good enough' software can lead to happier users and productivity for developers, promoting the importance of discipline in coding practices.

00:00:19.609 My name is Brandon Keepers. I work for a company called Ordered List, and today I'm going to talk about something that we're all somewhat familiar with: bad software. As you're getting ready to do anything in life, it seems like everybody's always willing to give you advice.

00:00:25.830 As I talk to people about speaking at this conference, everyone kept saying, 'Oh, you have to be funny. If you're not funny, then people will stop paying attention.' Second, you have to use some kind of meme in your slides, like pictures of cats or the 'Y U No' guy. So, I thought I would go ahead and get that out of the way right now, and then we can get on to the content.

00:00:40.890 So, I'm from up north, like Michigan, where it gets cold and freezes in the winter. This weather is hot; I can't handle it. You can find me on Twitter as @beekeepers. I'd love to hear your feedback or thoughts on this presentation. I by no means know everything about bad software; I'm good at writing it, but that's about all I know. So feel free to let me know what you think. I also blog at opensourcely.org if you want to follow along with some of the things I've been working on or thinking about.

00:01:12.950 Today, I'd like us to have a little bit of a group therapy session. It seems that this is a topic we all love to complain about. If this is emotional for some of you, you're welcome to come up afterward, and I can give out some hugs or whatever you need. Alright, so let's get on.

00:01:43.200 About a year ago, I had this realization: everywhere around me was really bad software. It’s not just the kind of stuff that we like to complain about, but rather the world runs on bad software. The software that runs the systems we use every day is surprisingly fragile compared to other public infrastructures, like highways or buildings. These are all systems we completely depend on, and like I said, this is a topic that we love to hate.

00:02:11.550 We all have stories about bad software. So, if we can, let's start. Let's see who has the worst experience with software. Does anyone here think they have the ugliest app they're maintaining? Raise your hand.

00:02:28.200 Okay, we have a couple of hands. Someone, prove it to me. Tell me about your app. How big is it? 3,000 line functions? Wow! What about the user interface? Does anyone have a really ugly interface? Who thinks they have the ugliest? I need one person.

00:02:44.310 Alright, Twilio? No, you guys don’t have the ugliest. Oh! Alright! Well, I have something for you to take back to them. This is a certificate of bad software. You can tell them that their software is so ugly that their users vomit and that they should have something prettier to look at.

00:03:10.350 This realization of bad software is something that was hard for me to swallow. I mean, I take a lot of pride in what we do in our industry. We may have started with underprivileged hackers in mom's basements, but now we've grown up. We've become professionals, and we’re valued members of a world that everyone depends on.

00:03:49.739 Like probably many of you, I consider myself somewhat of a craftsman. I don't do this job just because it pays the bills, although that's nice. I don’t do it just because I’m good at it, and there’s nothing better to do. I actually do it for the art; I love software, I love code, and I love the difficult challenges that feel impossible to solve until, all of a sudden, you somehow crack them.

00:04:12.780 So, I’m motivated as much by reputation and what other people think of me as I am by the desire to impress myself. Like I said, I work at Ordered List, where we take great pride in our software too. We have a bunch of products. Harmony is a hosted website management system. I’m not trying to promote our services or products here, but Harmony is awesome. It's flexible and well-architected, helping people build websites without all the complexity of many other options.

00:04:47.500 We’ve got Gauges, which is a real-time web analytics package. This app was a ton of fun to write; there were interesting problems and lots of data coming in. We launched it earlier this year, and it’s been going pretty well. We also have Speaker Deck, which is something we have no idea how we’re going to make money on, and we don’t really care at this point.

00:05:19.450 Speaker Deck was created because we got frustrated with all the slide-sharing sites out there. So, we wanted something simple, clean, and beautiful. Plus, we spent the last year working on a project called "Words with Friends," which is now owned by a company called Zynga. When we came in, Words with Friends was struggling to scale. They couldn’t handle the amount of traffic they were getting.

00:06:02.170 Over the last year, we’ve worked with them to handle nearly a million requests at peak load, which is quite significant. Recently, we launched an HTML5 version of their Scrabble-like game for Facebook.

00:06:28.110 I tell you all these things not to impress you but to convince you that I love good code. Anyone that's worked with me knows that I become paralyzed anytime there's anything ugly or nasty within the viewport; it’s quite a debilitating disease. However, even these things—the products that I talked about and the clients we’ve worked for—are far from perfect.

00:07:02.530 In fact, a lot of them have really bad code, and some of the code in there makes me very ashamed. Realizing that I’m a part of this industry that produces bad software has been a bit of a struggle for me. Once I got over the emotional trauma of that, I had this interesting thought process. I realized that the world hasn't imploded yet, and hopefully, it's not going to be our fault if it does.

00:07:29.400 But the world actually runs on bad software for the most part. There aren’t any major issues with it. We somehow manage to navigate the inconvenience—a flight getting canceled or some other issue related to bad software—but we're still able to do productive things.

00:07:51.729 So, it makes me wonder: What would a world of good software look like? What would happen if we waited until the software was at whatever standard we expected it to be before we launched it? That's kind of what I’d like us to explore a little bit more today.

00:08:04.479 In spite of that bad software, I think we are where we're at because of it. Today, I want to look at this balance between pragmatism and perfection and see what that means for the way we write our code.

00:08:21.460 We'll start by looking at what is good software, how does it go bad, and what we can do to build better software. Let’s kick things off with my definition of good software: good software meets the present objective at an acceptable cost without incurring an unacceptable amount of risk. There’s a lot of jargon in there, but basically, it does what we want it to do, it doesn’t cost too much, and we don’t go broke or get killed in the future.

00:09:03.970 Another definition could be adequacy. It’s adequate for the task that we're asking it to perform. And by this, I don’t necessarily mean mediocrity; it means that it’s good enough. So, let's look at Objectivity: one of the hard things for me to accept is that software is not just code. Software is so much more than code. As a developer, it’s easy for me to get stuck in the details and make sure the code looks perfect without stepping back to think about: what’s the experience of the software?

00:09:50.789 A quote by Zach Dennis states, 'People fail to realize that quality is merely a property of the code that exists, not the reason for its existence.' As developers, especially, I think this is something we must keep in mind. Unfortunately, users don’t care what the code looks like. I wish they did. I wish I could show them: look how beautiful this is, it works so well. But they don’t care; they care that it works. Scott Berkun said, 'We are most frustrated in life by things that come close to our deepest needs but don't deliver.' I think that’s really true.

00:10:23.599 If I think about the things that frustrate me the most, it’s not the things that are buggy; those are annoying, but you can learn to work around that. It’s the things that just don’t meet your needs. Next, let’s talk about cost.

00:10:36.100 Virtually all the costs of software are directly or indirectly the cost of design. I think that’s a really interesting idea for an industry that’s built around fabrication—which is what software is. We’re producing either tools or products consumed by other people, and we have almost zero cost of production. We have a little bit of expense in the laptops that we code on and the infrastructure we use to run our services, but we don’t have to manufacture parts or face any expenses related to that.

00:11:15.549 However, design costs money. Writing code costs money—that’s how we work. Obviously, we're all here doing this professionally. We either get paid directly from clients, either in the form of an hourly rate or a salary. So, what software does is figure out what we want to build, define the objective, and then keep iterating on it until we run out of money. The hope is that by the time we’re out of money, we have something that meets our starting objective.

00:11:54.889 Next, let’s talk about risk. I know many people will probably disagree with me, but for me, risk is not a science at the scale we’re operating on, which is actually much smaller compared to federal government jobs or huge corporations. We don’t have time to calculate the risks. We can make educated guesses, but basically, to lower the risk in the future, we have to increase the cost now.

00:12:21.910 So what does that mean? If we want to ensure the software won't break at a future point, we’re going to spend more time testing at deeper levels. Maybe we’ll write unit tests, functional tests, and integration tests. This is the trade-off we’re constantly trying to make, and it’s one of the aspects that I think goes into good software.

00:12:55.820 So again, what is good software? It meets the objective at an acceptable cost without incurring an unacceptable amount of risk. So, why does software go bad? I think the main reason is entropy. Entropy is this thermodynamic property, and while there are lots of aspects to it, the main idea is that in an enclosed system, things tend towards chaos. Pragmatic programmers say that while software development is immune from almost all other physical laws, entropy hits us hard.

00:13:33.679 Why is that? I have a couple of ideas, but it seems like the hardest part of my job is ensuring that things stay orderly. My first idea is that software is malleable. This is a good thing; we have zero cost of production. We can write code, see if it meets our needs, and if not, we can keep changing it.

00:14:02.600 We continue to modify software as our requirements change because change is really cheap for us. So, we build something, test it, and then we change it. The hard part, though, is that every time we touch code, we have the risk of introducing complexity. Unless we’re really intentional about minimizing that, it’ll eventually continue to descend into chaos until it’s almost impossible to recover from.

00:14:33.500 Another difficulty arises when more and more people touch the code. If you’re the only one working on it, you can keep the entire system in your mind. Every time you go through it, you know what your intention was. It’s easier to keep it clean. But as you add more hands, the style and ideas about how the system should work can differ. So a lot of people working on it increases the likelihood of greater complexity.

00:15:13.590 Out of curiosity, how big are the teams you guys are working on? Is anyone working on a team of five or fewer? Wow, that's actually impressive. How about more than that? More than ten? Like ten coders or ten people committing to the code? Twenty?

00:15:26.110 Looks like you're the winner! I have another certificate for you: you have the Bad Software Award for too many cooks in the kitchen! Make sure you get that framed.

00:15:44.110 Another reason that software easily succumbs to entropy is that it's extremely complex. Every line of code is essentially a moving part. Think about some of the more complex things in the world, like a car. A car has 30,000 moving parts. It’s not unlikely for a Rails app to have 30,000 moving parts too. There’s an app that we've been working on lately that has 28,000 lines of code. If each one of those lines of code is out of place or poorly structured, we increase the chance for the system to break.

00:16:19.180 We’re dealing with things that are of immense magnitude compared to manufacturing a chair or table. It’s impossible to keep the entire system in your head. Take the Space Shuttle for example, which has two and a half million parts. I’ve heard of apps that have two and a half million lines of code. This is extremely complex stuff. Does anyone have an app with that level of complexity? No.

00:16:51.060 Does anyone have an app with more than 50 thousand lines of code? Whew! Seventy-five thousand, two hundred thousand? Let’s go in bigger leaps here. Alright, back in the corner, guess what? You get a certificate too for bloated software! Come up and grab this.

00:17:18.800 Another reason I think software goes bad is that our industry is actually quite immature. On the grand scheme of things, we've only been around for about 50 years. Other disciplines, like math and science, have a lot of history behind them, and they have expertise and knowledge that’s been passed down. While some of that knowledge is changing, it’s not changing as fast as our industry.

00:18:05.490 If I want to build something physical, like a bridge or a chair, I have to go through a process before anyone will trust me to do that. I have to go to school, learn all the tools of the trade, and even after school, I have to climb the ladder enough to earn enough respect to design and manufacture it. In software, we have teenagers writing code which is awesome, but it also means we don’t have the shared knowledge that other industries possess.

00:18:44.110 I think we’re more open than other industries, sharing in a more organic way, but we lack the formal education that other industries have. In this room, let’s find out: who’s the youngest person here that writes code? Is anyone under 20? No? Anyone younger than that?

00:19:21.570 Alright, we have another certificate: congratulations for learning to tie your shoes! I hope there aren't any civil engineers that are 20 years old. It’s amazing power, that at 20 years old, someone could be working professionally building software.

00:19:57.230 Not to say that young people are necessarily the main culprits for writing bad software, but I think they represent a part of the problem. Another reason is that software is opaque compared to other things in the real world. Most of software is hidden behind layers and layers of complexity; very little of it is actually visible.

00:20:21.890 We might have some thin user interface, but most of it is hidden from anyone except the people that wrote it. As long as each layer responds properly, the software can continue to get more and more chaotic. We have these APIs that are defined, and as long as a method gives the same output with the same input, it doesn’t matter what it looks like inside.

00:20:42.130 Finally, the last reason is that software is incredibly durable. Given all these other challenges, software doesn’t degrade over time. With physical things, if I made a chair that was inadequate, it would eventually stop working. But software, as long as we’re giving it the same input, will continue to provide the same output.

00:21:10.890 You can call the same bit of code, no matter how messy it is, over and over again, and it will continue to work. It’s actually kind of unfortunate that it works that way because if it actually degraded and eventually just broke, it might lead to better maintenance.

00:21:49.950 If I don’t touch an app for three years, and it’s on Rails 1.2, it’ll be a pain to handle. Sure, you can bring it up to Rails 3. But that 1.2 app will probably still perform. I know a couple of 1.2 apps that I wrote, and they’re still out there, which scares me.

00:22:26.840 Who here is still working on code that they wrote three years ago? Five years ago? Wow! How about 15 years ago? Seriously? Twenty? It's not like it's Rails, is it? Alright, for you, you wrote code when dinosaurs roamed the earth! Congratulations, that’s awesome!

00:23:08.230 It’s interesting to think about software and its durability. Knowing all this, how do we build better software? How do we overcome some of these obstacles? I don’t have any great answers for you, just a few things that work for me and that I think are important.

00:23:49.320 I know that these are only a few aspects, and there are tons of other things that are also important. The first one is that we can’t meet objectives unless we actually talk to the people that will be using the software. I don’t mean send in some account exec and have them make a 20-page document that outlines everything.

00:24:17.200 As you're writing code, put it in their hands and watch them use it and let them react to it. Some of the most rewarding projects I’ve worked on recently have been those where I could write code in the morning, deploy it, and by afternoon, see ten million people using it. So you know immediately whether it’s working for them or is broken.

00:24:50.750 Even if you don’t have ten million users to try it out, asking for feedback is incredibly valuable. This point likely goes without saying in the Rails community, but another way we fight entropy is through automated testing. I think it was Glenn who mentioned this morning that sometimes we in the Rails and Ruby community can be unreasonable about it.

00:25:17.870 We say, 'At all costs, you must test.' I’m not advocating that we need to enforce this, but I am saying that if we want to find the balance between risk and cost, testing is a crucial part of that. To me, I see testing as an insurance policy.

00:25:51.930 There are aspects of testing that help you design and navigate through the process of figuring out what to build, but for me, I do it so that when I change things two weeks from now, I'm not worried about if what I just wrote breaks.

00:26:23.200 Some people might argue that testing takes too long or it's not worth it. To those people, I would ask, 'Why are we even programming in the first place?' Essentially, you're saying that computers aren’t good at automation, and that it’s inefficient to automate the testing of your application.

00:27:03.510 I think that’s a poor excuse. I understand there are certain constraints, such as needing to have something done by the end of the day, but in the long run—even within days or weeks—testing pays off.

00:27:49.470 A little more audience participation here: who is currently maintaining an app that doesn’t have tests? How long has that app been in progress? One month? Six months? I think you’re the only one. It’s okay, you can admit it! Realize that it was your co-worker’s fault that it doesn’t have tests; you can give them this certificate for preferring to bungee jump without a cord!

00:28:27.680 Another thing we often talk about in the Ruby community is DRY: Don't Repeat Yourself. I realize these are simple and obvious points, but we violate them all the time. Every time you copy and paste code, you're essentially doubling the amount of work you'll have to do next time you touch that software. I wish I could say I never do this, but there are times when it seems faster just to copy and paste.

00:29:12.540 Does anyone have a co-worker who likes to copy and paste code? Anyone? Oh, Glenn, I saw that! Go back and give them the certificate for only knowing how to code using the C and V keys!

00:29:52.260 The last point is one that I need to hear pretty much daily when I come across neglected code: my inclination is to throw it all out and rewrite it. I spent two weeks rewriting a bit of code that sent notifications, either push notifications or emails, to users. There were about eight different places in the code where we were sending Apple push notifications. For certain users, we wanted to send them emails instead.

00:30:41.100 Initially, I had a few if-statements. For example, if the user was using an iPhone, we would send a push notification; otherwise, we would send an email. A few days in, I realized this was a really bad pattern. I had all these conditionals. What I needed to do was abstract this out into a notification class that knows whether the user needs a push notification.

00:31:35.970 It took me two weeks to write the code that sends an email, and I could have probably done it in two days had I not been so complicated with conditionals. The point being, small incremental refactorings: anytime you go in to touch code, change one thing at a time and commit it. If the method takes four arguments and you want it to take three, just fix that; change it where necessary and commit it.

00:32:00.580 If you also want to rename the method, don’t do it at the same time. Commit it and come back to rename it later. These may seem like common sense, but I think they’re essential aspects of good software.

00:32:43.530 So, we’ve defined good software, examined what makes it go bad, and talked about simple ways we can make it better. I want to give you a few examples of some code that I’ve seen recently or worked on and let you decide if they represent good software.

00:33:25.267 The first example is a method that was missing. A couple of years ago, a client of ours in Australia asked if we could come out and do a couple of days of Rails training. At the end, we were to help them evaluate their current Ruby codebase. They had this GTK desktop application written in Ruby. During our review, one of the first things I found was a function that returned nil regardless of the method called on it.

00:33:54.309 When I asked them what in the world this was doing in the app, they said they kept having problems where users would click on something, and the app would crash. So, I said, 'Okay, so to you, that was the acceptable solution? You make it so the app never crashes, but what did that gain you?'

00:34:14.227 They replied, 'Well, then someone will call us and say, “I had a problem when I clicked on this; it doesn’t work.”' So, here's what you get when you do that.

00:34:41.192 Has anyone seen anyone do something like this? You can take this certificate back to them: 'You can’t tell good code from the broad side of a barn!’

00:35:05.167 Occasionally, when writing code, we need to optimize for performance over maintainability. I hate bringing this point up because I’m not ready to accept it. However, there have been very few cases where we’ve had to do this.

00:35:29.687 This is some code from the Words with Friends app. It’s an XML API that talks to an iPhone client. This method was getting called thousands of times a second, and our benchmarks showed that we could save three to four milliseconds if we coded the XML by hand. So, we defined this on our ActiveRecord model to call to XML. This pains me. I still give a hard time to the person who wrote this code every time it comes up.

00:36:14.400 I think, like the other example, the overriding nil is not maintainable or pretty, but it accomplished what we needed at that moment: to decrease the response time by a few milliseconds.

00:36:35.473 I lost track of time when my battery died. Does anyone know how much time I have left? Ten minutes? Okay, thank you!

00:37:22.150 Another example: I’m one of the maintainers of Zuv Mapper, which is a Ruby mapping library for MongoDB. Last fall, we were working on porting a bunch of custom code that we had written to ActiveModel in Rails 3. We encountered a challenge relating to callbacks on embedded documents.

00:37:56.650 In Rails 2, if you wanted to define callbacks, the original code would look like this: we would run the before callbacks and if the result of the save returned true, very carefully, we would then run the after callbacks. Rails 3 changed this a little because this pattern is somewhat obnoxious and requires you to do this anytime you want to add callbacks.

00:38:28.690 They made the API cleaner by allowing us to just pass a block. The problem arose with embedded documents. You could have a root-level record and inside that record in the same database, you could have basically a many association. We wanted those callbacks to propagate to those embedded documents, but there wasn't an easy way to do it.

00:39:02.780 Here’s what we came up with: we would override the 'run callbacks' method and loop over each of the embedded associations, sending the call back to propagate on them. While this was clever and worked—tests passed—the issue arises if you have more than 100 to 200 documents; you eventually get a stack level too deep since we are creating one giant stack of these lambdas.

00:39:32.880 The question then becomes: is this good software or bad software? For me, it hasn’t broken yet. I’ve never had a document with more than a handful of embedded assurances. We’ve had people run into issues because of this and post on the mailing list. I think this illustrates how good and bad software can be a bit tricky because it’s not black and white.

00:40:09.700 What I’m asking you to do is be pragmatic. Good software is important; it’s something I’ve struggled with a lot. But it’s not always clear whether it’s good or bad software. It is design.

00:40:50.340 Earlier, I stated that one of the important aspects of design is to embrace the constraints given to you. Good designers figure out what the boxes they’re working in are and they learn to make the best things that they can to fit into that box.

00:41:28.619 While some are tempted to consider us engineers, I think we do similar work, but engineering is just a different application of design. There are parts of engineering that we can incorporate into software development, but we are not just engineers.

00:41:58.800 Earlier I used the word craftsman, and that applies. We’re doing similar things to what a craftsman does—they design, but their constraints are different from ours. For some clients, the art is the most important aspect.

00:42:39.450 Sometimes we have to move forward with weak evidence and weak conclusions because we don’t have time to prove them. I think that's one of the most important things we do. Sometimes we have to take risks that we wouldn’t normally take, but because of the constraints imposed upon us, we have no other option.

00:43:06.570 If we waited to write good software, we probably would have missed many opportunities. For example, consider if Twitter had waited until they could scale before launching. Where would they be now? I think we could come up with a lot of examples like that.

00:43:30.780 So be pragmatic! That’s what I’m trying to say. Alright, one more certificate left: who here is working on an app that doesn’t have any users, meaning you haven’t launched yet?

00:43:48.900 How long have you been working on it? Anyone been working on it for six months? A year? No one? Well, you came in with this certificate for polishing turds!

00:44:24.670 I’ll leave you with this: you can discipline yourself to write software that is good enough—good enough for your users, for future maintainers, and for your own peace of mind. You'll find that you’re more productive, and your users will be happier.

00:44:54.350 It’s something I struggle with, and I hope we can all struggle with it together and ultimately accept it. Thank you.

00:45:10.410 Yes, what’s your position on 'worse is better'?

00:45:51.229 To clarify, worse is better refers to how you prioritize pragmatism over purity of design—making practical choices now to get ahead. I generally agree with that. As the developer and craftsman side of me hates it; I hate writing code that way. However, stepping back, I think that’s the best way to go. Alright, any other questions or comments? Thank you very much!

LoneStarRuby Conf 2011