Real Developers Ship (a.k.a Tenets for Software Delivery)

by Sebastian von Conrad

In his talk "Real Developers Ship" at RubyConf AU 2014, Sebastian von Conrad shares the journey of a development team at Envato as they built and shipped a new product, Microlancer. Focusing on fundamental principles for effective software delivery, Von Conrad discusses actionable tenets that transformed their workflow and led to improved results in shipping products.

The key points of the talk include:

Fake it until you make it: The team quickly launched a low-fidelity version of the product to gather real user data and validate their assumptions.
Cut the crap: They simplified their task management by focusing on essential actions instead of adhering strictly to methodologies like Agile or Scrum.
Elevate your CI to Continuous Deployment: By leveraging Continuous Integration that automatically pushed successful builds to production, the team minimized risks and increased the frequency of updates.
Test in production: Utilizing feature flags permitted the team to test new features in a live setting while maintaining control over when they were activated.
The only way to move is forward: Instead of rolling back code, the team focused on fixing issues directly, ensuring continual progress.
Live on the bleeding edge: Keeping dependencies up-to-date through daily bundle updates helped the product remain optimized and functional.
Pay the refactor cost upfront: They emphasized writing code with the expectation of future changes and cleaning as they went, rather than scheduling dedicated time for refactoring.
Stay lean with YAGNI: The principle of "You Ain't Gonna Need It" encouraged the team to prioritize essential features and avoid premature implementation of unnecessary functions.
Do things that don’t scale: The team began with manual processes to understand workflows fully before automating, ensuring effective solutions.
Time is value: Finally, they highlighted the importance of optimizing time spent on coding, testing, and deployment to continually provide users with added value.

In conclusion, the talk presents a comprehensive approach to software delivery that prioritizes speed, efficiency, and customer value. By adopting these principles, developers can significantly enhance their workflows and product outcomes.

00:00:07.120 Hello, everyone! In case you didn't hear it, I lost all my slides. I was meant to talk a little bit earlier today, but we had to shuffle things around. Everything you see here has been recreated in the last couple of hours, so hopefully it goes alright.

00:00:12.719 My name is Sebastian von Conrad, and you can find me on Twitter under the same name. In case you haven't noticed, I have an accent, which is because I grew up here in Sweden, where it's very cold, and we have the lovely aurora borealis in the wintertime. I also lived here for a while, which is very different from Sweden. Now, I have been living in Australia for five years.

00:00:25.600 To tell you the truth, the worst part about being a Swede living in Australia isn’t all the dangerous animals that want to kill you. No, the absolute worst part is that the first thing that people say when I tell them my nationality is 'ABBA.'

00:00:37.120 To set the record straight, I don’t like ABBA, and if you start talking to me about them, we can no longer be friends. I will admit, however, that I have a deluxe set of Allen keys that I use to assemble IKEA furniture, not just my own but I’ll happily help anyone who’s bought something from IKEA.

00:00:50.320 I also do guided tours at IKEA, where I not only show you the way out but also explain what all the funny names of the furniture actually mean. Believe it or not, they all mean something.

00:01:01.440 I work for Envato over in Melbourne, as Matt mentioned. More specifically, for the last 18 months, I’ve been working on Microlancer, and I wanted to share a little bit about the Microlancer story. About 18 months ago, I came into work one morning, and the CEO called me into his office, which is never a good sign, but in this case, it actually was.

00:01:16.159 He told me about this new product idea he had been considering for a while, and he wanted me to build it with a new team and just go for it. He had a 14-page Google Doc and two PSDs that he’d cobbled together over the weekend before, and that was effectively the product plan.

00:01:35.280 We started with a blank page—well, a terminal window, really—and had a really tight timeline with lots of new and exciting ideas. Everyone was suggesting things we should do, but we needed to get something together fast. We had to ship better and faster than we had before.

00:01:52.080 We took this opportunity to reflect on what had worked in previous jobs and teams and extracted the best parts to create a workflow and delivery process that really worked, allowing us to ship effectively.

00:02:08.479 The title of this talk is based on a quote by Steve Jobs, which is 'Real artists ship.' I believe this is equally true for developers. As Cameron mentioned in his keynote yesterday, you’re not really a developer unless people are actually using your code.

00:02:14.479 We did ship, but after we shipped, we had to continue to improve. Suddenly, we had to worry about bug reports from users, maintain a production environment, provide operational support, and manage many other responsibilities, all while delivering new features.

00:02:30.160 This talk is really about the things we did that allowed us to do great work, shipping better and faster. I call it the tenets of software delivery, which might sound a bit fancy.

00:02:42.080 The word 'tenet' refers to principles or beliefs held as true by members of a profession or movement. The key point here is that these are principles we decided worked for our team and allowed us to get things out there.

00:02:56.480 Let’s dive into it with the first tenet: 'Fake it until you make it.' I’ll leave this slide here for a bit.

00:03:01.440 We shipped the first version of the product in two weeks. Initially, it was called Graphic Lancer, not Microlancer, and it was a seriously low-fidelity version cobbled together on Heroku.

00:03:14.720 We used Zendesk and email to manage interactions, essentially duct-taping everything together like this slide.

00:03:26.320 There were many features we knew we would need later, such as a messaging system for in-product communication, but we started with email to validate our assumptions.

00:03:35.360 We wanted to test if people would be willing to pay upfront, what price points would work, and what categories people were interested in.

00:03:43.680 Making it live and allowing real users to buy real things with real money was an incredible way to validate our direction and build a technological foundation.

00:03:55.520 The blank terminal window can be intimidating, but Graphic Lancer ultimately evolved into Microlancer. We changed almost every line of code, renamed the app, and adjusted the process as needed.

00:04:11.760 The next tenet I call 'Cut the crap.' This refers to simplifying your processes in the background. At one point, we had a wall with cards representing tasks.

00:04:27.120 We forgot about Agile, Scrum, XP, and all the different methodologies that people often insist you should use. Instead of classifying tasks as defects, chores, or bugs, we focused on just getting things done.

00:04:39.520 We created three columns: to do, doing, and done. The only thing on the cards was the task at hand.

00:04:51.360 I'm not sure how well you can see it, but if you're remote, you can use Trello to achieve the same results. However, there's no substitute for having a real card wall somewhere.

00:05:06.720 We realized that we needed to take an agile approach to our agile process. Just like your software, your process will and must change over time.

00:05:21.760 The more you add on top of it, like estimates and various methodologies, the harder it gets to iterate on your process.

00:05:38.080 So, introduce complexity only when it makes sense. We had estimates for a while, but we don’t anymore.

00:05:49.760 We began classifying cards and writing them differently, changing our approach continuously. The main takeaway is that your process needs to be what you want it to be.

00:06:06.080 Don't feel bound to Scrum, Agile, or XP practices as they might not fit your team. Instead, take whatever works for you from all these methodologies.

00:06:21.840 The next tenet is to elevate your Continuous Integration to Continuous Deployment. If you don't do CI, I highly recommend that you get onto it quickly.

00:06:34.560 For us, we had a Continuous Integration environment that, for every successful push to master, would deploy that build to our staging server, run some smoke tests, and then go straight to production automatically.

00:06:48.960 The merge pull request button turned into our deploy button. This is possible when you trust your test suite, and it really changed our mindset.

00:07:05.120 We started committing smaller bits of code and pushing out updates at a much higher frequency. This meant lower risks when shipping things that could potentially break.

00:07:17.920 Moreover, we saw that the overall quality of our features improved with fewer bugs and a more polished experience.

00:07:31.440 The focus was on delivering value as early as possible. Instead of waiting weeks before releasing features, we shipped pieces of value incrementally.

00:07:44.879 Although the value of each small release might not be as significant as a larger feature set, it allowed us to provide value to users much sooner.

00:07:57.760 It also meant that, if we had several tasks to complete, we could still ship the first few successfully and leave the less critical ones for later.

00:08:10.720 We didn’t have any holdups either. I’ve been in many teams where the master or production branch was not deployable due to bugs or additional testing needs.

00:08:22.560 With Continuous Deployment, we could bypass those issues. Continuous deployment is a topic worth exploring more.

00:08:37.920 Fortunately, two of my colleagues, Keith and Mario, will be discussing Continuous Deployment following my talk.

00:08:50.960 The next tenet is 'Test in production.' I had a slide about this, but it didn’t make it into the second version.

00:09:01.440 One of our first challenges with continuous deployment was that the business side said we needed to control when things went live.

00:09:15.760 They were concerned about marketing strategies that required precision timing, or they were awaiting approvals.

00:09:27.920 We solved this by introducing feature flags, which have been crucial to our strategy. They allow us to push changes whenever we want but activate them later, at a controlled moment.

00:09:41.760 This method enables us to test new features in production with real user data before broader rollout.

00:09:56.960 There are various feature flag gems that let you target specific users, such as admins or staff, to test features before they go live.

00:10:09.920 It’s powerful as it allows us to ensure everything works optimally before the public release.

00:10:24.080 We even feature-flag our infrastructure services; for instance, we handle payments through credit cards and PayPal.

00:10:38.720 In case our credit card gateway (Braintree) is down, we can toggle that off with a click rather than going through a lengthy deployment process.

00:10:51.760 This ability helps us provide consistent user experiences despite backend issues.

00:11:06.080 We’ve gone so far that we even schedule feature flags to turn on and off automatically, helping in case of maintenance windows without requiring manual intervention.

00:11:20.960 We realized the importance of using production data in development. We initially tried using seed data, but maintaining it became a tedious task.

00:11:34.080 Instead, we set up a script to pull down our latest production database backup, scrubbing sensitive information while keeping the rest intact.

00:11:50.640 Though this may not work for very large production databases, using production data enables us to spot issues earlier, especially with user-generated content.

00:12:04.080 The next tenet is 'The only way to move is forward.' We never roll anything back; we only roll forward.

00:12:16.680 We’ve never used git revert on a commit that went bad. Instead, when issues arise, we focus on identifying and fixing the problems.

00:12:30.080 With feature flags and small, incremental changes, it should be quick to find what went wrong and address it rather than reverting.

00:12:42.080 This method allows us to recover quickly and maintain our momentum in shipping.

00:12:57.200 The next tenet is to 'Live on the bleeding edge.' This may not be visible, but if you look closely, you’ll see that our Gemfile is lacking specific versions.

00:13:10.080 We aim to keep our dependencies up-to-date, which we accomplish through daily bundle updates, sometimes several times a day.

00:13:24.560 This approach works well when you have confidence in your test suite. Don’t let your application become stale, and take advantage of new features and optimizations.

00:13:41.600 Of course, this can backfire. I’ve encountered issues, such as when a minor update broke visual layouts due to relying on an underlying bug fixed in that version.

00:13:55.120 Even if our test suite passes, sometimes we face unexpected changes. Going forward, I’d like to see automated visual testing to catch such issues before they reach production.

00:14:10.080 Despite challenges, we continue our routine bundle updates because they provide significant value.

00:14:21.920 The next tenet, which we've emphasized along the journey, is to 'Pay the refactor cost up front.' You will be wrong.

00:14:37.760 Whether your code is inaccurate or the product decisions you've made are off, account for that reality. Write your code with the expectation of refactoring.

00:14:51.040 Refactor promptly if you find you've rushed things. For instance, we deployed our product without a way to disable problematic users.

00:15:05.120 When we faced issues with misbehaving users, a developer quickly hacked a solution to disable them. However, we then focused on properly refactoring this code.

00:15:21.840 I view technical debt as akin to financial debt; you should eliminate it as quickly as you can.

00:15:38.640 One crucial lesson is not to schedule time specifically for refactoring. Don't pause delivering value to users.

00:15:50.640 Instead, adopt a philosophy of 'Clean as you go,' leaving code better than you found it. Take the chance to improve code while you're in that area.

00:16:01.440 We implemented the single responsibility principle closely, using service objects and form objects for clarity.

00:16:13.760 We aim for skinny controllers and models, allowing for clear responsibilities and easy understanding of dependencies and interactions.

00:16:26.480 The next tenet is 'Stay lean with YAGNI.' Raise your hand if you've heard of YAGNI before? Great! For those who haven’t, it means 'You Ain't Gonna Need It.'

00:16:40.960 Many product people love features, often suggesting we should build this or that, but the truth is, we often don’t need those additional features.

00:16:55.040 I might not always agree with the 37 Signals philosophy, but they hit the nail on the head when they say to build half a product, rather than a half-assed full product.

00:17:09.760 Limit the feature scope to what's necessary. You need to focus on the great ideas rather than all good ideas.

00:17:23.360 We launched Microlancer without a way to disable user accounts, which took us three weeks to implement when we finally realized it was necessary.

00:17:39.040 Another example is our payment reversal feature; we didn’t implement that until eight months later because we didn’t need it sooner.

00:17:55.760 The next tenet is 'Do things that don’t scale.' There's an xkcd comic about decision-making based on the time saved by automating.

00:18:09.600 Regarding fraud handling, we started with a very manual process involving Rails console commands.

00:18:23.680 Over time, it evolved into a rake task, but it took us a long time to build a button and integrate it into the interface.

00:18:35.520 Only after we fully understood the process did we automate it. Don’t automate until you’re certain that it makes sense.

00:18:49.520 The final tenet is that 'Time is value.' I often hear people express a desire for more time in their day.

00:19:05.680 Everyone has the same amount of time, so the key is to spend it wisely. Ensure your tests run quickly, deploy faster, and establish shorter feedback loops.

00:19:18.800 Focus on optimizing the parts you can control, so you can devote more time to aspects that you cannot.

00:19:32.720 The emphasis should always be on delivering value and continually shipping with improved speed and efficiency.

00:19:45.680 That’s everything I had for today. I’m not sure if there's time for questions, but thank you for listening!

00:19:56.560 A question from the audience: Thank you, Sebastian, that was excellent.

00:20:03.760 I was interested in your comments about forgetting formalities during early-stage development. At what stage should we re-adopt those formalities?

00:20:13.120 I believe that the process for its own sake won't aid speed. You should introduce more formalities when you feel like you’re slowing down without them.

00:20:26.080 When communication becomes an issue or confusion arises to the point where others don’t understand your work, that’s when it’s time to reconsider.

00:20:37.920 Optimize for speed and shipping, and bring in formalities whenever they make sense.

00:20:45.280 Another question: Can you elaborate on your view that you ain't gonna need it?

00:20:52.960 A tricky question, indeed! People often have a complete view of a feature in their minds, imagining how user management systems should operate with roles and alterations.

00:21:04.000 Typically, they think we need to build everything at once. I push back, asking what the minimum requirement is.

00:21:18.720 If all you want is for users to sign up, let's start with that and not worry about other features right now.

00:21:27.920 There may be a later need for additional features, but often, they aren't necessary.

00:21:36.560 Here's another question: Could you explain your approach to not rolling back changes?

00:21:47.600 By not rolling back, I mean we don’t revert code to a previous state. Instead of utilizing git revert, we fix the existing problems.

00:22:00.640 If your production environment has issues, we’ll instead address the problem directly. We’ve never rolled back prior code revisions.

00:22:13.760 If significant problems arise, we may temporarily use maintenance pages while we correct the issues quickly.

00:22:29.440 I've had to use maintenance pages often, generally not exceeding 15 minutes at a time.

00:22:44.560 You discussed the importance of a test suite as a guard when deploying continuously, right?

00:22:53.920 Yes, keeping good coverage is essential. We've focused a lot on optimizing our test suite for speed.

00:23:06.800 Our entire test suite runs in about a minute and a half, even with a year and a half of code and thousands of tests.

00:23:18.240 Unit tests are typically quick, while integration tests can take more time. We prioritize 'happy path' tests to maintain speedy feedback.

00:23:31.760 This approach helps to ensure we're effectively covering core functionalities without letting testing slow down our process.

RubyConf AU 2014