Continuous Deployment

Talks

Matt White

1 talk

#continuous-deployment

#continuous-integration

Continuous Deployment

by Matt White

In the presentation "Continuous Deployment" by Matt White at the MountainWest RubyConf 2012, the speaker discusses the concept and practice of continuous deployment within software development, particularly within their company, Gold Star. The session outlines how deploying code multiple times a day can significantly enhance efficiency and reduce pain associated with traditional release cycles.

Key points discussed include:
- Definition and Importance: Continuous deployment involves integrating code changes continuously, automatically running tests, and deploying code without manual intervention. This practice allows for faster and more reliable releases.
- Survey on Deployment Frequency: A brief survey revealed varying deployment frequencies in the audience, emphasizing that those deploying less frequently are often dealing with preventable headaches associated with lengthy release processes.
- Simplification of Release Processes: White highlights the need for a simplified release process, suggesting that organizations often face bottlenecks due to unnecessary procedures and stakeholders. Simplifying these processes can therefore lead to more efficient deployments.
- Testing and Continuous Integration: The importance of thorough testing was emphasized, noting that tests should reflect user interactions with the software. Continuous Integration (CI) is also critical; fast CI allows developers to receive quick feedback on whether their changes will succeed.
- Upgrading Infrastructure: White shared their experience upgrading from a slower CI server to a more powerful machine, which reduced test execution time significantly, facilitating quicker feedback loops.
- Effective Monitoring: Proper monitoring tools like New Relic are important for maintaining awareness of the deployment's health in real-time, allowing teams to react promptly to issues.
- Myths and Objections: White addressed common concerns about increased pain due to rapid deployments, explaining that continuous deployment often leads to less stressful and more manageable releases.
- Benefits of Continuous Deployment: Frequent deployments help in spotting problems quickly, encourage smaller incremental changes, and reduce the pressure of large release deadlines. The phrase 'failing fast' was introduced, explaining that rolling back becomes simpler with smaller changes, thus enabling more agile project management.

In conclusion, White argues that implementing continuous deployment practices at Gold Star resulted in more efficient workflows and fewer release-related crises. As development teams integrate automated testing and monitoring, they can achieve a smoother, more responsive deployment cycle, leading to overall improvements in software quality and user satisfaction. The presentation serves as a motivational guide for those hesitant towards adopting continuous deployment practices, emphasizing the tangible benefits and advantages experienced firsthand by the speaker's team.

00:00:15.200 Continuous deployment. Okay, so who am I? I'm Matt White. I've worked with a lot of you before. I currently work at Gold Star, which is based out of Pasadena. However, I'm the one remote developer here in Utah, and I have a co-worker here speaking right after me.

00:00:24.080 What Gold Star does is sell half-price tickets for live entertainment. Most of our app these days is in Rails, but we still have some legacy Perl that we're transitioning away from. Believe it or not, our boss had a hard time finding Perl developers to continue building the Perl app. Go figure! I don't know how that happened. But anyway, he decided to switch to Rails, and it's been a lot better since then.

00:00:43.360 So I want to talk about what continuous deployment is at Gold Star. We do something close to continuous deployment. I wouldn't say we're perfect at it, but we're working on getting better. Continuous deployment is nothing new; it's something that a lot of people have been doing for a while. However, I've noticed that a lot of Rails shops aren't really doing it for some reason. We started doing it probably close to a year ago, and it's been an amazing journey to see how much better things are since we got there.

00:01:06.799 Some of you may have heard of Timothy Fitz; he's one of the big proponents of this. He defines it as follows: The overview definition is that you continuously integrate, commit early and often, commit automatically, and on commit, automatically run all your tests. If the tests pass, deploy the cluster automatically, so you are not actually clicking deploy all the time. I think at Etsy they still click deploy, but where he worked before they didn’t; it just went automatically. If the deploy succeeds, repeat.

00:01:34.000 If this is being done automatically, you could be deploying 30, 40, or 50 times a day depending on how fast your tests run and how much you’re committing. I want to take a quick survey of how often we are deploying. Who here is deploying more than once a day? Oh, good, we have about a quarter of you. Who's deploying about once a day, maybe three or four times a week? Okay, so that’s still okay. Who's doing less than that? Okay, you are the ones that really need to listen to this because you are the ones dealing with the headaches that we no longer deal with at Gold Star.

00:02:11.520 I want to talk about our journey, how we got there, and how it’s actually not that difficult. Most of the things that you need to do, you either are already doing or should be doing. If you work on those things, you'll get there really easily. The things that we needed at Gold Star to get to continuous deployment include a simplified release process. I'm sure some of you working at bigger companies have a release process that's ridiculous, taking weeks, months, or longer, involving code freezes and lots of stakeholders doing QA. That was the first thing we did: simplify the process and make it as quick as possible.

00:02:48.239 We had to ask ourselves, "What is the least amount of things we need to do to get code to the server?" What’s holding us back? Is it people or processes? Find out what those things are, eliminate them, and move on. Next, we needed thorough testing. This doesn’t mean you have to have 100% test coverage; it means you need confidence in your release, whatever that means for you. For most people discussing continuous deployment, including us, it means having tests that do what your users do.

00:03:18.880 If you can work on that, you will get a lot closer to having that confidence. Then there’s continuous integration. Hopefully, everyone here is doing that—automatically checking your build to ensure everything is green. Monitoring is essential; you need to know if things have broken, and if they do break, why. You need to know quickly, too.

00:03:44.080 I want to go into each of these points and what they mean. When you want to simplify your release process, you mostly need to ask who is holding us back. If it’s a person, you need to convince them that your release process is quite painful. If it’s something you’ve just grown used to, you may not even realize how painful it is. A simple release process will eliminate that pain, and then you'll understand what you’ve been going through all that time.

00:04:28.720 You need to convince those people that for the sake of sanity and for being able to pivot quickly on your new features, your releases need to improve. This is how a lot of releases are, and I've noticed this in companies I've worked at. It becomes a big deal, with people staying up at odd hours of the morning trying to get everything ready, and everyone freaking out. Who has ever had a release where, as soon as you deployed, things just went crazy? Yeah, probably everyone.

00:05:08.480 And who has had a release that went like that and was really hard to roll back? The fewer hands go up here, but I'm sure you all know what that’s like more or less. Part of why you want to simplify your deploy process is so you can deploy easily and roll back easily, avoiding those nightmare days where everyone’s freaking out and staying up for days trying to fix issues.

00:05:40.720 If this is how you're working with branches, I know many people do it this way. At GitHub, they've discussed this a lot—their Git flow works just like this. They have one main branch and then branch off when they want to build a new feature. Once they finish the feature, they make a pull request. If you're using GitHub, this is extremely easy to do, and this is what we do at Gold Star.

00:06:06.720 It’s nice. Once pull requests are approved, you merge into the master branch, which goes to CI, and everyone’s happy. This is a way to simplify your process. People don’t necessarily need to know you’re branching differently; they don’t care.

00:06:25.760 Thorough testing is one of the most important parts of continuous deployment. As mentioned before, you don't need to aim for 100% test coverage. What matters is that your tests reflect what your users do. When you deploy, you want to ensure your users can still sign up for your service or make purchases. We use Cucumber for that, and it runs all the JavaScript on the page to ensure users have a similar experience.

00:07:04.240 If Cucumber tests fail, that means we don't deploy. Your tests need to be fast and efficient because if your CI process takes an hour or two, it creates bottlenecks. If you're deploying three or four times a day, you're left waiting for CI almost all day, which is not ideal.

00:07:39.760 Here are some quick ideas: Profiling your tests can show where the bottlenecks are, and it’s relatively straightforward to implement. If you find tests that take too long, try to identify and resolve those issues. Avoid describing integration tests as unit tests; this can lead to confusion.

00:07:56.640 If your tests feature lots of database queries or multiple objects interacting with each other, those are integration tests and could be slowing you down. We used to have a significant amount of redundancy in tests, which slowed our test suite. Reducing duplication allows your tests to run faster.

00:08:57.280 Another way to speed up your test suite is to create fewer objects when using factors or similar tools. In unit tests, you don’t need to save objects to the database unless you're specifically testing Active Record's behavior. Load dependencies only as needed to avoid creating unnecessary overhead.

00:09:34.480 To run tests effectively, consider how you structure your tests; you may be unnecessarily loading Rails or Active Record. Some developers do this because they are accustomed to it, but this can slow testing significantly. We’ve had many classes that function without Active Record, which allows for incredibly fast tests when it's required.

00:09:54.000 Now let's talk about continuous integration. We use Jenkins, which was easy to set up. CI is crucial for continuous deployment; it allows you to know if your build is green or not. Fast CI is essential to prevent slowdowns in your development process; our CI server is among the fastest servers in our organization.

00:10:33.120 Previously, our CI server was a quad-core machine with 16 GB of RAM that took around 30 minutes to run tests. We realized we needed to be faster to accommodate regular deployments. We upgraded to a machine with 16 cores and 32 GB of RAM plus an SSD, reducing our test run time to just seven minutes. This investment was worth every penny, as it facilitated faster development cycles.

00:11:16.880 Make use of your multiple CPU cores. Many CI tools run tests on one core by default. You can significantly increase efficiency by utilizing the full capability of your server. Running tests in parallel can drastically reduce execution time; we can now run tests that previously would have taken an hour in just minutes.

00:11:58.720 Monitoring is another critical aspect of deployment. If your code is live and the first person to inform you something’s broken is a user or someone not on your development or operations team, it indicates that your monitoring is lacking. Set up monitoring tools that alert your team immediately when there are issues. Using tools like New Relic allows us to visualize response times and error rates effectively.

00:12:49.760 New Relic integrates with our workflow, notifying our ops team when errors exceed our defined thresholds. There are other solutions available for monitoring, like Nagios and Ganglia, which can help you keep track of your systems.

00:13:04.480 When discussing continuous deployment, it’s common to encounter objections such as increased pain with more releases. However, if you’re deploying multiple times per day, releases become less stressful. In our case, we don't even notice when a deployment happens because it's just a non-event now.

00:14:06.000 Concerns arise about whether users and stakeholders will be comfortable with frequent deployments. Again, in our experience at Gold Star, no one even seemed to notice when we transitioned to better practices. The frequency of releases has allowed us to reduce downtime significantly.

00:14:44.480 If your tests are inadequate, that’s a valid concern, but the solution is simple: commit to starting testing everything. Ensure that the most critical and frequently changing parts of your application are well tested. Start with those, and you will be on your way to having sufficient tests for continuous deployment.

00:15:21.920 If you're cautious about rolling out features incrementally, there are tools like 'Rollout' that can help you manage how new features are released. This allows you to control which users see new functionality and test the waters before a full deployment.

00:16:08.000 Some of the benefits of continuous deployment include reducing waste caused by big release deadlines, allowing teams to focus on smaller increments of change. Instead of massive release weeks, which can create chaos, you can continually improve your software without the pressure of last-minute scrambles.

00:16:40.000 Failing fast is another key benefit. If something goes wrong, you can roll back quickly and fix it. This encourages the practice of releasing smaller changes, making it easier to identify and address issues that arise. With smaller, simpler changes, you can detect problems openly and iteratively.

00:17:16.960 In conclusion, many people joke that deploying frequently can lead to unexpected benefits like more ponies, as I promised in my abstract. The more often you deploy, the more you’ll find solutions surfacing naturally as you refine your processes.

00:17:37.520 Now, I’ll take any questions. I emphasize that the tests we need should match user behavior, as unit tests may not fully capture user interactions. They might validate isolated functions but fail to simulate how users engage with the application as a whole.

00:18:49.280 We often use Optimizely for A/B testing, making it easier to understand how changes affect user behavior. It's essential to have robust monitoring alongside your deployment practices to detect issues early and react promptly.

00:19:25.440 Yes, we've done certain rollouts, often utilizing Optimizely for controlled testing among specific user groups. We gradually increase visibility for new features so we can track and evaluate their performance carefully.

00:20:10.560 I'd like to clarify that while we prefer incremental modifications, significant changes can still be implemented as needed while ensuring thorough monitoring is in place. This minimizes potential disruptions during rollouts.

00:20:52.000 Thank you for your time! If there are any additional questions, feel free to ask.

MountainWest RubyConf 2012