DevOps for the Rubyist Soul

Big Ruby 2013

John Downey

#configuration-management

DevOps for the Rubyist Soul

by John Downey

In the video titled 'DevOps for the Rubyist Soul' presented by John Downey at the Big Ruby 2013 event, the focus is on how Ruby developers can effectively transition to using Ruby for DevOps tasks. Downey introduces his background and the significance of DevOps at Braintree, painting a picture of the company's infrastructure management and the imperative of maintaining high availability for their services. The main theme revolves around leveraging automation, repeatability, and verifiability in server operations, paralleling agile development principles.

Key points discussed include:
- The Importance of DevOps: Downey emphasizes that effective DevOps is essential for Braintree, especially given their responsibility for managing infrastructure that affects both their revenue and customer satisfaction.
- Key Tools Utilized: The talk showcases tools like Capistrano for server orchestration and Puppet for configuration management, detailing how these tools facilitate automation and streamline processes.
- Integrating Git with DevOps: Downey highlights their heavy reliance on Git for version control and collaboration, along with the advantages of a symbiotic relationship between development and operations teams.
- Transition to DevOps Roles: Reflecting on his personal journey from a .NET developer to a DevOps engineer, he narrates how team dynamics at Braintree shaped their approach to DevOps.
- Puppet and Hiera: He covers the use of Puppet, explaining its master-agent architecture and the advantages of organizing configuration management with Hiera for hierarchical data management.
- Virtualization with Vagrant: Downey introduces Vagrant as a critical tool for creating consistent development and testing environments, which enhances productivity and simplifies onboarding processes for new developers.
- Lessons Learned: Throughout the session, he shares lessons from their DevOps experience, such as the importance of consistent coding practices, modular configurations, and frequent mergers to ensure smooth deployment operations.

The session concludes with light-hearted references to DevOps culture, suggesting that humor can also play a role in team dynamics. Overall, attendees gain insights into implementing DevOps effectively in a Ruby-centric environment, learning not just from tools and frameworks but also from the collective experiences of their teams.

00:00:19.410 Hello, everyone! I would like to talk about what I have titled 'DevOps for the Rubyist Soul.' It's about how we, as Ruby developers, transition into using Ruby for DevOps tasks. This is my first time in Texas, specifically in Dallas. I was in New York last week for another conference, and I live in Chicago. I must say, everyone here is much nicer than what I am used to!

00:00:48.399 So far, everything has been going well. I want to discuss Ruby beyond web applications, as most of the Ruby I work with doesn't generate any kind of HTML. How many of you have heard of RubyParley? A few of you? That's great! So, Josh Nassar always suggests starting with some audience participation.

00:01:08.020 Here’s my attempt at audience participation: If you recognize this, understand what it is, and have seen it before, please raise your hand. Now, keep your hand up if you verified the fingerprint before typing in your response. Great! You can now figure out if that was a joke or if I was just trying to identify who to target later.

00:01:31.270 As I was introduced, I’m John Downey, and I am an Infrastructure Engineer at Braintree. For those who don't know, Braintree is a payment service that helps businesses accept credit card payments online. If you're in the audience and use Braintree, I would love to talk to you afterwards and get your feedback, as we always appreciate hearing from our customers.

00:01:54.069 Now, let me give you some background about Braintree, so I can discuss why DevOps is important to us and how it came to be. Every startup has a business plan, and typically, they're more well-prepared than we are. However, we excel in what I call phase three, which is how we actually turn a profit. We primarily achieve this through banking integrations.

00:02:27.489 Fortunately, you never have to deal directly with a bank. It’s not merely talking to a computer system from the 70s, but rather challenging when our customers interact with those legacy systems.

00:02:33.790 Another major aspect of what we do is secure credit card storage. This significantly reduces the compliance burden on our merchants, but it does place more of the burden on us. Therefore, security is a crucial part of what we do. Additionally, we take pride in providing robust developer support with comprehensive documentation and personalized phone support.

00:02:52.569 This brings us to the main point of why I am sharing all of this: at Braintree, we essentially manage other companies' infrastructure. We have to ensure high availability because if we experience any downtime, not only do we lose money, but our customers do as well. Typically, customers get very upset when that happens.

00:03:30.430 So, we strive for zero downtime, which includes both planned and unplanned maintenance. Although this is a different scale of operation than companies like Twitter, we cannot afford to use a 'fail whale' approach.

00:03:45.669 For those who are unfamiliar with DevOps, its goal is to apply the rigor of agile development to the complexities of managing servers and server operations. Our approach focuses on three principles: automation, repeatability, and verifiability. This is analogous to how agile developers think about testing.

00:04:05.560 Dev and Ops are strongly intertwined; it's really a symbiotic relationship. There's so much overlap in roles that we find it beneficial to have combined roles, which help avoid finger-pointing. I want to share a couple of stories, each illustrating different aspects of how Braintree arrived at adopting DevOps.

00:04:49.440 At Braintree, we wholeheartedly embrace Git — we use it for almost everything. Alongside our love for Ruby, many of our team members come from diverse programming language backgrounds. Before Braintree, I worked as a .NET developer; now I describe our developer team's skill sets as a Venn diagram intersecting at our shared love of Ruby.

00:05:01.470 This intersection has greatly influenced the decisions we've made throughout our journey. Yes, we use an agile process; in fact, we used to have a physical card wall to track tasks. However, as we've expanded to multiple locations, that system became impractical.

00:05:43.740 We prioritize pairing on work whenever possible, especially for production tasks. Having a second set of eyes and hands can make a significant difference, especially when things go awry.

00:06:05.670 Early on, we noted our tasks on physical cards using stickers: red for pick-up work, green for new projects, yellow for maintenance, and blue for infrastructure. As the team grew, it became clear that sub-teams began to form around these categories. Remarkably, around the time I joined, a small group, which we referred to as the 'blue team,' formed organically.

00:06:37.530 Initially, we had no intention of becoming DevOps engineers; it simply developed from our collective experiences. We navigated this territory together, learning and figuring things out as a team.

00:06:58.060 I have a huge passion for Linux — not just liking it, but truly loving it. If Linux were in my high school, I would have nervously passed it a note asking it to prom! When I joined Braintree, there was some discussion about which team I should join, and I ended up on the blue team, likely due to my fondness for Linux. Initially, I didn't fully grasp what that meant, as I had only managed servers manually without any automation.

00:07:48.310 Suddenly, I found myself in the role of a DevOps engineer. Fortunately, I had an amazing team to collaborate with that guided me through this transition. I apologize that some of these scenes are out of order; it happens!

00:08:12.730 As a DevOps engineer, I had to learn a new set of tools and a new vocabulary, such as Capistrano and Puppet. Puppet, in particular, is crucial for managing our servers; it allows us to write scripts that control them efficiently. While Puppet is implemented in Ruby, it has its own unique configuration language, which can be somewhat confusing.

00:09:12.370 Typically, Puppet operates using a master-agent system where a central server issues commands to nodes through agents. While this is standard practice, it's not the approach we've taken at Braintree.

00:09:57.560 Let’s quickly break down some Puppet concepts: Puppet files are called 'manifests' and have a .pp file extension. The main file that Puppet uses is site.pp, which essentially bootstraps the system. Within Puppet, you specify how to configure nodes, such as installing Ruby via the operating system’s package manager.

00:10:38.900 Through Puppet, if Ruby isn’t installed, it will automatically install it; if it is, it won’t change anything. You can declare variables in your configuration — a useful feature I want to illustrate with an ERB template. This template allows us to define important attributes like ownership, mode, and more. It’s similar to how Ruby developers typically work in Rails.

00:11:50.930 In Puppet, we can also organize our code into classes, which allows us to build more comprehensive and reusable configurations. By utilizing classes, we can create a generic manifest that can be shared across our nodes.

00:12:25.960 Puppet also provides a tool called Facter, a Ruby gem that gathers system facts and exposes them to Puppet for use in configurations. For example, if you're managing different operating systems, you can use these facts to conditionally apply settings.

00:13:05.420 When it comes to organizing Puppet code, it’s typically organized in modules. Modules are akin to Ruby gems, providing structure for our Puppet manifests and templates. This organization allows our code to be modular and manageable, which is critical as the size of our infrastructure grows.

00:13:50.430 There's a collaborative aspect to this as well. The Puppet community has created a resource called Puppet Forge, which allows developers to share their modules. Instead of each developer creating their own modules for common services, they can leverage existing work, saving time and effort.

00:14:37.540 Another critical tool we utilize is Hiera, which enables hierarchical configuration management. Hiera's YAML format allows for organizing configuration data in a structured way, useful for managing complex environments.

00:15:23.540 Here's a sample from our infrastructure repository, illustrating our Hiera setup. In our Chicago data center, every server with a name starting with 'DB' picks up a specific file representing its configuration. These files can overwrite each other based on server names, streamlining our configuration process.

00:16:07.440 For virtualization, we use Xen, a virtualization platform that is battle-hardened for our virtual machines. This snippet showcases a variable setup in Puppet, allowing for management of virtualized nodes from a central configuration.

00:16:40.220 Hiera also lets us interpolate Puppet variables, which helps avoid redundancy and enables dynamic configuration setups. With Hiera, we develop a Puppet model of our desired system state, allowing easy extensions and updates.

00:17:27.559 We’ve learned many lessons through our use of Puppet, primarily focusing on best practices. One of those lessons involves adopting a style guide for our Puppet code, which aids in maintainability. Following consistent practices encourages clearer communication and reduces complexity.

00:18:15.740 We also emphasize the importance of utilizing Puppet modules for composability. Modules allow cleaning up and structuring our configurations, allowing better organization of resources.

00:19:02.340 Puppet has a file backup feature that retains every version of files over time; however, this may not be ideal for large files. For larger binaries, it is recommended to turn off the file backup feature to avoid excessive versions.

00:19:55.480 As a final thought, we prioritize merging frequently to maintain momentum within the team. The more often we merge, the smoother the deployment process becomes. We aim to conduct our infrastructure updates weekly.

00:20:37.480 Now, let’s talk about Vagrant, another essential tool. If you haven't heard of Vagrant, brace yourself for a mind-blowing experience! Vagrant allows you to automatically create and configure VirtualBox machines. Using commands like 'vagrant init' and 'vagrant up,' you can spin up an environment.

00:21:22.130 With Vagrant, you can provision machines using Puppet, allowing seamless integration between development and production environments. This integration drastically elevates testing and development productivity.

00:22:08.370 Additionally, Vagrant excels in facilitating rapid prototyping, helping us test clustered and distributed technologies locally before deploying to cloud environments. For instance, we tested RabbitMQ locally before deciding to move forward with it.

00:23:00.110 Moreover, Vagrant can streamline development processes. Rather than running builds on local machines, which might be OSX, we provision development boxes that mimic our production environments. By using Vagrant for builds, we ensure consistency and reduce conflicts.

00:23:51.045 This method accelerates the onboarding of new developers, allowing them to kickstart their work promptly without the need to navigate extensive setup processes. Meanwhile, they can immediately begin contributing to our projects.

00:24:30.960 Before wrapping up, I’d like to share a little DevOps humor. If you’re not familiar, 'DevOps Reactions' is a Tumblr filled with amusing animated GIFs relevant to our field. One example humorously depicts a term known as 'Donath,' which stands for 'shoot the other node in the head.' This is common in clustered services to manage state.

00:25:18.160 Another fun resource is 'DevOps Borat,' which tweets humorous takes on our field. If you’re into DevOps, I encourage you to check it out. Thank you all for your attention!

00:26:02.820 My name is John Downey, and I work at Braintree. You can follow me on Twitter for more insights.

Big Ruby 2013