The Setup: Managing an Army of Laptops with Puppet

Talks

Will Farrington

2 talks

#configuration-management

#puppet

#development-environment

The Setup: Managing an Army of Laptops with Puppet

by Will Farrington

In this presentation at RubyConf AU 2013, Will Farrington discusses a project called "Boxing", developed at GitHub to simplify the onboarding process for developers. The session emphasizes the frustrations many developers face when setting up their project environments and introduces a solution designed to streamline this process with automation. \n\n### Key Points: \n- Origins of Boxing: The project was born out of frustration from developers, John Barnett and Will Farrington, who found the setup of development environments time-consuming and cumbersome. \n- Objective: The primary goal is to allow developers to ship code on their first day by making setup easy and intuitive. Users can quickly run a command to get started without extensive preparation. \n- Constant Updates and Security: Boxing facilitates continuous updates to dependencies, reducing issues of environment drift and enhancing security by allowing immediate response to vulnerabilities. \n- Community Involvement: Boxing has been open-sourced and benefits from community contributions, with statistics showing significant engagement. \n- Puppet as a Core Tool: Boxing operates on Puppet, a configuration management tool, which ensures consistent environments, idempotency, and easy version control across different systems. \n- Ease of Use: Developers can automate environment setup through a simple command-line interface, significantly reducing manual errors and setup time. \n- Error Reporting and Management: Boxing simplifies error tracking by automatically generating reports when failures occur, allowing developers to focus on coding rather than troubleshooting. \n- Personal Manifest Customization: Users can customize their development environments by creating personal manifests to manage additional required tools and applications. \n\n### Conclusions and Takeaways: \n- Boxing aims to make developer onboarding seamless and efficient, leveraging automation to minimize friction in setup processes. \n- The initiative emphasizes the importance of community collaboration and maintaining consistency in development environments to enhance productivity. \n- Overall, Boxing represents a significant advancement in making the development experience smoother for new GitHub team members and the larger open-source community.

00:00:06.240 All right, how's everybody doing? Yeah, come on! Get excited!

00:00:12.320 Today, I want to talk to you guys about boxing. I'm @dubar on Twitter and GitHub. I work on the operations team, as you can tell from my lovely neck beard.

00:00:17.840 I also work on something we’ve started developing at GitHub called the Friction Team. It's kind of an unofficial team right now, but it consists of a few people who dedicate a lot of their time to making it easier to develop and design things at GitHub.

00:00:30.720 You might ask, 'What is boxing?' Well, it all started with John Barnett and me. John Barnett is a fantastic developer who has released all sorts of cool projects.

00:00:42.239 One day, we got really frustrated with how working on projects—whether at GitHub or any other company—was just awful. It took days to set up your environment. If you had to wipe your machine and start again, it was a horrible experience.

00:01:01.879 So, we developed boxing, which is a framework and a standard library. It embodies a philosophy for how you should do things. Boxing tries to install dependencies for you, manage your system configuration, and generally stay out of your way while making things easier to get done.

00:01:14.680 Why should you use boxing? At GitHub, we want you to be able to ship code on your first day. It might sound adventurous, but that’s what we aim for. Let’s say you show up at a new company and want to ship something. You think, 'Great, I’ll get a new laptop, spend three days setting it up, and fight with RVM, Nginx, Apache or whatever.' By the end of the week, maybe you’ll have written a line of code. That’s lame!

00:01:40.880 We said, 'Let’s lower the bar! Let’s make this as easy as possible.' You’re going to open a website, it’s going to give you a command, you run that command, and you’re ready to go.

00:02:00.960 Another benefit of boxing is constant updates. If you've ever used systems like Cinderella or Pivotal Workstation, you know they emphasize having a blank machine and then getting a usable development environment. If it breaks, you better wipe your laptop and do it all over again. We think that’s lame too.

00:02:13.440 Boxing is designed to be run daily, weekly—whatever you need. We encourage GitHubbers to run it all the time because we are constantly pushing updates for various dependencies.

00:02:30.360 For example, GitHub recently started testing Ruby 1.9.3 in development, while GitHub itself still runs on 1.8.7. It’s a large site, so migrating takes time. To roll out the new version, I simply pushed an update to boxing that indicated GitHub should now use our custom fork of 1.9.3. I informed the team through a pull request explaining the changes.

00:02:55.240 Microscopically, all you have to do is run boxing, followed by running the server for GitHub, and everything will work seamlessly. We aim to take care of everything for our developers.

00:03:15.400 If you make it hard for developers to update, they won’t do it. They prefer things to be easy, intuitive, and understandable. When updates don’t happen, it leads to issues of drift between development and production environments.

00:03:39.799 For instance, if some developers are using one patch release of 1.8.7, while others are on Ruby Enterprise Edition or even 1.9.3 or a 2.0 RC, it creates a mess! Lack of consistency can lead to weird, latent problems, especially with different Ruby versions affecting extensions.

00:03:55.920 One of the other benefits of constant updates is enhanced security. Consider a situation where Java has a serious issue with Safari as a web plugin.

00:04:02.159 We send an email notifying users to disable it due to security concerns, but only about 10% will read it and respond. Many will archive the email without a second thought. So, what can we do? We can roll out security updates through simple code changes.

00:04:26.000 For example, if Atmos opens a pull request to disable a security risk, we notify users and provide full reasoning. If necessary, users can enable it on a case-by-case basis, enhancing repeatability in the process.

00:04:51.240 How many times have you encountered onboarding struggles for new employees? Maybe you're trying to install RVM for them, and it's not working as expected. You end up troubleshooting outdated files and versions.

00:05:09.080 That’s another instance of manual processes being error-prone. Mistakes happen because we're humans. Yet, with computers, you can automate and reduce errors by providing clear directions.

00:05:26.800 One other aspect that's really cool about boxing is the community support. We opened sourced boxing recently, and I’m excited to share some stats. As of yesterday, we have about 134 pull requests and 230 change issues.

00:05:43.760 That shows how much the community is interested in boxing. There’s been significant interest and contribution from many individuals, improving usability for different environments. That’s why we open source—it’s about sharing the lessons we’ve learned.

00:06:04.380 GitHub is invested in this project as well. Both John and I are going to stay as full-time maintainers of boxing until we bring on more people who want to contribute.

00:06:19.840 Another important note is that we use the same modules you do. We don’t fork puppet modules or use unique Ruby modules. It's a commitment to using open source principles without exclusivity.

00:06:40.280 An example of this is how we handle hubot scripts where we use the same hubot scripts available to the community. The boxing template repository is synced periodically from GitHub’s own boxing repository.

00:07:02.640 We maintain a stringent syncing schedule to ensure a consistently effective environment for users. It provides a wide array of standard libraries as well.

00:07:21.040 Puppet has a feature called Puppet Forge. If you’ve used it, you’ll know it has several implementations of the same functionality with varying standards.

00:07:37.680 It’s a chaotic space with many engine X or Apache modules that don’t behave consistently. However, boxing emphasizes creating plug-in modular components designed for reuse.

00:07:56.240 That means you can mix and match exploring a variety of modules, ensuring they work seamlessly. While this isn't solely a puppet issue, other configuration management systems like Chef share similar problems.

00:08:20.480 Given the advancement in Ruby gems over the years, configuration management communities can learn from these developments regarding dependency management.

00:08:37.160 So how does boxing work? Under the hood, there’s a lightweight Ruby wrapper, but the core component is Puppet. Puppet is a widely adopted configuration management tool.

00:08:56.960 Using Puppet, you define what the desired state of your system should look like. When you run Puppet, it tries to apply those settings consistently.

00:09:13.199 The concept behind Puppet is idempotency. This means no matter how many times you apply something, you achieve the same end state.

00:09:29.840 This is essential for ensuring that developers upgrading from any previous state can smoothly attain the same end environment without needing to wipe their machines.

00:09:46.320 Puppet's structure involves building blocks known as resources, which come in various types like package management, service controls, user management, etc.

00:10:00.680 Each resource type offers different capabilities, like managing package installations across various systems through providers.

00:10:16.080 As a user, you can specify your requirements in a straightforward way. Puppet’s design adheres to a single namespace for variables and gives an opportunity for problematic configurations.

00:10:33.280 Facts offer a glimpse into some useful runtime data, which Puppet fetches automatically. These details enhance our overall understanding and tracking of the system state.

00:10:50.240 Puppet classes are just as simple, allowing for resources to be packaged together. This modular approach leads to cleaner organization and easier adjustments.

00:11:07.560 When working with relationships between resources, Puppet establishes a notification framework that ensures things happen in the right order. The order matters for consistent results.

00:11:23.840 Puppet is not procedural in nature; it is declarative. This means you define what is desired rather than specifying the exact steps to reach that state.

00:11:40.960 Each module in Puppet is a collection of organized classes and other necessary configurations, making management and maintenance of the configuration stack much simpler.

00:11:58.240 To get started with boxing, you can visit our repository, follow the instructions to build a custom setup for your organization or team.

00:12:17.480 Once you have it, you can utilize tools like boxing web to quickly deploy your setup with minimal friction. This process only takes about 30 minutes to complete.

00:12:35.000 Every step is automated through this simple command-line interface, allowing developers to get to work faster.

00:12:58.760 Project manifests streamline this process further, enabling individual developers to customize their environments by adding any necessary extras specific to their workflows.

00:13:15.760 For example, you can create a personal manifest to install programs like Minecraft or zsh simply by following a straightforward syntax.

00:13:34.080 Integration with existing developer workflows is seamless, with many personal manifests already supporting a variety of developer preferences and tools.

00:13:50.960 Boxing also provides a streamlined error-reporting process: anytime a run fails, an issue is automatically created, reporting all necessary context for debug tracking.

00:14:09.560 This ensures developers don’t waste time reporting bugs—they run boxing, and any issues that arise will automatically generate reports for the maintainers.

00:14:28.560 We handle the problem-solving process, applying patches as needed and notifying users when their issues are resolved. It’s an easier development cycle overall.

00:14:46.040 Updates are carefully managed to ensure there are no under-the-hood surprises. If boxing detects a dirty tree, it will not automatically proceed with updates.

00:15:03.440 It clearly communicates the need for reconciliation, making sure developers are in control of their environments.

00:15:20.720 If you’re sold on adopting boxing, we encourage you to start using it to automate tasks in your environment, as it dramatically enhances efficiency.

00:15:37.560 After running training sessions for developers on GitHub, we found that automation truly makes onboarding smoother without the cumbersome setup stage.

00:16:00.280 If you have any questions about boxing, I'm here; feel free to reach out after the talk.

00:16:20.640 Are there any questions?

00:16:37.920 Yes, we have an open pull request that will support GitHub Enterprise, which means we'll have the necessary adjustments for repository management.

00:16:58.480 Currently, boxing is OS X centric because the majority of our developers use Mac machines.

00:17:18.960 Puppet supports multiple operating systems, and we are more than happy to work with contributors to enhance support for other distributions.

00:17:40.560 Though some developers prefer their existing processes, we provide environments that are both accommodating and efficient.

00:18:00.399 In sum, we make it easy for developers to oversee their customizations while also bridging compatibility and usability with boxing.

00:18:20.160 So long as the basic principles are followed, onboarding developers sight is a straightforward experience.

00:18:40.080 What matters most is making the setup process as enjoyable as the end result.

00:19:00.560 Thanks for attending—let’s keep this conversation going!

00:19:20.480 Thank you all very much for your time. If you have further inquiries or are looking for ways to contribute to boxing, feel free to reach out and talk.

00:19:40.080 I look forward to connecting with you!

RubyConf AU 2013