Chef Cookbook Design Patterns

MountainWest RubyConf 2011

by Joshua Timberman

In this talk titled "Chef Cookbook Design Patterns," Joshua Timberman discusses effective practices for designing Chef cookbooks to enhance their usability across different environments and infrastructures. With an extensive background in web operations and automation, he outlines how Chef helps streamline systems administration tasks through its infrastructure as code model. The presentation covers significant design patterns in cookbook development, emphasizing the importance of attributes, dynamic configurations, and community collaboration.

Key Points Discussed:
- Introduction to Chef: Timberman introduces Chef as a tool that enables infrastructure as code, showcasing its benefits for both developers and systems administrators.
- Cookbook Structure: He emphasizes the importance of organizing cookbooks, where each serves specific functionalities, promoting modularity and maintainability. For example, separate cookbooks should exist for each service type.
- Using Attributes and Internal Values: The talk highlights utilizing attributes to define defaults in cookbooks and the importance of Chef's internal values for dynamic configurations, allowing flexibility across different environments.
- Modularity and Maintenance: Timberman encourages avoiding hardcoded values and advocates for clear documentation, suggesting that cookbooks should focus on best practices and modular designs.
- Dynamic Configurations: He explains how Chef's rich search API enables powerful discovery of node attributes, helping create dynamic configurations that adapt based on the current infrastructure.
- Data Bags and Templates: The use of data bags for sharing arbitrary data and ERB templates for generating dynamic configurations based on node attributes is discussed as essential components of good design in cookbooks.
- Testing and Documentation: Emphasizing the significance of testing cookbooks like other Ruby projects, Timberman also encourages thorough documentation to facilitate community sharing and collaboration.
- Conclusion and Community Involvement: He concludes by inviting further discussions at the Engine Yard Hack Fest, promoting an open community for sharing insights and resources related to Chef cookbooks.

00:00:12.320 All right, hopefully this is for everybody. So, welcome to Chef Cookbook Design Patterns! I'm Joshua Timberman. Thanks for picking my talk during this time slot. I know we had a lot of choices today. I went ahead with a one-hour talk recommendation that Mike sent out, but I still have nine slides to cover, so let's get started.

00:00:18.640 So, who am I? I write cookbooks. Enough said. I also work in web operations. I've worked for IBM and have experience in e-business, handling enterprise systems with Java and XML. I managed systems at IDE and later at Stands, where I automated infrastructure using Puppet, Capistrano, and various internal tools.

00:00:24.400 This journey brought me to HJ Solutions, which provided systems administration consulting for startups, focusing heavily on Puppet, Capistrano, and Ruby on Rails startups. HJ Solutions eventually became Opscode when we published Chef. Currently, I write many cookbooks, lead the training program, conduct consulting services, and engage in evangelism by speaking at conferences.

00:00:31.520 Now, who are you? Who here is a developer? Alright, quite a few! Who’s a systems administrator? And who is a developer that also does systems administration? Who wants to eliminate their systems administration tasks? Well, Chef will help you with that!

00:00:43.920 Now, let’s take a look at business people. Hint: if you’re an independent consultant, you are indeed a business person. I’m excited to share that we now have a logo for Chef, which is pretty cool. This means we can stop using icons like the chef from the Muppets or South Park.

00:00:52.719 Let’s have another show of hands – who is using Chef? Solo? A few people. Who is using the Chef server? Awesome! We love you all, thank you. And we really appreciate everyone here. Just for fun, I will hand out hugs afterwards. If you’re not familiar with Chef, it enables infrastructure as code. It may seem hard to read since it’s displayed in dark gray here.

00:01:10.080 But this isn’t a talk on how Chef works or other details. I have a number of talks and training classes available for those questions, so feel free to check out those resources. Also, if you scream angrily on Twitter, I will probably respond to help you!

00:01:20.200 Chef provides an MVC framework, which is really neat. So, how does it do that? The node attributes contain information about the system we are configuring. The data we have is what we will use in our configurations. The configured node acts as the view and the recipes are the controller. Since Chef recipes are just Ruby, it gives us a lot of power and flexibility.

00:01:30.720 And because Ruby is a programming language, we want to do things properly and design them well. We’ll package those recipes into what we call cookbooks, which are collections of recipes and supporting code. These cookbooks contain applicable design patterns.

00:01:42.919 Now, I'm not a computer scientist, so if you are and I’m misusing the term 'design patterns,' I apologize. We can have that discussion at the hack fest, which could be pretty cool. However, these cookbooks represent best practices, and when you apply good design patterns and build them into best practices, those cookbooks can be utilized by others.

00:01:55.000 When it comes to best practices, opinions vary. There’s a lot of opinionated software that people are familiar with and love, and there are many best practices encoded into the way systems are built and managed. Thus, it's a great idea to share those opinions, allowing others to offer their feedback.

00:02:07.679 We will be discussing cookbooks. We have many repositories and branches of cookbooks here. How many of you are contributors to the cookbooks repository? A few hands. Okay, now who here has contributed a patch to the cookbook project? Anyone? All right!

00:02:19.920 The next question is: if I told you to sign a contributor license agreement, it’s because we chose the Apache License and wrote a blog post about it. The important part of this agreement is that we’re not going to re-license your contributions; you get to keep those. We just redistribute them for you.

00:02:26.960 So, contributor license agreements are beneficial. You can still have your code reach a wider audience through our community site. We really appreciate our community, and we want to provide a platform for sharing Chef code and cookbooks, enabling ratings and social networking.

00:02:33.760 This is the newly redesigned cookbook site where you can share your work. You’re not required to sign a CLA, and you don't even need to license yourself to post, but please ensure you have the rights to redistribute the content.

00:02:44.519 If you add non-redistributable code to a cookbook and someone complains about it, we will have to take it down due to DMCA requests. Therefore, make sure you can distribute the code you provide. If you’re writing cookbooks yourself, it's essential to follow good design patterns to receive favorable feedback.

00:02:56.360 Cookbooks package up configurations and have various components apart from just recipes. So, let's discuss those components.

00:03:02.720 Who here saw the blog post by Tom Preston-Werner from GitHub? He wrote about readme-driven development, emphasizing that the first thing you should do when starting a new project is to write the README. This helps clarify the code you're set to write and allows for brief documentation.

00:03:18.400 Make sure you write the manual, as the users of your cookbook will appreciate it. After documenting, you should focus on writing some recipes. I attended a large conference in Europe this past year called FOSDEM, the largest open-source convention in the world, with around 5,000 attendees.

00:03:28.119 The talks at FOSDEM were split into different dev rooms, including the Canoe Dev room, where I heard a talk on 'Canoe Parallel.' It replaces Xargs and allows you to run piping commands in parallel. During this talk, I decided to write a cookbook, not because the session was boring but because I wanted to!

00:03:44.399 The software installation process generally involves downloading, extracting, and running configuration commands like 'configure, make, make install.' In Chef, we can achieve this using a remote file resource that downloads a .tar.gz file, followed by a bash script that extracts and builds it. However, this presents a challenge.

00:03:56.079 The challenge includes the static nature of the resource definitions, which do not allow file retrieval from alternate locations. For instance, if we want to install a specific version, we need to account for possible updates that may have occurred during our jobs.

00:04:05.760 Additionally, every time Chef runs, remote resources will download the file and check the checksum, which is inefficient. To address these issues, we can utilize customizable attributes in our cookbooks and approach this design with proper planning.

00:04:15.079 This leads us to our first design pattern: using attributes. In our cookbooks, we can create a default attributes file to define defaults, such as the URL for downloading the desired software and the version we want to use.

00:04:22.879 When defining defaults, we should also explain that some systems may clear the /tmp directory upon reboot. In fact, we can use an internal Chef configuration value called 'file_cache_path'.”

00:04:38.640 This allows us to download files to a semi-temporary location, which is more reliable. This brings me to the second design pattern: exploiting Chef’s internal values. We will rewrite our recipe for the remote file resource using these attributes and the internal values.

00:04:47.439 Now our recipes can incorporate strings and hashes, which allows for more dynamic configurations. The bash script must also utilize attributes, including version attributes, during installations.

00:05:03.799 Moreover, we can utilize Chef's node platform value in attributes files to select specific configurations based on the node's platform. By doing so, we can enhance the flexibility of our cookbooks, allowing them to operate across a range of environments.

00:05:23.520 For example, through platform-specific conditionals, we can modify configurations to meet individual requirements. This versatility is critical for deployment across various infrastructures.

00:05:30.960 So now, let’s run Chef. We can download the remote file, set the file modes, and then execute the script, which builds the software. I wrote all this in about 30 minutes because I’m particularly skilled.

00:05:45.439 Next, we will look at recipes. It’s good practice to split recipes by functionality when building Chef cookbooks. Typically, you will have a cookbook per service type in your environment, promoting modularity and maintainability.

00:06:03.360 For example, later today you will hear about service-oriented architecture and how this affects the way applications are developed. In practice, for each major service, you may have separate cookbooks and recipes tailored to respective services.

00:06:20.080 To achieve good structure, each cookbook should fulfill specific functions, such as having an Apache cookbook that builds Apache and a MySQL cookbook that configures the MySQL server. With clear responsibilities, each cookbook becomes easier to maintain.

00:06:37.440 Additionally, these recipes should strive to avoid hardcoding values since attributes should dynamically encapsulate relevant data. Chef excels in expressing infrastructure data in flexible formats, promoting best practices.

00:06:53.840 One approach involves utilizing data bags – a built-in Chef server feature that creates an accessible store for arbitrary shared data. This allows you to link various resources and data in your recipes.

00:07:12.520 Another key feature of Chef is its rich search API, providing users with powerful discovery capabilities related to node attributes, data, and other information within the Chef environment.

00:07:30.000 The combination of these features enables dynamic configurations, making it possible to publish cookbooks that adjust based on the current infrastructure. By using Chef effectively, cooks can share invaluable configurations and resourceful recipes, allowing other users to connect intelligently.

00:07:45.920 Design credit four is about creating separate recipes. For instance, applying this to a Nagios example involves organizing defined recipes that concentrate on common components for monitoring servers.

00:08:03.200 The client recipes will use Chef's server search feature to identify monitoring servers and add their configurations accordingly. This enables seamless integration of new nodes, allowing for efficient monitoring setups.

00:08:23.040 As monitoring servers dynamically gather statistics, they create an environment where new instances are discoverable, significantly streamlining configuration across the infrastructure.

00:08:40.800 For example, in our example, the server recipe will include code that searches for all nodes with a designated hostname, allowing for appropriate host configuration file generation.

00:08:59.760 We can accomplish this through data lookup and configuration templating. Each time a new server joins the Chef system, it automatically comes under monitoring without any manual intervention, showcasing Chef's streamlined operation.

00:09:20.000 If you use Chef Solo, you won’t have access to many of these features. The Chef server provides persistent node data, arbitrary infrastructure data in data bags, search indices, and a robust API for creating rich environments.

00:09:38.759 Should you wish to continue using Chef Solo, you can check for it using an internal configuration value, allowing you to adapt your environment accordingly.

00:09:58.960 Hardcoding values presents an anti-pattern tendency, making it crucial to share customizable cookbooks that communicate clearly with other users.

00:10:14.760 It's vital to document default attribute values thoroughly within your recipes, as it promotes usability among users – roles play an essential part in this.

00:10:29.680 Once users define roles, you must ensure they permit attribute value overrides, enabling adaptability across shared systems.

00:10:48.160 Moving on to templates and files, these assets are crucial during configuration. For instance, you may need to distribute static configuration files uniquely tailored to different platforms.

00:11:06.759 In the case of MySQL, varying config options arise across platforms, illustrating the necessity to maintain platform-specific attributes.

00:11:17.919 Templates with embedded Ruby (ERB) work extraordinarily well with Chef for generating dynamic configuration files based on node attributes or generated values, making sharing files across systems considerably easier.

00:11:33.839 Libraries within Chef can provide a variety of benefits, including lightweight resource and provider helpers. If Chef doesn't have a specific resource for managing an application, you can create a custom solution.

00:11:54.480 By utilizing library resources within your cookbooks, you can effectively manage additional functionalities and customize code logic for complex applications.

00:12:05.920 Definitions are becoming obsolete since they often resemble resources but lack flexibility. Rather than using definitions, leverage lightweight resources for building new resource types.

00:12:24.159 These lightweight resources and providers contain both actions and attributes, lending themselves to a more organized, maintainable code.

00:12:42.960 As you design new resources, ensure there's clarity in their attributes, especially concerning databases and other crucial components. Utilize community resources to enrich your cookbooks and share openly!

00:12:58.560 Let’s not forget about the importance of cookbook metadata. These definitions play a critical role in establishing dependencies, especially when you're sharing or installing pre-existing cookbooks.

00:13:12.120 Include essential dependencies from standard cookbooks, such as the build-essential cookbook, which supplies necessary compiler tools.

00:13:28.560 I understand we’re fast approaching the end of our time, so let’s quickly go over testing cookbooks. You can treat these cookbooks similarly to other Ruby projects using different testing frameworks.

00:13:40.560 After validating your code, make sure you document the processes to share your efforts and insights with the community. Cookbook examples are available on the community site, ensuring that resources are easily accessible.

00:13:58.480 I will be at the Engine Yard Hack Fest tonight if anyone has further questions about Chef or anything we've discussed today. Thank you all for your time!

MountainWest RubyConf 2011