5 Things You Didn't Know About Chef

by Joshua Timberman

In the video titled "5 Things You Didn't Know About Chef," presented by Joshua Timberman at Big Ruby 2013, the speaker shares five lesser-known features and capabilities of Chef, an automation platform for infrastructure management. The talk aims to enlighten both novice and experienced users about various aspects of Chef that may not be widely recognized.

Key points discussed include:
- In-Place File Editing for Greater Good: Timberman explains a method to modify a single line in a configuration file using Chef's Chef::Util::FileEdit, which although considered an anti-pattern, can be useful in specific scenarios. This method allows for targeted file changes without needing to manage the entire file, highlighting the flexibility Chef offers.
- Using Chef's Built-in Version Matching: This feature helps manage software versions within cookbooks by applying version constraints. Timberman shows how to check version compatibility for operating systems and cookbooks, elucidating how to ensure the correct versions are in place to avoid potential issues during deployment.
- Chef Shell (REPL): The video introduces Chef Shell, a Read-Eval-Print Loop (REPL) that allows users to interactively test and debug recipes. It operates in three modes: main, attributes, and recipe mode, enabling users to explore node attributes and execute recipes dynamically.
- The Anatomy of Loading and Executing a Single Recipe: He discusses the introduction of "Chef Apply," allowing users to run recipes without building complete cookbooks. This functionality simplifies the process of prototyping and testing Chef code.
- Conclusion and Community Engagement: Timberman emphasizes the growth of the Chef community and advises viewers on engaging with resources like learnchef.com and ChefConf. He encourages attendees to explore various tools available for testing and managing cookbooks.

The main takeaways from the presentation are Chef’s versatility in configuration management, its robust tools for handling versioning, and the importance of community support in navigating the complexities of infrastructure management. This talk not only addresses functional features of Chef but also promotes a cultural understanding of collaboration within the Chef community.

00:00:19.359 To get started, I need to take off this chef hoodie and change it to my chef t-shirt because that thing's warm. All right, so let's talk about five things you didn't know about Chef.

00:00:26.800 Before I dive into Chef, I want to share a bit about myself since this really is about me. I'm a Technical Community Manager at Opscode. What that means is I handle a lot of project management related to our community.

00:00:39.680 By background, I am a systems administrator, so I'm an ops guy. I enjoy looking at sparkly graphs of all those cool code metrics you just added to your application.

00:00:46.800 I also get paged in the middle of the night when your code falls over for some reason—not that your code falls over. That never happens to anybody's code here. I'm just speaking theoretically. I work on our open-source Chef cookbooks.

00:01:04.400 If you've used Chef, you may have come across cookbooks. If you have used the ones from Opscode, chances are my code is running in your production systems. Sorry or, you know, you are welcome, depending on how well that has worked out for you. I like to drink coffee and beer. I haven't had any beer yet, so that means I've only drunk coffee so far. So, I'm going to talk really fast, and I apologize for that in advance.

00:01:33.840 So, who are you? Who by trade are systems administrators who have mainly done operations work? A couple of people, maybe? Who's touched systems, logged in, and done SSH? Okay, cool! You recognize that prompt. Did you ever check the RSA fingerprint of your SSH keys? No? I do, because I'm weird like that.

00:01:51.759 Who's primarily a developer? It should be like everybody, right? Who are the business people? I have 'manager' in my title; that kind of makes me a business person. If you’re a consultant, you're business people; you've got to do the accounting, coding, and everything else, right? Who's in DevOps? A few people are DevOps, actually. If you listen to Adam Jacob's talk on the topic, DevOps is a movement. You can no more be a DevOp than you could be hip-hop, as hip-hop is a cultural movement.

00:02:22.800 But that’s neither here nor there. Who’s using Chef now? A few people. Who wishes they were using Chef? And who wants to hear me talk about Chef versus Puppet? Sorry, I have 72 slides—actually, 74; I added a couple after I added this one. I did a check: I have 97 build steps, so there's a lot of content in here. I will actually go through some of these slides fairly quickly, so let's get started.

00:02:59.760 So, first of all, what is Chef? It's an automation platform for developers and systems engineers to build and manage infrastructure as code. It's a technical domain revolving around building and managing infrastructure programmatically—that's infrastructure as code.

00:03:22.319 So, who doesn't like ducks? Nobody here dislikes a duck; everybody loves ducks. Chef is a framework built to bring you the benefits of configuration management across your entire infrastructure.

00:03:38.799 We are talking about fully automated infrastructure—soup to nuts, the whole deal. Chef has design principles that are very important to understand. The first one is that it's reasonable. You can look at the code in your Chef scripts and understand what's going on; you can reason about it. It has a declarative syntax, which is pretty easy to string together. You can make sense of things.

00:04:14.640 So when you're trying to fix systems at 3 AM, you can reason about it and figure out what's going on more easily. Chef is idempotent and convergent. Idempotent, in this regard, is not in the pure mathematical sense but it reflects the idea that when Chef is configuring a system, it only takes action if it needs to. If a package is already installed, Chef doesn't try to reinstall it. But Chef is convergent, so if the package is already installed but you told it to upgrade to a newer version, Chef will handle that for you.

00:04:44.479 Chef is also designed to be flexible. In the world of managing systems and infrastructures, every infrastructure is a unique and special snowflake, even though they are all pretty much the same. You know you've got a back-end database, a queue, a cache, a web app, an iPhone app, and so forth. However, there are all these little differences that make companies unique.

00:05:26.639 Their infrastructure is unique and brings its own unique set of problems and challenges, so you need a tool like Chef that is flexible in helping you overcome those challenges. Chef offers a library and a set of primitives. Some of the things I'm going to talk about involve using those primitives to build the automation platform that you need. How many of you have ever been involved in Perl or the Perl community? You might recognize this acronym: ‘There's more than one way to do it.’ This idea conveys that there isn’t just a single way to write a particular code; there are many ways.

00:05:56.560 Chef adopts this idea to infrastructure, as every infrastructure is different, so the best practice for managing infrastructure with Chef at a bank will differ from that in a media company and also differ from a web startup.

00:06:01.600 Putting all of this together, Chef is a framework that embraces these principles and enables infrastructure as code. You write Chef resources into recipes, which are essentially blueprints for managing your desired state.

00:06:30.080 For example, consider a recipe for installing and managing HAProxy. You use a declarative syntax to collate all your recipes into cookbooks. These cookbooks are then composed together with roles, and you can store the entire setup as source code in a Git repository, Subversion, or Bazaar, or whatever version control system you use. Now that we all know what Chef does and we're becoming experts on the subject, let’s discuss some of the things on my agenda. I'm not going to read through all the slides because I have several slides about each of these topics.

00:07:17.200 I've got five topics here, and given the amount of time, I may or may not reach the bonus round. So if we get to it, there will be a bonus round, as well. The first topic is in-place file editing for the greater good. I will explain what that means shortly. First, I need to update one line in this config file. Does anybody recognize what config files are in the background? Call it out if you know it.

00:07:47.840 That's right! This is the default IPv4 and IPv6 settings. I couldn't find an example that had a good mix of other tuning, but maybe you want to modify just that one right up there, TCP SYNACK retries, because that is the only one you care about in all of sysctl. Perhaps you're a PHP developer or you maintain a legacy PHP application and need to update the .ini file to a single directive. There's a danger in this, and some might label it as an anti-pattern, but I prefer to refer to alternate implementations.

00:08:29.760 The danger here is that it makes it hard to reason about managing this file. If I've got sysctl with a plethora of configuration settings and my application needs to change just this one line, what if someone else's application running on the same system needs to change another line? Now we've got two potential places where we’re making edits to the same file, and that brings a risk.

00:09:15.200 Generally, what we recommend is managing the whole file. Chef has several resources for managing files, such as templates, cookbook files, and remote files, which manage the full content of a file. You can render out a template like the earlier ERB Puppet example. There’s similar functionality in Chef.

00:09:53.440 We recommend doing it that way, but sometimes you just need to modify a single line in a single file. A common example of this is modifying /etc/hosts, so Chef has a Ruby block resource, and we can insert some Ruby code there to be awesome.

00:10:25.520 The heart of it is this class built into Chef that isn’t widely documented because we consider it an implementation we'd prefer not to use. However, we will explore how to use Chef::Util::FileEdit, where we will pass in the argument for the path to the file we want to manage and then send the appropriate file edit method to that object.

00:10:55.760 The first argument for this particular method is a regular expression, and the second argument is the new string that we want to insert into the file. So when Chef processes this code, it will look at /etc/hosts and replace the localhost entry with the new fully qualified domain name and hostname that we've set for localhost. Then it writes the file to disk.

00:11:22.480 This is a Chef resource. However, we haven’t ensured that this implementation is idempotent. So every time Chef runs, it will execute this code. Hopefully, our regular expression is accurate, and we haven’t introduced new problems with it. We could put in a guard that says do not perform this if it already exists. The search file replace line has some of this functionality using the regular expression.

00:12:03.600 Chef::Util::FileEdit has several other methods that are less commonly used but can be applied. One is the search file under place, which replaces all occurrences of the regular expression with the text to replace it. There’s also the insert line if no match, which appends a line at the end of the file if it doesn't already exist.

00:12:39.040 This method ensures a configuration setting is always in the file. Of course, there's also the write file method, because it wouldn’t help to make all these text replacements without writing the file content out.

00:13:06.560 Sean O'Mara, one of Opscode's consultants, wrote a cookbook called 'Line' that gives you a resource and provider for managing single lines. We hadn't done this until he wrote it, as we generally consider this to be an anti-pattern. However, as we see Chef used and adopted by more organizations, we note different implementations and pain points that users want to address.

00:13:42.360 This particular issue pertains to the monolithic, huge config file for an application where all someone needs to do is update this one line for a new deployment method or any such change. If you go to ckbk.it/line, it takes you to the Chef community site where you can download that cookbook.

00:14:31.440 On to the next topic: using Chef's built-in version matching. When managing infrastructure and software, there are many places where we want to match versions. Platforms all have different versions, which can be fun—or challenging. Cookbooks have different versions as well. When you use Chef, cookbooks are bundles of code. When you have bundles of code, you usually want to package and version them so you know which iteration you are working with.

00:15:06.079 Version constraint support for Chef environments was added in Chef version 0.10 to match cookbook versions. However, it has many uses beyond that inside Chef. The version constraint class can match versions that are in dotted notation up to three places. For example, you can use 5.6, 1.0, or 1.8.0, but it does not support alphabetical versions like 1.0.0.dev or rc.1.

00:15:39.201 You can, however, use it for things like operating system platforms, cookbook versions, and most software that follows the numeric, three-dot notation. The common examples I've found while implementing this have been checking the version of Ubuntu—ensuring I’m running a version greater than or equal to 10.04 or checking for 6.3 on CentOS because a change in 6.4 could impact an application if I modify it.

00:16:30.080 The version constraint class allows the use of standard comparison operators to work with these version strings. It's not a lossy system as it might be if you were operating with floating points. The documentation for this is available on the Chef website for additional information.

00:17:05.760 To use it, require the library, initialize an object with the version constraint as its argument, and then use the include? predicate method with the version you want to match. The constrained operators should look familiar to anyone who uses any version constraint matching system such as RubyGems. You have operators like equal to, greater than, less than, and others.

00:18:02.080 Here's some code that showcases how to use the version constraint. You require the Chef version constraint. I've got three comparisons on an OS 10 system where I'm checking if the platform is greater than or equal to 10.7, testing on 10.6—is false. If I’m on a lion system at the 0.3 release, then it will evaluate to true, or on mountain lion, that’s also true.

00:18:38.080 We can also use any node attribute that could contain that version string. For example, on CentOS 5.8, node platform version will be 5.8, while 6.3 will satisfy the criterion of being greater than 6.0. This is quite straightforward, just a couple of lines of code enable neat version comparisons.

00:19:06.720 If we’re on Debian 6.0.3 and then we apply a floating point comparison with dot 2f, we get 6.0 as the result, but that's dropping the third digit, which might be fine.

00:19:36.640 So if we conduct a comparison and verify if it’s less than or greater than, well—6.0 is less than 6.0.3. But when we drop the .3, it’s less, which may or may not be what we want.

00:20:07.440 Similarly, if we check for being greater than, 6.0.3 is indeed greater than 6.0, but when we drop it, it’s equal.

00:20:38.000 Chef version constraint helps resolve this complexity when you check if it’s greater than 6.0; it includes that condition, and it's quite handy. That's all I have to say about Chef version constraints. It's reasonably straightforward.

00:21:14.160 Now, who likes REPLs? REPLs are pretty awesome! Who’s used IRB? If you've never used IRB, you should. REPLs are particularly handy when debugging your code and understanding how things function. Chef has a REPL called Chef Shell. In Chef 11, the REPL is called Chef Shell, while Chef 10 had it named Chef.

00:22:31.680 There's a reason for the name change, as it was infamously called Chef with an S. Although great for typing, it didn't fit phonetic pronunciation in English very well. Chef Shell is built on IRB, so it's essentially IRB running with some Chef components loaded in, with three modes or contexts: main mode, attributes mode, and recipe mode.

00:23:02.640 The main context is generally used if you want access to node attributes, allowing you to explore and examine the available attributes. You can also interact with the Chef node object. Let's fire up Chef Shell, my live demo. I've learned my lesson: don’t run live demos unless you're Jim Wyrick because he does great live demos.

00:23:40.640 I’ve fired up Chef Shell, and it prints out some information. I'm using Chef version 11.2, and I have a Chef prompt, which indicates that I'm in the main context. From here, I can get help by typing 'help' or even 'halp' due to internet memes!

00:24:06.000 A lot of commands are available to interact with the Chef Server, including switching to other modes. In the main context, many commands are available for working with the API; for example, retrieving information from the nodes endpoint to query all your nodes.

00:24:32.080 This only works if you're configured to use a Chef Server. Let’s restart Chef Shell with a Chef Client configuration. If you're using Chef Shell Z (or dash Z if you're in the UK), it will automatically load /etc/chef/client.rb. You can pass in a different configuration file using the -C option.

00:25:02.080 You can enter Chef::Config and specify the Chef Server URL. This is a hash-like object with all your configuration. As a result, we can connect to the Opscode hosted Chef server and get information about the node, such as its run list.

00:25:33.680 Given that I have this run list and the knowledge of the node object, I can talk to the Chef Server and determine which cookbooks need to be downloaded. I can download those cookbooks automatically within Chef Shell and debug any issues or explore how something works.

00:26:18.960 The Attributes mode corresponds to a cookbook's attribute file. How many people here have worked with Chef and understand attribute files? This is where you set up cookbook-specific information, like which port Apache should listen on or what username should deploy an application.

00:27:01.760 You can enter Attributes mode in the same context as a cookbook attribute file. This feature is helpful if you want to paste content from your attributes file into Attributes mode, and it will operate as Chef would during a run. The order of operations for loading attributes aligns with documentation on the Chef website.

00:27:39.360 We’ve already started Chef Shell. We go into Attributes mode and use the 'default' method. Default is a node method on the Chef node object, and I'm going to give it a key name—let’s call it chip—with my amusing personal touch. It allows us to explore that functionality.

00:28:15.920 In older versions of Chef than Chef 11, you couldn't modify the node attribute directly. You had to use one of the precedence methods, like default, to set the attribute. So, if I try to set 'node.chef equals awesometown,' it promptly throws an exception stating the variable is immutable.

00:28:42.560 For that reason, we use node.default. Now that node's attribute equals awesometown. There’s no semantic difference between 'node.default' and 'default' in an attributes file or mode; it's all managed under the context of a Chef node object.

00:29:18.720 Now you might ask, why not just always require 'chef.' and then use 'default' from the attributes files? This reflects the evolution of a product; we want users to not have to modify all their attribute files for legacy cookbooks.

00:29:59.120 Chef Shell's Recipe mode is likely the most engaging element, as it behaves as if you are writing Chef recipes directly. This is akin to the context of a Chef recipe, meaning all the resources and DSL methods within Chef are available. If you’ve configured this correctly, you can even retrieve the necessary cookbooks.

00:30:41.440 This high functionality is exceptionally valuable for debugging, similar to any other REPL. You can run it in client mode to interact with the Chef Server and verify if a search operation in your recipe behaves as expected.

00:31:25.120 You can also inspect the effects of your search or figure out why it’s not working correctly. Starting Chef Shell in client mode allows you to write a couple of resources. For instance, if I have a package, say Git, and I want to install Emacs, it doesn't need to be fancy.

00:32:07.520 So I’ll write a recipe to ensure both get installed. After adding those two resources to the recipe, running "resources" returns an array of all the resources Chef understands about.

00:32:52.480 When Chef runs, a run context is created. I've only introduced a few resources during this Chef run. After Chef processes all the resources, it iterates through them and configures them in the order they were added.

00:33:29.600 In a frequently used recipe like this, we can't start, enable, and run the Apache 2 service until the Apache 2 package is installed. So if we utilize the 'resources' method exemplified in Chef Shell, we will see that with the index allowing for proper processing.

00:34:06.400 By using run_context.resource_collection, we can access a resource collection object with all information about the resources. There’s a lot of flexibility in adjusting the resources as needed.

00:34:34.560 Moving on to the next topic: the anatomy of loading and executing a single recipe. Chef Apply is a new program included in Chef 11 and is accessible when you install the Chef packages from Opscode. For Windows users, it is available in your default path.

00:35:06.080 This concept originated from a gist by Dan DeLeo, an Opscode engineer. He showcased that you can run a Chef recipe without creating the full structure of a cookbook. Subsequently, Brian Berry, a community member, created a ticket to add Chef Apply to Chef.

00:35:39.920 Brian's contribution included various additional features, which I will cover towards the end of this section. First, we'll need to require the necessary libraries for Chef, including the Chef client, resources, and providers.

00:36:19.200 We'll also set up the run context. This involves ensuring Chef operates in solo mode, which prevents it from attempting to connect to a Chef Server—helping avoid timeouts.

00:36:50.080 Next, we create an instance of the Chef client class to keep track of the main object during the Chef run. You will then build the node, providing Chef with the name of the node they’re managing—though it won’t matter for intents and purposes due to not communicating with a Chef server.

00:37:34.000 Afterward, you instruct the client to run Ohi, a discovery tool giving attributes about the node (e.g., IP address, platform, and version). This process culminates in building the node object with applied attributes.

00:38:15.360 After setting up the run context—which requires a node object, cookbook collection, and client events—you load the recipe. The Chef Recipe class takes in both the cookbook and the specific recipe you intend to load.

00:39:00.800 Once again, the context here would differ if using a full Chef cookbook collection with Chef solo or Chef Client, where it wouldn’t be an empty hash but filled with several cookbooks and recipes.

00:39:32.000 After creating the recipe, you can load it from a file. Then you establish an instance of the runner class, which executes Chef, managing the resource collection during the run.

00:39:50.000 So taking the previous three resource example and executing Chef Apply against that file will install the Apache 2 package. The service configuration might fail if there isn’t a linked cookbook for that template, but you can adapt as needed to control the local source.

00:40:17.440 Chef Apply allows for prototyping recipes and quickly executing them. It is possible to integrate with standard input, such as endlessly running a netcat session on a specified port, where you can run Chef Apply with specifics.

00:40:50.000 However, be careful, as you should never execute such code directly unless in a VM. In conclusion, there’s a vast expansion to Chef beyond what I’ve covered. We recently launched learnchef.com, which provides several quick start guides and walkthroughs.

00:41:16.000 We are developing use cases and resources for learnchef.com, including the Opscode installer, a complete stack installation package for Chef. You can also find extensive documentation.

00:41:43.520 The mailing list hosts the majority of discussions, and there’s also IRC for real-time conversations. We have a community site where you can download pre-made cookbooks.

00:42:18.320 I’m Joshua from Opscode.com and jay timberman on pretty much anything related. Chef Conf is coming up in April; you can use the promo 'big ruby' for a 10% discount.

00:42:43.680 If you want to attend ChefConf and meet other Chef users, where we’ll have workshops and presentations galore. You can visit chefconf.opscode.com to claim your discount.

00:43:13.279 Finally, I may have time for a bonus round; I’ll run through some frequently asked questions. How do you test cookbooks? First, attend Nathan’s talk tomorrow; he will cover testing extensively.

00:43:44.160 Chef 1014 above has a run mode like 'no-op' or 'dry run. In the community, we have a tool called Test Kitchen, along with community tools like ChefSpec and Cucumber Chef.

00:44:08.640 Nathan Harvey will delve into this and more, so don’t miss it! Next FAQ: 'Why is Chef Server so hard to install?' Is it because you want people to use Hosted Chef and Private Chef? Well, yes, we do want high utilization of hosted solutions.

00:44:45.520 However, that’s not the reason. It isn’t hard anymore. We offer a full stack package that streamlines installation. Just retrieve the package and install it, and boom—it’s functioning. Remember, config management systems have complex components.

00:45:06.080 There are databases, search engines, APIs, and other facets making it non-trivial to build. The Chef Server is no different. Also, for those wondering about Chef versus Puppet, I will be at the GitHub drink-up tonight.

00:45:32.080 That’s it! Thank you!

Big Ruby 2013