RailsConf 2015

Summarized using AI

Test Driving your Rails Infrastructure with Chef

Nathen Harvey and Nell Shamrell-Harrington • May 21, 2015 • Atlanta, GA

The video, titled "Test Driving your Rails Infrastructure with Chef," features Nathen Harvey and Nell Shamrell-Harrington presenting at RailsConf 2015. The main focus of the workshop is utilizing Test Driven Development (TDD) principles to manage Rails infrastructure through Chef, a configuration management tool. The presenters emphasize the significance of integrating development and operations practices, showcasing how TDD can enhance infrastructure management and deployment practices.

Key Points Covered:

  • Introduction to Chef and Infrastructure as Code:

    The session begins with an introduction to Chef, describing it as an open-source framework to handle infrastructure management using a policy-based system. The presenters discuss the importance of treating infrastructure like code by employing version control and testing.

  • Why is DevOps Important?

    Nathen Harvey shares a personal anecdote about how the collaboration between different music genres in the '80s exemplifies the concept of DevOps, highlighting the need for collaboration between developers and operations.

  • Infrastructure Automation:

    Key strategies in automating infrastructure are discussed, emphasizing the necessity of automation as applications scale. Harvey underlines that infrastructure as code enables teams to programmatically provision and configure their infrastructure components.

  • Policy Management with Chef:

    The presenters explain how Chef allows users to capture the desired state of their infrastructure, managing its components effectively. The Chef server holds policies and information about various nodes, streamlining management efforts.

  • Hands-on Lab Activity:

    A practical lab section is included where attendees are guided to install a text editor using Chef, demonstrating the process to apply Chef components in real-time scenarios.

  • Testing Infrastructure Code:

    The importance of testing in the infrastructure code lifecycle is emphasized, suggesting that rigorous tests are essential before deployment. Key questions for validation include checking the success of the Chef client, ensuring nodes are in the desired state, and adherence to style guides.

Conclusion and Takeaways:

  • The workshop ultimately encourages infrastructure management with a TDD mindset, ensuring safety and reliability in deployments.
  • Attendees are expected to experiment with Chef to manage their Rails applications effectively, applying the principles of TDD to their infrastructure needs.

In summary, the session provides insights into the integration of development practices with operations through robust infrastructure management, driving home the importance of collaboration, automation, and testing in modern software development environments.

Test Driving your Rails Infrastructure with Chef
Nathen Harvey and Nell Shamrell-Harrington • May 21, 2015 • Atlanta, GA

by Nathen Harvey and Nell Shamrell-Harrington

Managing your infrastructure with configuration management tools like Chef melds the practices of development and operations together. This workshop will focus on a development practice - Test Driven Development - and how that method can be applied to managing your Rails infrastructure and deployments. You will learn how to: Analyze your application and define your infrastructure needs (databases, load balancers, etc.), define unique infrastructure requirements for Rails applications (i.e. asset pipeline), capture your requirements in tests using Test Kitchen, ServerSpec, and other frameworks

RailsConf 2015

00:00:12.200 What's up? Welcome to our talk on infrastructure coding with Chef.
00:00:18.800 We're going to introduce ourselves, and then we'll get into the presentation.
00:00:25.760 If you want to grab all of the slides and code that we're going to walk through right now, you can click the link.
00:00:31.400 I understand it’s difficult to click from your chair, but give it your best shot.
00:00:37.000 Alright, let's continue. Why don't you introduce yourself while I figure out why my slides aren't working?
00:00:43.239 I'm Nell Shamrell-Harrington, a software development engineer with Chef. If you've seen me speak before, you may know about my deep love for regular expressions.
00:00:49.000 I won't talk about that today, but if you're interested, I have plenty of work on it, and I'm always happy to discuss it.
00:00:54.879 The main thing I work on in Chef is the Supermarket product. It’s a Ruby on Rails application where we store all community cookbooks.
00:01:03.280 People can upload cookbooks, and others can download and install them directly from Supermarket.
00:01:08.960 You can check it out at supermarket.chef.io.
00:01:14.000 Finally, I'm a naginata practitioner, which is a Japanese martial art. If you're interested in learning more about that, just let me know.
00:01:19.479 Now, over to you, Nathan. Awesome, thanks Nell.
00:01:24.560 Hi everyone! My name is Nathan Harvey, and I'm a community director at Chef.
00:01:29.640 This means that if I tell you how to run your infrastructure, you should probably not believe me.
00:01:36.600 I don’t manage infrastructure; I manage a community and work with people.
00:01:42.159 However, I used to manage infrastructure, so I had some credibility in the past.
00:01:49.640 It's cool, though. I also co-host the Food Fight Show, a podcast about DevOps and Chef.
00:01:56.280 I'd love for you to subscribe and listen to that. You might be disappointed, but let me know if you do.
00:02:01.960 I occasionally farm and love eggs, though if you click on either of those links, you'd understand why I probably no longer love eggs.
00:02:08.599 You can follow me on Twitter or email me if you want to get in touch.
00:02:14.319 Now, let's get started! I'd like to know a little bit about you.
00:02:19.560 If you are a system administrator, could you raise your hand? Or if you have been one in the past, please do the same.
00:02:26.319 Okay, cool! How many of you are developers?
00:02:32.519 Awesome! And even some of those system administrators raised their hands, which is a crazy thought.
00:02:39.680 How many of you are into DevOps? That's good stuff!
00:02:45.239 And business people, how many of you are business people?
00:02:51.959 You're all business people. If you write software, you are a business person. If you are a system administrator, you are a business person because you work at an organization that cares about customers who use your stuff.
00:03:07.080 You have a vested interest in that. How many of you are experienced with infrastructure as code or with configuration management?
00:03:13.959 Awesome! And how many of you are experienced with Chef? You can lie; I'd rather you didn't lie, but it’s cool to do so.
00:03:28.319 When you're experienced with Chef, and you're here, what I always tell folks is that when we do introductory talks, your job is to keep your mouth shut if we lie. Don't let anyone else in on the secret that what we’re doing is telling them the hard way how to learn.
00:03:40.480 I want to start with a story. I grew up in the '80s, and a little while ago I was watching a show on National Geographic called 'The '80s: The Decade That Made Us.' This show defined DevOps in a really clear way for me.
00:03:53.319 Many developers come up to me and say they hate DevOps. The problem is that they just don't understand what DevOps is, or rather, we don't have a common understanding of what the definition is.
00:04:05.400 This amazing show on National Geographic explained DevOps to me, so I want to share that.
00:04:11.120 To do so, I have to take you back in your mind to the '80s. Back then, there were bands on the radio, and you couldn’t listen to them on the internet yet. You know, Al Gore hadn't invented it, actually.
00:04:28.280 There were rock bands and rap bands. The two would never collaborate with one another - ever! One of my favorite bands was Run-D.M.C. and they had a crazy idea.
00:04:38.840 What if Run-D.M.C. made a rock song? It’s important to note that they were a rap band, and as I said, they would never think about making a rock song.
00:05:00.919 Somebody said, "You should totally do this!" So they listened to the rock song, and they said, "Oh hell no! This is hillbilly gibberish!" They did not understand what was happening.
00:05:14.440 As an assistant administrator, I related to that. When developers hand me code, I say, "I don’t understand this code!" How am I supposed to run this in production?
00:05:26.960 However, they talked to the rock band Aerosmith, whose song they were going to cover. Steven Tyler said, "What the hell are they doing to my song?" That sounds like a developer coming up to me and saying, "Hey, it works fine on my machine! What’s your problem?"
00:05:42.760 But this amazing thing happened. The producer Rick Rubin put them both in the same studio, and in his words, it was crazy good. The video was a perfect description of DevOps.
00:06:06.440 In the music video, you start off with the rockers on one side and the rappers on the other side, with a literal wall separating them. During the video, they tear down that wall and start to work together.
00:06:29.020 The thing about DevOps is that they didn't ask Run-D.M.C. to become rockers, nor did they ask Aerosmith to start being rappers. They said, "Keep what you're good at; just work together to make the audience happy."
00:06:41.639 I want to propose a definition of DevOps, so whenever you hear someone saying 'DevOps,' think of it as a cultural and professional movement that focuses on how we build and operate high-velocity organizations born from the experiences of its practitioners.
00:06:58.199 This is how I define DevOps, and when someone says they hate DevOps, I respond that we do not have a common understanding.
00:07:12.960 It doesn't matter, because I’m on the stage. Now, you all agree this is our common definition of DevOps. Are we good? Alright, let’s talk about infrastructure as code.
00:07:29.639 I had to rant a bit about DevOps; I thought that was a fun journey.
00:07:37.639 So when we treat our infrastructure as code, we start automating the infrastructure that runs our applications.
00:07:49.840 Applications that work on your laptop are not of much value to anyone other than you. We want to take our applications and put them into real infrastructure.
00:08:01.440 This could be on the cloud or in your data center, and by doing so, we want to automate the entire process.
00:08:07.199 When do I need to automate? Isn’t git push Heroku a form of DevOps? Sure, of course you are, if you’re git pushing to Heroku.
00:08:18.960 But at some point, you may want to put your applications on some other infrastructure. You will know this because you will have reached some scale.
00:08:30.120 And that scale doesn't necessarily mean the number of machines; it can correlate with the number of people who build your infrastructure, the complexity of those applications, and so forth.
00:08:54.240 Therefore, you need an automation platform that can handle your infrastructure needs and help you treat it as code.
00:09:06.399 I’m not going to review all the bullet points on this slide; the slides are available for you to read for yourself.
00:09:19.760 The platform you choose should be one that works well for you and your team.
00:09:25.959 When it comes to infrastructure as code, it means programmatically provisioning and configuring components within your infrastructure, whether it’s hardware, cloud instances, or whatever.
00:09:32.200 Treating your infrastructure code like any other code base is essential. This means using source control and testing your infrastructure code.
00:09:49.040 So, what does it mean to treat your infrastructure like any other code base? The audience can participate here.
00:10:00.399 First, you need source control. You put your infrastructure code into a version control system.
00:10:10.399 What else does it mean? Writing tests for your infrastructure code. We should write tests.
00:10:19.160 Your infrastructure code is a continual experiment; it’s never done, and changes must be tested just as application code is.
00:10:29.880 If you treat your infrastructure as code, you can reconstruct your entire business with three things: your code repository, a backup of your data, and compute resources.
00:10:41.480 To share a personal experience, when I first became a system administrator, I was terrible.
00:10:53.320 At that point, I realized my next change to my production environment was going to start with a commit to a Git repository.
00:11:01.479 That day, I felt free; no longer would I log into production and bang on the keyboard, hoping for the best.
00:11:12.200 With Chef and with many infrastructure-as-code platforms, you're working with a policy-based system.
00:11:24.320 In your code, you capture the policy for your infrastructure. What are the requirements of your infrastructure?
00:11:36.839 Of course, we’re mostly talking about Chef today, but each of these systems will have an application to ensure the policy is continually applied.
00:11:49.360 As your needs change, you can change your policy, and your infrastructure will be updated. This is great unless you write the wrong one.
00:12:04.440 For instance, if you change your policy and incorrectly apply it, you could bring down your entire infrastructure.
00:12:19.600 This is why we test. So let's take a very simple infrastructure: we have some chefs, some Rails, and a little database.
00:12:29.400 Let's say a security firm comes in and conducts an audit. They might say, "Your infrastructure is not secure because SSH is listening on port 22."
00:12:48.560 To secure it, they will recommend changing the SSH port.
00:12:53.880 The thing is, manually editing configurations can be a headache. A platform can solve that for you!
00:13:05.000 Chef is one such platform. What is Chef? It is an open-source framework that helps you manage your infrastructure as code using a policy-based system.
00:13:21.480 With Chef, you can capture the desired state of your infrastructure, influencing how your nodes behave. It allows them to be consistent with the policy you defined.
00:13:46.760 The Chef server stores the policies and states of your infrastructure. When a Chef client runs, it asks the Chef server, "What is my policy?"
00:14:04.840 Each machine in your infrastructure follows a subset of those policies we refer to as the Run List.
00:14:17.720 For instance, if a node needs to operate as a web server, the Chef server knows it must run NTP to synchronize its clock.
00:14:35.680 This makes sure you are always aligned with the configurations set in the Chef server. The node itself will take steps to ensure compliance.
00:14:53.920 So that’s the first half of storing policies and states within your Chef server. It also has a searchable index of data about every node.
00:15:09.960 This is extremely helpful, for example, when you are doing some load balancing. Let’s take HProxy as an example.
00:15:27.320 When configuring HProxy, you need to determine who can send requests and to whom those requests will be sent.
00:15:35.160 The problem is that your infrastructure is dynamic, and you're adding and removing servers all the time.
00:15:49.840 With Chef, because it has a view of your entire infrastructure, HProxy can ask the Chef server, Who are the web servers?
00:15:56.720 The Chef server consults its search index and returns a list of the web servers. HProxy can then rewrite its configuration file accordingly.
00:16:05.120 And the next time it runs the Chef client, it can ask again, ensuring the latest changes are reflected.
00:16:21.599 What questions do you have before I move on? Is this all clear? How many of you knew everything I said when you walked in the room?
00:16:33.360 If you learned something today, that would be awesome. Let’s discuss building out your policy next.
00:16:50.560 How do we actually take Chef and use it? At its very core, we have a concept called a resource.
00:17:03.080 Resources are pieces of the system in their desired state. For example, it may be a package that should be installed.
00:17:12.000 It could be a service that should be running or a cron job on your system that you need to manage. Each one of these different things is a resource.
00:17:35.240 We’ll give you a workstation to play around with and write some Chef code here.
00:17:46.960 In our first lab, the problem is that our workstation doesn’t have a text editor installed on it. So we’re going to solve that.
00:18:02.160 You will need a card to do the labs. Go to the URL provided to view cards on the back.
00:18:10.960 If you don’t have a card and want to participate in the labs, raise your hand, and we will get you a card.
00:18:25.160 You’ll need an internet connection, which RailsConf has been great about, and something with which you can SSH.
00:18:36.160 If you’re on Mac or Linux, you can use Terminal. If you’re on Windows, make sure you have PuTTY or any SSH client installed.
00:18:49.040 Find the color of your card and match it up with an IP address. Then you can SSH into the box.
00:19:03.360 The username will be Chef, and the password is chef.io.
00:19:18.000 I mention this only now because I didn’t want to put it on the slides, so everyone would know the password.
00:19:34.080 I will also do the lab with you, so you can see how it’s done.
00:19:50.400 Let’s show the slides where we’ll describe how we’re going to install a text editor, which seems to be the clear winner.
00:20:07.960 The goal is that we will install Vim, but we won’t use the standard package manager because that wouldn’t be very Chef-like.
00:20:24.559 Instead, we will use Chef to do this. First, you'll type `sudo chef apply` followed by `-e`.
00:20:42.720 You’ll type in the package details, which should resemble something like: `package 'vim'`.
00:21:03.960 Once you’ve done that, hit the magic button. Be sure to replace 'vim' with your preferred text editor if necessary.
00:21:19.840 After that run, it should take a minute, and then you can verify by running `which vim`.
00:21:39.740 That command should ensure Vim is now installed on your system, and you will be able to use it.
00:22:02.280 Again, I stress that you need a proper understanding of how the Chef works and how it automates the management of resources.
00:22:11.240 Focusing on our test-driven approach, we want to ensure our code works reliably.
00:22:24.000 We will explore both unit and integration tests to validate our Chef code.
00:22:37.640 The ultimate purpose of tests is to ensure the safety of our code before deployment.
00:22:50.360 The four main questions we need to ask before deploying are whether the Chef client completed successfully, if the recipe set the node in the desired state, if the resources are properly defined, and if the code follows our style guide.
00:23:02.440 Let’s jump into answering these questions using a simple scenario to explore the facets of test-driven infrastructure code.
00:23:10.520 But first, let's take a ten-minute break. We'll resume shortly with an exercise where you will write code for test-driving Chef.
Explore all talks recorded at RailsConf 2015
+122