Nathen Harvey and Nell Shamrell-Harrington
Test Driving your Rails Infrastructure with Chef
Test Driving your Rails Infrastructure with Chef
by Nathen Harvey and Nell Shamrell-HarringtonThe video, titled "Test Driving your Rails Infrastructure with Chef," features Nathen Harvey and Nell Shamrell-Harrington presenting at RailsConf 2015. The main focus of the workshop is utilizing Test Driven Development (TDD) principles to manage Rails infrastructure through Chef, a configuration management tool. The presenters emphasize the significance of integrating development and operations practices, showcasing how TDD can enhance infrastructure management and deployment practices.
Key Points Covered:
Introduction to Chef and Infrastructure as Code:
The session begins with an introduction to Chef, describing it as an open-source framework to handle infrastructure management using a policy-based system. The presenters discuss the importance of treating infrastructure like code by employing version control and testing.Why is DevOps Important?
Nathen Harvey shares a personal anecdote about how the collaboration between different music genres in the '80s exemplifies the concept of DevOps, highlighting the need for collaboration between developers and operations.Infrastructure Automation:
Key strategies in automating infrastructure are discussed, emphasizing the necessity of automation as applications scale. Harvey underlines that infrastructure as code enables teams to programmatically provision and configure their infrastructure components.Policy Management with Chef:
The presenters explain how Chef allows users to capture the desired state of their infrastructure, managing its components effectively. The Chef server holds policies and information about various nodes, streamlining management efforts.Hands-on Lab Activity:
A practical lab section is included where attendees are guided to install a text editor using Chef, demonstrating the process to apply Chef components in real-time scenarios.Testing Infrastructure Code:
The importance of testing in the infrastructure code lifecycle is emphasized, suggesting that rigorous tests are essential before deployment. Key questions for validation include checking the success of the Chef client, ensuring nodes are in the desired state, and adherence to style guides.
Conclusion and Takeaways:
- The workshop ultimately encourages infrastructure management with a TDD mindset, ensuring safety and reliability in deployments.
- Attendees are expected to experiment with Chef to manage their Rails applications effectively, applying the principles of TDD to their infrastructure needs.
In summary, the session provides insights into the integration of development practices with operations through robust infrastructure management, driving home the importance of collaboration, automation, and testing in modern software development environments.
00:00:12.200
What's up? Welcome to our talk on infrastructure coding with Chef.
00:00:18.800
We're going to introduce ourselves, and then we'll get into the presentation.
00:00:25.760
If you want to grab all of the slides and code that we're going to walk through right now, you can click the link.
00:00:31.400
I understand it’s difficult to click from your chair, but give it your best shot.
00:00:37.000
Alright, let's continue. Why don't you introduce yourself while I figure out why my slides aren't working?
00:00:43.239
I'm Nell Shamrell-Harrington, a software development engineer with Chef. If you've seen me speak before, you may know about my deep love for regular expressions.
00:00:49.000
I won't talk about that today, but if you're interested, I have plenty of work on it, and I'm always happy to discuss it.
00:00:54.879
The main thing I work on in Chef is the Supermarket product. It’s a Ruby on Rails application where we store all community cookbooks.
00:01:03.280
People can upload cookbooks, and others can download and install them directly from Supermarket.
00:01:08.960
You can check it out at supermarket.chef.io.
00:01:14.000
Finally, I'm a naginata practitioner, which is a Japanese martial art. If you're interested in learning more about that, just let me know.
00:01:19.479
Now, over to you, Nathan. Awesome, thanks Nell.
00:01:24.560
Hi everyone! My name is Nathan Harvey, and I'm a community director at Chef.
00:01:29.640
This means that if I tell you how to run your infrastructure, you should probably not believe me.
00:01:36.600
I don’t manage infrastructure; I manage a community and work with people.
00:01:42.159
However, I used to manage infrastructure, so I had some credibility in the past.
00:01:49.640
It's cool, though. I also co-host the Food Fight Show, a podcast about DevOps and Chef.
00:01:56.280
I'd love for you to subscribe and listen to that. You might be disappointed, but let me know if you do.
00:02:01.960
I occasionally farm and love eggs, though if you click on either of those links, you'd understand why I probably no longer love eggs.
00:02:08.599
You can follow me on Twitter or email me if you want to get in touch.
00:02:14.319
Now, let's get started! I'd like to know a little bit about you.
00:02:19.560
If you are a system administrator, could you raise your hand? Or if you have been one in the past, please do the same.
00:02:26.319
Okay, cool! How many of you are developers?
00:02:32.519
Awesome! And even some of those system administrators raised their hands, which is a crazy thought.
00:02:39.680
How many of you are into DevOps? That's good stuff!
00:02:45.239
And business people, how many of you are business people?
00:02:51.959
You're all business people. If you write software, you are a business person. If you are a system administrator, you are a business person because you work at an organization that cares about customers who use your stuff.
00:03:07.080
You have a vested interest in that. How many of you are experienced with infrastructure as code or with configuration management?
00:03:13.959
Awesome! And how many of you are experienced with Chef? You can lie; I'd rather you didn't lie, but it’s cool to do so.
00:03:28.319
When you're experienced with Chef, and you're here, what I always tell folks is that when we do introductory talks, your job is to keep your mouth shut if we lie. Don't let anyone else in on the secret that what we’re doing is telling them the hard way how to learn.
00:03:40.480
I want to start with a story. I grew up in the '80s, and a little while ago I was watching a show on National Geographic called 'The '80s: The Decade That Made Us.' This show defined DevOps in a really clear way for me.
00:03:53.319
Many developers come up to me and say they hate DevOps. The problem is that they just don't understand what DevOps is, or rather, we don't have a common understanding of what the definition is.
00:04:05.400
This amazing show on National Geographic explained DevOps to me, so I want to share that.
00:04:11.120
To do so, I have to take you back in your mind to the '80s. Back then, there were bands on the radio, and you couldn’t listen to them on the internet yet. You know, Al Gore hadn't invented it, actually.
00:04:28.280
There were rock bands and rap bands. The two would never collaborate with one another - ever! One of my favorite bands was Run-D.M.C. and they had a crazy idea.
00:04:38.840
What if Run-D.M.C. made a rock song? It’s important to note that they were a rap band, and as I said, they would never think about making a rock song.
00:05:00.919
Somebody said, "You should totally do this!" So they listened to the rock song, and they said, "Oh hell no! This is hillbilly gibberish!" They did not understand what was happening.
00:05:14.440
As an assistant administrator, I related to that. When developers hand me code, I say, "I don’t understand this code!" How am I supposed to run this in production?
00:05:26.960
However, they talked to the rock band Aerosmith, whose song they were going to cover. Steven Tyler said, "What the hell are they doing to my song?" That sounds like a developer coming up to me and saying, "Hey, it works fine on my machine! What’s your problem?"
00:05:42.760
But this amazing thing happened. The producer Rick Rubin put them both in the same studio, and in his words, it was crazy good. The video was a perfect description of DevOps.
00:06:06.440
In the music video, you start off with the rockers on one side and the rappers on the other side, with a literal wall separating them. During the video, they tear down that wall and start to work together.
00:06:29.020
The thing about DevOps is that they didn't ask Run-D.M.C. to become rockers, nor did they ask Aerosmith to start being rappers. They said, "Keep what you're good at; just work together to make the audience happy."
00:06:41.639
I want to propose a definition of DevOps, so whenever you hear someone saying 'DevOps,' think of it as a cultural and professional movement that focuses on how we build and operate high-velocity organizations born from the experiences of its practitioners.
00:06:58.199
This is how I define DevOps, and when someone says they hate DevOps, I respond that we do not have a common understanding.
00:07:12.960
It doesn't matter, because I’m on the stage. Now, you all agree this is our common definition of DevOps. Are we good? Alright, let’s talk about infrastructure as code.
00:07:29.639
I had to rant a bit about DevOps; I thought that was a fun journey.
00:07:37.639
So when we treat our infrastructure as code, we start automating the infrastructure that runs our applications.
00:07:49.840
Applications that work on your laptop are not of much value to anyone other than you. We want to take our applications and put them into real infrastructure.
00:08:01.440
This could be on the cloud or in your data center, and by doing so, we want to automate the entire process.
00:08:07.199
When do I need to automate? Isn’t git push Heroku a form of DevOps? Sure, of course you are, if you’re git pushing to Heroku.
00:08:18.960
But at some point, you may want to put your applications on some other infrastructure. You will know this because you will have reached some scale.
00:08:30.120
And that scale doesn't necessarily mean the number of machines; it can correlate with the number of people who build your infrastructure, the complexity of those applications, and so forth.
00:08:54.240
Therefore, you need an automation platform that can handle your infrastructure needs and help you treat it as code.
00:09:06.399
I’m not going to review all the bullet points on this slide; the slides are available for you to read for yourself.
00:09:19.760
The platform you choose should be one that works well for you and your team.
00:09:25.959
When it comes to infrastructure as code, it means programmatically provisioning and configuring components within your infrastructure, whether it’s hardware, cloud instances, or whatever.
00:09:32.200
Treating your infrastructure code like any other code base is essential. This means using source control and testing your infrastructure code.
00:09:49.040
So, what does it mean to treat your infrastructure like any other code base? The audience can participate here.
00:10:00.399
First, you need source control. You put your infrastructure code into a version control system.
00:10:10.399
What else does it mean? Writing tests for your infrastructure code. We should write tests.
00:10:19.160
Your infrastructure code is a continual experiment; it’s never done, and changes must be tested just as application code is.
00:10:29.880
If you treat your infrastructure as code, you can reconstruct your entire business with three things: your code repository, a backup of your data, and compute resources.
00:10:41.480
To share a personal experience, when I first became a system administrator, I was terrible.
00:10:53.320
At that point, I realized my next change to my production environment was going to start with a commit to a Git repository.
00:11:01.479
That day, I felt free; no longer would I log into production and bang on the keyboard, hoping for the best.
00:11:12.200
With Chef and with many infrastructure-as-code platforms, you're working with a policy-based system.
00:11:24.320
In your code, you capture the policy for your infrastructure. What are the requirements of your infrastructure?
00:11:36.839
Of course, we’re mostly talking about Chef today, but each of these systems will have an application to ensure the policy is continually applied.
00:11:49.360
As your needs change, you can change your policy, and your infrastructure will be updated. This is great unless you write the wrong one.
00:12:04.440
For instance, if you change your policy and incorrectly apply it, you could bring down your entire infrastructure.
00:12:19.600
This is why we test. So let's take a very simple infrastructure: we have some chefs, some Rails, and a little database.
00:12:29.400
Let's say a security firm comes in and conducts an audit. They might say, "Your infrastructure is not secure because SSH is listening on port 22."
00:12:48.560
To secure it, they will recommend changing the SSH port.
00:12:53.880
The thing is, manually editing configurations can be a headache. A platform can solve that for you!
00:13:05.000
Chef is one such platform. What is Chef? It is an open-source framework that helps you manage your infrastructure as code using a policy-based system.
00:13:21.480
With Chef, you can capture the desired state of your infrastructure, influencing how your nodes behave. It allows them to be consistent with the policy you defined.
00:13:46.760
The Chef server stores the policies and states of your infrastructure. When a Chef client runs, it asks the Chef server, "What is my policy?"
00:14:04.840
Each machine in your infrastructure follows a subset of those policies we refer to as the Run List.
00:14:17.720
For instance, if a node needs to operate as a web server, the Chef server knows it must run NTP to synchronize its clock.
00:14:35.680
This makes sure you are always aligned with the configurations set in the Chef server. The node itself will take steps to ensure compliance.
00:14:53.920
So that’s the first half of storing policies and states within your Chef server. It also has a searchable index of data about every node.
00:15:09.960
This is extremely helpful, for example, when you are doing some load balancing. Let’s take HProxy as an example.
00:15:27.320
When configuring HProxy, you need to determine who can send requests and to whom those requests will be sent.
00:15:35.160
The problem is that your infrastructure is dynamic, and you're adding and removing servers all the time.
00:15:49.840
With Chef, because it has a view of your entire infrastructure, HProxy can ask the Chef server, Who are the web servers?
00:15:56.720
The Chef server consults its search index and returns a list of the web servers. HProxy can then rewrite its configuration file accordingly.
00:16:05.120
And the next time it runs the Chef client, it can ask again, ensuring the latest changes are reflected.
00:16:21.599
What questions do you have before I move on? Is this all clear? How many of you knew everything I said when you walked in the room?
00:16:33.360
If you learned something today, that would be awesome. Let’s discuss building out your policy next.
00:16:50.560
How do we actually take Chef and use it? At its very core, we have a concept called a resource.
00:17:03.080
Resources are pieces of the system in their desired state. For example, it may be a package that should be installed.
00:17:12.000
It could be a service that should be running or a cron job on your system that you need to manage. Each one of these different things is a resource.
00:17:35.240
We’ll give you a workstation to play around with and write some Chef code here.
00:17:46.960
In our first lab, the problem is that our workstation doesn’t have a text editor installed on it. So we’re going to solve that.
00:18:02.160
You will need a card to do the labs. Go to the URL provided to view cards on the back.
00:18:10.960
If you don’t have a card and want to participate in the labs, raise your hand, and we will get you a card.
00:18:25.160
You’ll need an internet connection, which RailsConf has been great about, and something with which you can SSH.
00:18:36.160
If you’re on Mac or Linux, you can use Terminal. If you’re on Windows, make sure you have PuTTY or any SSH client installed.
00:18:49.040
Find the color of your card and match it up with an IP address. Then you can SSH into the box.
00:19:03.360
The username will be Chef, and the password is chef.io.
00:19:18.000
I mention this only now because I didn’t want to put it on the slides, so everyone would know the password.
00:19:34.080
I will also do the lab with you, so you can see how it’s done.
00:19:50.400
Let’s show the slides where we’ll describe how we’re going to install a text editor, which seems to be the clear winner.
00:20:07.960
The goal is that we will install Vim, but we won’t use the standard package manager because that wouldn’t be very Chef-like.
00:20:24.559
Instead, we will use Chef to do this. First, you'll type `sudo chef apply` followed by `-e`.
00:20:42.720
You’ll type in the package details, which should resemble something like: `package 'vim'`.
00:21:03.960
Once you’ve done that, hit the magic button. Be sure to replace 'vim' with your preferred text editor if necessary.
00:21:19.840
After that run, it should take a minute, and then you can verify by running `which vim`.
00:21:39.740
That command should ensure Vim is now installed on your system, and you will be able to use it.
00:22:02.280
Again, I stress that you need a proper understanding of how the Chef works and how it automates the management of resources.
00:22:11.240
Focusing on our test-driven approach, we want to ensure our code works reliably.
00:22:24.000
We will explore both unit and integration tests to validate our Chef code.
00:22:37.640
The ultimate purpose of tests is to ensure the safety of our code before deployment.
00:22:50.360
The four main questions we need to ask before deploying are whether the Chef client completed successfully, if the recipe set the node in the desired state, if the resources are properly defined, and if the code follows our style guide.
00:23:02.440
Let’s jump into answering these questions using a simple scenario to explore the facets of test-driven infrastructure code.
00:23:10.520
But first, let's take a ten-minute break. We'll resume shortly with an exercise where you will write code for test-driving Chef.
