Michael Fairchild
Poolparty - Jump In The Pool, Get In The Clouds
Poolparty - Jump In The Pool, Get In The Clouds
by Ari Lerner and Michael FairchildThe video titled 'Poolparty - Jump In The Pool, Get In The Clouds' is a technical presentation by Ari Lerner and Michael Fairchild during the LA RubyConf 2009. The primary focus of the talk is on Poolparty, an open-source software project designed for engineers and developers who are working with interactive software and cloud resources. The presenters discuss various aspects of Poolparty, including its decentralized architecture, configuration management using Chef, and specific functionalities that enhance the deployment and scaling of applications in cloud environments.
Key Points Discussed:
- Introduction to Poolparty: A software designed to run in cloud environments, which facilitates as a tool to manage distributed resources efficiently.
- Decentralized Server Architecture: Every instance acts like a master, and they communicate to ensure the system is homogenous and can easily update configurations seamlessly.
- Configuration Management: Emphasizes the use of Chef for managing server configurations through recipes, allowing for rich customization and efficient updates.
- Cloud Configuration: The presenters explain how to set up a cloud environment and provide commands for users to easily manage their clouds using a simple 'cloud.rb' file.
- Scaling and Metrics: Discuss the importance of managing resources effectively, especially during peak loads, and how to scale up or down based on user demand, utilizing multiple clouds for different applications, such as database management.
- Future Directions: Mention plans to expand Poolparty’s integration capabilities including support for VMware and enhanced feature sets that allow easier deployments.
- Community Engagement: Highlight the role of community in developing and improving Poolparty, fostering collaboration especially with tools like Chef.
Throughout the talk, the speakers also touch upon practical examples such as deploying an application using various cloud infrastructures and the significance of load balancing through DNS improvements. They conclude by encouraging audience participation and sharing knowledge through their community channels.
Main Takeaways:
- Poolparty is a robust tool designed for efficient management of cloud resources, promoting decentralization and flexibility in deployment.
- Utilizing configuration management tools such as Chef can vastly improve the deployment process and provide significant customization options.
- Collaboration with the community is essential for the continued development and enhancement of Poolparty functionalities.
- The future of Poolparty will include expanded capabilities and increased ease of use across different cloud services.
00:00:12.160
All right, this is a talk about engineers and interactive software, particularly a piece of software known as Poolparty.
00:00:29.199
We are working on this project with some open-source software called Poolparty.
00:00:42.320
That's a great idea! Oh yeah, I think it's on your site. Unless you message your friends, there's no way.
00:00:49.520
Yeah, and they can follow you. No way. Yeah, I don't know.
00:01:16.000
Installing software is going to be popular, so we're going to need to scale them up. Hold on, because...
00:01:22.240
On the day it opens, there are going to be like four billion hits!
00:01:33.200
We want to change the messages to include videos instead of just text.
00:01:53.680
That’s a drag. I mean, the rest of it is kind of fun, it's just a lot of configuring.
00:01:59.520
So it would be nice if we could have just one file here to define our infrastructure.
00:02:06.079
We can check it into version control and run tests against it. That would be awesome!
00:02:11.840
Yeah, that’s something beautiful to do every year.
00:02:34.000
We should probably instantiate some servers to limit expenses.
00:02:39.680
We don’t want to pay for too many in case there are four billion, but we also don’t want them to be zero.
00:03:00.239
I think I've heard about that. Okay, let’s do it.
00:03:06.879
Oh, you know what? We should keep it all in one place, absolutely.
00:03:16.080
Let’s tell that.
00:03:30.959
We can put a file on our cloud. And by 'cloud', we mean a homogeneous set of resources.
00:03:37.280
It doesn't matter how many instances you have; our view is that a cloud is just a collection of resources that you can use.
00:03:42.560
The work unit in our case is an EC2 instance.
00:03:49.519
The important thing when designing is to think of a cloud as a set of services. It’s got HTTP, of course - 80 or 480, and maybe XMPP, you know, whatever we want.
00:04:00.560
But we don’t want to be designing our servers specifically on Ubuntu or Yellow.
00:04:22.880
So, let’s go!
00:04:29.120
So, let's write this application, configure our servers, scale them up, and let everyone in this room hit it quickly.
00:04:36.000
Then, let it scale down when nobody is hitting it.
00:04:46.800
Oh God, that is so much nicer, isn’t it?
00:04:52.240
That's right. This is our live pairing session.
00:05:10.880
So, actually, I did a cloud start already because it takes a little while to set up.
00:05:16.160
But I promise you it’s there; I can show you if you really want to see it.
00:05:22.000
However, we want to reserve some space to show you some of the cool features.
00:05:28.960
So, yes, Cloud Start was initiated, and I took a screenshot, so you can see it.
00:05:35.120
That’s not entirely true; I copied against the text.
00:05:41.199
What it does is launch our servers, picking up EC2 instances.
00:05:47.520
If you have the defaults in that EC2 directory, it will use them. You can specify your keys.
00:05:53.039
Everything's configurable, but this will instantiate instances and return food scraps.
00:05:58.400
This way, we have all the software that we need for the Poolparty to run on it.
00:06:17.440
It will also configure based on your settings, which is another nice thing about the bootstrap process.
00:06:22.479
That will package up any local dependencies you have on your machine.
00:06:29.120
This is great in case you change some of the gems you want to use.
00:06:34.400
Now that our Cloud Start has been initiated, let’s actually go look at it.
00:06:48.000
If you don’t follow the Poolparty conventions, you can just pass your clouds.rb file to every command line.
00:06:54.880
Yes, okay, but we are using the Ruby convention or the Poolparty convention.
00:07:00.960
Keep your clouds.rb file in a subdirectory of the current working directory.
00:07:07.360
You can also set it as an environment variable so you never have to remember where it’s at.
00:07:14.080
The other default location is where Poolparty expects you to put your cloud setup.
00:07:19.599
So, we can run a command to list the clouds.
00:07:31.840
That’s right! We can also associate all of it together.
00:08:13.360
Now, we're going to dive deeper into Poolparty.
00:08:19.120
Poolparty is pretty much completely decentralized. We don’t have a master server.
00:08:24.479
Every instance or node is treated equally, which is part of our principle that clouds are homogenous.
00:08:30.400
You can connect to any of them, update your configuration, and they will broadcast updates to the rest.
00:08:42.640
In our setup, we have a file that serves as a message of the day for anyone logged into a server.
00:08:49.399
This cloud will only show the Chef cookbook, and we'll get to that in a minute.
00:08:56.160
Then we'll update the message of the day on any of the servers when we log in.
00:09:03.680
Using the default AMI, we are using the Ubuntu Elastic 8.1, an Amazon Machine Image.
00:09:09.519
I haven't even started the template, sorry guys!
00:09:16.160
Poolparty has some basic resources, including a patch file.
00:09:23.040
It also has a directory where you can run command executions on any of the servers.
00:09:30.160
You should know that some of these problems have already been addressed by others.
00:09:37.760
A great example is Google’s use of desktop computers.
00:09:44.720
But another tool I am excited about is Chef.
00:09:50.960
Chef is a configuration management engine where you write recipes to define how your server should look.
00:09:58.560
Chef actually parallels Poolparty closely.
00:10:05.360
This is a basic Chef recipe. Chef is outside the scope of this talk, but I want to show how you include it in Poolparty.
00:10:12.000
Just create a Chef block and specify what recipe you want.
00:10:19.680
Now that our cloud is set up, let’s go through the configuration quickly.
00:10:26.240
You can simply type 'cloud configure' as the command to update.
00:10:32.160
What’s neat is that if you define a deployment in your clouds.rb file, each time "cloud configure" is used, it will update resources.
00:10:39.840
This way, you will not forget about dependencies or miss installations.
00:10:45.440
Poolparty doesn’t require you to use a given repository; you can still use Flat or Capistrano.
00:10:50.400
We provide a basic deployment bootstrapper as well as Capistrano deployment options.
00:10:57.600
However, we felt Capistrano was a bit more than we needed for our purposes.
00:11:05.000
The way configure will look can be switched back and forth quickly.
00:11:08.240
Cloudwise means unchanged by itself.
00:11:14.399
In simpler words, run it a bunch, and it won’t change anything unless it really needs to.
00:11:20.960
This is a big reason we switched over to using Chef, allowing for support of many cookbooks.
00:11:28.320
We’ll be diving into the Opscode repository shortly.
00:11:35.600
They have a cornucopia of recipes available, with much more being developed.
00:11:42.400
I just heard Twitter Club was mentioned on Oprah! We better expand fast.
00:11:51.120
How do we do that? With twelve actors, that’s how!
00:11:57.680
When you load greater than 0.8, it would be a nice baseline to consider.
00:12:05.440
The next slide clarifies that these metrics are not just per instance, but averages across the entire block.
00:12:11.760
The challenge we face in cloud resource usage is variability based on specific usages.
00:12:18.560
Resource consumption can differ significantly depending on the load.
00:12:24.960
For example, a Rails site may function smoothly, but the database might be overworked.
00:12:30.560
But how do we solve that problem? With multiple clouds!
00:12:36.080
We’ll place our database server in one cloud.
00:12:41.120
Based on availability and distribution, that will work effectively.
00:12:46.960
As we mentioned earlier, Poolparty is decentralized.
00:12:53.360
Any instance can act as a sort of master, and they function equally.
00:13:02.480
Poolparty itself does not create applications; it's just a tool to facilitate that.
00:13:10.080
Poolparty is built in Ruby because Ruby is beautiful.
00:13:17.120
But if you want some sort of particle simulation, Poolparty can accommodate that as well.
00:13:23.919
It’s flexible for various applications, just like a video sharing site.
00:13:30.639
You can manage a web front end while using different clouds for advanced processing.
00:13:38.880
Maybe you'll need a separate cloud for sending emails or updates.
00:13:45.360
Looking ahead, we aim for more remote resources.
00:13:52.080
Currently, we support EC2, but we are also planning to add VMware functionality.
00:13:58.400
In addition, if you have VMware on a Mac, you'll be able to instantiate servers using command line.
00:14:05.280
The goal is to make it cheaper and easier for instantiating servers.
00:14:12.480
Many people have services they’re using, which means they’re racking up credit.
00:14:18.560
So, you can easily implement your instances using port parties.
00:14:23.840
Just four methods are needed to add additional instances, requiring a valid ID.
00:14:31.520
You must also return a hash providing a way to access it.
00:14:37.440
In terms of VM introductions, VM-based and EC2 services will handle them slightly differently.
00:14:44.880
The scripted automated deployment can also adapt to these changes.
00:14:52.359
We are currently also working on performance testing, and it’s in the works.
00:14:59.760
This aids deployment to VMware by allowing for configuration changes.
00:15:05.120
We want to ensure those ports open properly, making sure everything runs smoothly.
00:15:11.520
Additionally, Poolparty is undergoing extensive modularization.
00:15:18.080
You’ll notice several gems available on GitHub, with many more coming soon.
00:15:24.120
This modularization will allow you to use functionalities outside of Poolparty itself.
00:15:30.720
This flexibility allows Poolparty updates without affecting the gem functionality.
00:15:38.080
The latest addition from earlier today looks to harness HTTP for solid intercommunication.
00:15:44.720
Butterfly is a great example; you can send data over HTTP efficiently.
00:15:50.960
We store the state and update the necessary load information with the cloud.
00:15:57.760
Utilizing a cloud.json file, we determine the current state and refresh the information.
00:16:03.840
As you access endpoints, you’ll acquire relevant data in real time.
00:16:09.480
The goal is for nodes to update their own states and share information across the cloud.
00:16:15.440
This method eliminates redundant data pulling, making it very efficient!
00:16:21.760
Tighter integration with DNS is the next objective.
00:16:28.720
Instead of relying entirely on Hypoxia, we’ll leverage DNS for load balancing.
00:16:35.280
This creates a smarter way to handle traffic across your cloud.
00:16:41.360
DNS is a cornerstone in network communication, vital for long-term success.
00:16:48.080
When you launch a cloud, we’ll leverage these DNS capabilities automatically.
00:16:53.920
You'll enjoy effective load balancing and quick deployments.
00:17:01.440
Also, we can explore various alternatives.
00:17:06.560
Heroku is a great platform for rapid deployments.
00:17:12.560
But it might lack sufficient power for certain requirements, like transcoding.
00:17:18.320
Executing a cloud for transcoding operations can fulfill those needs.
00:17:24.800
Connecting these systems through Poolparty can enhance efficiency.
00:17:30.160
Once your development cloud is ready, you can easily switch to production settings.
00:17:36.320
When it’s time, you can cloud terminate and push to other environments.
00:17:41.760
Alternatives provide flexibility in how and where you deploy.
00:17:45.760
We’re generally very active in our IRC channel called Poolparty RB.
00:17:51.120
Additionally, Poolparty has been relaunched with much more content.
00:17:54.320
Feel free to reach us through email or our platform.
00:17:59.840
Thanks to everyone here, as well as to AT&T and HTTP Interactive for their support.
00:18:05.120
I have VMware mounted on the back, indicating progress for practice deployments.
00:18:10.560
The goal is to deploy applications using VMware with eventual moves to EC2.
00:18:15.480
Yes, you will be able to tweak files, switching from VMware to EC2.
00:18:19.760
The answer is yes; however, that feature is in the branch not officially released.
00:18:25.600
Current test cases verify compatibility across systems.
00:18:31.760
Booting with other Linux distributions is possible, but supportive configurations are preferred.
00:18:37.600
We can install Chef or Puppet to assist with those deployments.
00:18:43.200
It’s important to consider available recipes when configuring your environment.
00:18:50.000
The instances we configured will inherit properties set in the parent nodes.
00:18:55.680
If you’re unsure what that means, I encourage you to look it up.
00:19:01.200
Looking forward, we see the community wholeheartedly responding.
00:19:06.880
There are discussions surrounding remote bases for specialized deployments.
00:19:12.880
Keep an eye on our roadmap through the Poolparty Lighthouse platform.
00:19:20.320
Community involvement has yielded fruitful collaborations, especially with Chef developers.
00:19:25.760
It's been an engaging experience, and we appreciate all the support.
