Talks
Fog or: How I Learned to Stop Worrying and Love the Cloud
Summarized using AI
Fog or: How I Learned to Stop Worrying and Love the Cloud
by Wesley BearyIn the video titled Fog or: How I Learned to Stop Worrying and Love the Cloud, presented by Wesley Beary at the LoneStarRuby Conf 2011, the speaker discusses the complexities and anxieties that arise when working with cloud computing. Beary shares his personal journey of creating a Ruby library named Fog, designed to simplify the interaction with various cloud service providers by offering a standardized abstraction layer. The presentation realigns cloud computing's promise with practical tools to enhance usability.
Key Points Discussed:
- Definition of Cloud Services: Beary clarifies that he is specifically referencing cloud computing, DNS, and storage within the context of Fog, distinguishing these from the broad and often vague representations in marketing.
- Advantages of Cloud Computing: He highlights the flexibility, cost-effectiveness, and ease of resource management that cloud computing provides compared to traditional server management, promoting a shift towards on-demand resource usage.
- Challenges in Cloud Services: The myriad of available providers and their increasingly complex APIs can overwhelm users, complicating choices and stifling agility.
- Introduction of Fog: To address these challenges, he developed Fog, which allows for greater portability and ease of switching between providers with similar code utilizing different services. This framework substantially reduces the learning curve and subsequent costs of changing providers.
- Mocking Infrastructure: An integral feature of Fog is its mocking capability, which allows developers to test their cloud interactions without incurring costs or waiting for infrastructure to be set up.
- Real-world Usage: Beary cites companies like CarrierWave and Chef as users of Fog, indicating its practicality and relevancy in real-world applications. He elaborates on an example of creating an uptime monitoring service to illustrate how Fog simplifies server operations across different platforms.
- Community and Contribution: The library encourages community involvement through contributions and fosters discussions around open-source alternatives and private cloud setups.
In conclusion, Wesley Beary emphasizes the empowering nature of Fog, allowing developers to engage with cloud services without fear of the underlying complexities. His mission is not only to promote the use of Fog but to inspire more seamless cloud computing experiences for developers at all levels.
00:00:00.539
(laughs)
00:00:20.640
So I'm going to talk about Fog or how I learned to stop worrying and love the cloud.
00:00:25.740
Just to get started, I'm Wesley Beary. I go by Genus online; it's my GitHub and my Twitter handle.
00:00:31.980
I work at Engine Yard, where they're kind enough to basically sponsor the Fog project. I get to work full-time on it, attend conferences, and evangelize it, which is a pretty gnarly job.
00:00:39.360
So, I’m very grateful to them for that. First off, I wanted to give a baseline.
00:00:44.700
What are we talking about when we say 'cloud'? There's a lot of buzz in this space that honestly doesn't make much sense to me.
00:00:50.160
The whole "Microsoft to the cloud" thing annoys me because it feels like it's used as a catch-all. So, just to be clear when I'm talking about 'cloud' within the context of Fog, I want to narrow it down a bit. The core parts I’m referring to include cloud computing, cloud DNS, and cloud storage—services that you will use to do something interesting.
00:01:18.479
There are also a few peripheral services emerging, such as key-value stores, load balancers, and queues, which I think will continue to evolve.
00:01:30.299
So, what is the difference between this and the ads from Microsoft? The first thing is that it's on-demand. There are resources that you can use only as much as you need, which is incredibly powerful compared to the historical model in which we would request new servers and wait months for delivery. Then, we’d pay for those servers indefinitely, even as they degraded rapidly.
00:01:54.720
With cloud computing, you pay for exactly what you need when you need it, which makes it way more flexible. You can add and remove resources as needed. It's also repeatable, which is beneficial because, historically, you wouldn’t get the chance to add and remove servers for testing to see how it would all work. But in the cloud context, it’s much simpler to say, 'Let’s spin up another node, add it to the cluster, and see what happens.' By building systems with this idea in mind, we can create much more resilient setups.
00:02:30.299
We’ll know that adding and removing nodes won't bring everything down because we’ve practiced it many times before we need to do it for real. Now, that sounds pretty awesome, right? So why was I worried? For starters, there’s a massive overload of options.
00:03:08.760
Microsoft has certainly muddled things up with their ads about all of this. There are a bunch of different providers offering what appear to be very similar services. So, how do you choose? Do I go with Rackspace, EC2, or one of the dozens of other providers popping up every week? It’s complicated.
00:03:42.299
As you investigate, you realize you need specialized expertise for each provider. Using Rackspace Cloud requires a certain set of knowledge, as does EC2. Once you choose one of these paths, backtracking to another can be costly and scary.
00:04:12.420
There are tools out there that would ideally help with this, but when I started this project, I found that many of them had very different APIs from one another, which in many cases were also significantly different from the actual provider's API. Plus, some of these tools were poorly maintained. Working with APIs was hard enough without adding another layer of complexity.
00:04:59.580
There was a lot of hope pinned on the emerging cloud standards, but they were moving slowly, and we really needed these solutions right away. We didn’t want to wait for two years for cloud standards to stabilize, so in some ways, this project aims to be a grassroots effort to address the urgent need for some standardization.
00:05:30.960
The end result of all of this was a library that I called Fog. It’s a Ruby library that abstracts many cloud services and makes them much easier to work with.
00:05:58.440
There's more information available, and I will provide links at the end. So why did I create this? It's important because it offers portability. You can use similar code across various providers, which lowers both the barrier to expertise and the barrier to entry.
00:06:27.100
You don’t need to figure out all these details upfront, and if you make the wrong choice initially, the cost to switch providers becomes far less daunting.
00:06:56.760
It’s also powerful, with numerous abstractions at different levels, allowing you to accomplish many tasks that you couldn't otherwise do just out of the box. The library has been gaining traction; it’s getting close to a couple hundred thousand downloads, with around 1400 followers, 225 forks, and now about 110 contributors.
00:07:28.320
We've been adding new services regularly, and another cool feature is the built-in mocking infrastructure. Not every service has this yet, as it requires significant additional work, but, for example, if you're using EC2, you can enable mocking to emulate what EC2 would do.
00:07:50.160
You may not be able to log into this fake in-memory server, but you can practice infrastructure activities such as spinning up servers or attaching IPs without incurring actual costs and waiting for execution time.
00:08:02.340
So who is using this library and how? Some tools and companies using Fog include CarrierWave, Chef, and Gemcutter. While technically not at version 1.0 yet, it’s in production and running well.
00:08:24.740
I could probably push it to 1.0 tomorrow, and no one would bat an eye, but I just want to clean up a few minor issues first. Now, let's have a quick interactive session.
00:08:50.400
By a show of hands, how many of you are doing some kind of cloud work right now (not associated with Microsoft)? Great! And how many are already using Fog?
00:09:03.360
A couple—this is about what I've seen in the past. So hopefully, I can convert a few more of you today. Usually, I get responses from those not using cloud services saying, "That's great, but I don’t have a use case for it.” So let’s create an exciting use case.
00:09:17.800
We’re going to talk about building our own uptime service—a concrete example that shows why you might want to use multiple services simultaneously.
00:09:39.060
First, you will need to perform some setup and install the gem. The initial step is to connect to a service where we can spin up servers for our uptime service. This is what that will look like.
00:09:52.440
You need to connect to a provider, in this case, Rackspace, providing your credentials, and then pull up the server. However, as you can see, even spinning up a server isn't that straightforward.
00:10:16.560
It can be daunting, with numerous required arguments and syntax that can be hard to remember. It takes a lot of commands just to get things going, and you quickly realize that this setup is complex enough that unfortunately, it's only suitable for Rackspace. Without significant effort, it won’t work elsewhere.
00:10:42.000
Deciding these parameters and configurations is tiring. So, I went back to the drawing board and simplified things.
00:11:00.180
Now, you just pass in the attributes you want for the server and specify that it should bootstrap. It expects that you’ll end up with a server that runs has your SSH key set up and does all the necessary configurations.
00:11:24.720
For example, you state the image ID, along with the relevant private and public keys. This now gives you an easier route. Sure, the specifics of '49' as an image ID don’t tell you what image you’re dealing with, but at least you’ll know you're in the right neighborhood.
00:11:45.360
Regarding servers, you can call compute.servers to get a list back, and a call to servers.get with the relevant ID will return the specific server. Effectively, this aims to emulate something like Data Mapper or Active Record for cloud resources.
00:12:05.520
It's all about making it familiar to you, so it feels easy to go in and do what you need to do. You also have the option to call reload to refresh the state, create a local model without saving it yet, or call create, which is a combination of New and Save in one command.
00:12:21.420
Returning to our uptime example: we want to ping a remote service for our customer. We’ll initiate a ping—for instance, ping -c10 with our target, which might be food.com.
00:12:47.280
You’ll get the standard output and parse it to extract the minimum, average, maximum, and standard deviation from the ping results.
00:13:05.520
You've now spun up a server, initiated a ping, and parsed the data, with the most complex task being parsing the ping string itself. This is a result of the useful abstractions.
00:13:20.400
Cleaning up after your test is simple; just destroy the server afterward. You now have your data as a hash that you can store in your chosen database solution, and you're good to go with your uptime server.
00:13:45.420
Next, I want to show the difference between the long original version of booting an EC2 server compared to a Rackspace server. Essentially, the process for both is often very different.
00:14:22.520
By establishing a bootstrap process like we described earlier, you can take the same code and just pass a different set of credentials along with different attributes for your EC2 server.
00:14:39.300
This means that we can ping from multiple locations, allowing us to tell a client the response times from both the Amazon and Rackspace data centers.
00:15:08.400
You can also add further regions or providers easily; the process is very similar each time, making it straightforward to create a diverse monitoring setup.
00:15:30.540
Easily finding the relevant IDs, image IDs, or other parameters is a little easier with Fog’s interactive console. Simply typing 'fog' will indicate credential issues, giving you guidance on setting up your credential file similar to Rails' database.yaml.
00:15:46.320
With that in place, you can see which providers are specified and which services are available. You can also call compute with your provider to get direct access and see all the individual API calls available.
00:16:10.920
Typically, the models provided through Fog are sufficient for what you need. However, occasionally, you may want to dive down to those low-level API calls if you find that you're dealing with a unique case that requires it.
00:16:41.760
Just to clarify, the providers you'll work with include Amazon and Rackspace. Each offers various services like compute, DNS, and storage. The collection represents the actual objects manipulatable, while models make up a collection.
00:17:01.680
Requests reflect the lower-level API calls that models are built on and to which you can delve in case specific needs arise. For instance, a call to list servers on Rackspace provides an x-con response, which includes the response body, headers, and status.
00:17:20.760
The body is pre-parsed into Ruby format for your convenience, but not nearly as user-friendly as the model. I highly recommend using the abstractions available where you can.
00:17:39.600
For a sanity check, you can also select servers that are ready to run instead of digging into each provider's status names, which can differ across platforms.
00:17:55.560
Finding images is user-friendly too; you can display them in a tabular format, which is generally more readable than a standard inspected array from Ruby. You’ll save time and headache.
00:18:14.760
Exploring this is exciting but can be time-consuming, especially if you inadvertently leave servers running overnight.
00:18:30.000
So, I want to introduce you to the mocking infrastructure. You can enable it by setting the environment variable fog_mock=true or in scripts through fog.mock!
00:19:00.600
This creates a simulation based on an in-memory representation of cloud status, allowing most functions to work without issues. If there are unimplemented features, it'll raise an error to clarify.
00:19:34.800
Most of my tests are run using the mocked version, with the addition of main integration tests prior to deploying.
00:19:56.040
Now, let’s talk about utilizing cloud storage to aggregate data. You’ll use fog.storage.new to create a connection. This time, we’ll work with Amazon for S3, creating a public directory for ease of access.
00:20:25.920
As another example, once we switch credentials to Rackspace, we gain another backup storage location to ensure data availability even in the unlikely event of an S3 crash.
00:20:45.120
Cleanup is a straightforward process; you’ll iterate through your directory and delete any files no longer needed.
00:21:10.560
Now, let’s shift to profit. Why not turn this data into a freemium model?
00:21:31.200
For example, if foo.com pays for uptime monitoring, you’d set up something like foo.uptime.com to manage their service.
00:21:50.760
The setup is familiar; you create a connection to DNS using Zorigo and then establish a Zone and create an A record pointing to your customer’s domain.
00:22:15.600
And again, once you're done, cleanup is simple. Congratulations, you now have the beginnings of a startup!
00:22:39.300
This is Fog for you, folks! Quick recap: Fog simplifies painful cloud-related tasks to make your life easier.
00:23:00.960
It encodes the complexity and pain points into an easy-to-use Ruby library, allowing you to avoid worrying about the messy code.
00:23:23.160
It’s empowering. I started with little cloud knowledge and now I don't hesitate to spin up multiple servers to explore cool ideas without any financial burden.
00:23:47.400
Cutting-edge developments are happening in cloud computing, and I hope Fog provides an easy avenue for you to jump right in.
00:24:09.900
Your homework? Follow Fog releases to track new features and updates. If you’re interested in contributing, I'm also giving away stickers!
00:24:29.880
You can report issues on GitHub or write blog posts about your Fog usage, and you'll receive a t-shirt!
00:24:50.760
If you're interested in becoming a collaborator, I’d love to welcome you, and you could end up with a black t-shirt!
00:25:10.860
Now, I invite questions and comments. I know it was a whirlwind tour, but I wanted to show you what’s available and what's possible.
00:25:34.020
If you’re interested in open-source alternatives, let’s discuss. Also, I covered many Cloud offerings—any thoughts on that?
00:26:01.440
In terms of private cloud, while not explicitly supported, many private services offer EC2-compatible APIs.
00:26:36.840
This means you can use Fog, pointing at a different endpoint. Users have succeeded with setups like Eucalyptus and Rackspace's compatibility with OpenStack.
00:27:05.160
I've explored some service compatibility, though I struggle to keep up on that.
00:27:35.520
In collaborating with familiar tools like Vagrant, there is indeed a VirtualBox driver that uses the same gem as Vagrant for ease of usage.
00:28:06.060
I wanted more control than Vagrant level abstraction permitted, which led to a lower-level solution accommodating necessary flexibility.
00:28:38.760
Documentation is an area I'm continuously improving, and I'm adding further clarity for users regarding providers and their services.
00:29:08.640
We're working on a table structure as viewing aids for users to compare which providers are best suited for their needs.
00:29:37.920
Overall, make the best! Cloud computing allows countless opportunities for interacting with providers.
00:30:06.840
And if you'd like any specific recommendations, I can help guide you some based on local data centers or paper long-term value.
00:30:32.520
If there's nothing else to cover, feel free to grab some stickers and enjoy the conference. Thank you for having me!
Explore all talks recorded at LoneStarRuby Conf 2011
+15