Cutting Through the Fog of Cloud

Talks

Kyle Rames

1 talk

#cloud-computing

#ruby

#load-balancing

Cutting Through the Fog of Cloud

by Kyle Rames

Cutting Through the Fog of Cloud

In this presentation, Kyle Rames, a Developer Advocate at Rackspace, provides a comprehensive overview of cloud computing, its types, and practical implementation using the Fog gem with Ruby. The talk aims to demystify cloud concepts and offer insights into manipulating cloud resources effectively.

Key Points Discussed:

- Cloud Computing Overview:

- Cloud computing allows the consumption of computing resources like utilities, charging users based on usage (hourly compute power and per megabyte for storage).

- It is characterized by seemingly infinite resources and can be divided into three types:
- Software as a Service (SaaS): Familiar applications like Gmail and GitHub.

- Platform as a Service (PaaS): Provides a managed environment for deploying applications (e.g., Rails apps) but may not support certain services or libraries.

- Infrastructure as a Service (IaaS): Offers greater flexibility, allowing users to configure resources as needed, usually seen at the bottom of the cloud service pyramid.

Types of Clouds:
- Public Cloud: Offers inexpensive resources.
- Private Cloud: Involves renting or purchasing physical machines for cloud software, usually at a higher cost.
- Hybrid Cloud: Combines public and private cloud features for optimization in security and costs.
Open vs. Closed Cloud:
- Open clouds (e.g., OpenStack) allow access to source code and community collaboration, while closed clouds restrict such access.
Fog Gem Usage:
- Fog is a cloud SDK that facilitates cloud service interactions in Ruby.
- It boasts extensive usage (over 2 million downloads) and active community contributions.
- Rames discusses practical commands to provision servers, manage DNS services, and implement load balancers using Fog.
- The core concepts include collections, service interactions, and the distinction between model and request layers within Fog.
Creating and Managing Cloud Resources:
- Rames provides a step-by-step code example for provisioning servers, managing states, and handling attributes using Fog's methods.
- Emphasizes the importance of using SSH keys for authentication instead of passwords for enhanced security.
- Introduces a structured approach to creating DNS records programmatically using the request layer of Fog.

Conclusion:

Rames concludes the session by emphasizing the flexibility offered by cloud services, particularly through IaaS and the capabilities provided by the Fog gem in Ruby. He encourages the audience to explore further documentation and leverage the potential of cloud computing for their projects.

Overall, the talk effectively clarifies cloud computing concepts while presenting practical coding techniques that developers can apply in real-world scenarios.

00:00:15.360 Hey everybody, my name is Kyle Rames and I'm a Developer Advocate for Rackspace. Today, I'm going to cut through some of the fog of cloud for you.

00:00:27.680 On the agenda today, we'll have a brief five-minute "Cloud 101" to go through some basic cloud concepts, and then we'll jump into working with clouds using Ruby. You'll actually see some code in this talk.

00:00:42.239 So, what is cloud? Cloud computing involves consuming computing resources like a utility, much like the power from an electric outlet. You'll pay for compute time by the hour, and for file transfer and storage by the megabyte, similar to how you consume power. This resources feels seemingly infinite.

00:01:11.520 There are three different types of cloud computing: the first is Software as a Service (SaaS), which most people are familiar with, like Gmail and GitHub. We'll skip past this as I assume most of you know about it already, and move on to Platform as a Service (PaaS). This is where you upload your Rails or Sinatra application to a cloud provider, who manages it within a controlled environment, handling system administration and security, while helping you to develop a scalable solution.

00:02:10.000 The drawback of PaaS occurs when you want to do something outside of that environment, like running a necessary service that isn't offered by the provider or needing to use a library that your platform won't allow. This brings us to the third type of cloud: Infrastructure as a Service (IaaS), which is my favorite. IaaS provides a lot of flexibility, allowing you to combine various components like computing power, storage, and load balancers into virtually endless configurations.

00:02:41.280 Typically, the three types of cloud are represented in a pyramid diagram, with IaaS at the bottom, signifying its flexibility. As you move up the pyramid to PaaS and then SaaS, the flexibility reduces. Consequently, the time to build and deploy solutions increases — it's much easier to set up a solution like Github than to provision servers and install software.

00:03:24.480 When I mention cloud, many think of the public cloud, which offers inexpensive resources that feel nearly infinite. However, there is also a private cloud option where you purchase or rent physical machines and install cloud software, providing programmable infrastructure but potentially at a higher cost and with limited capacity. Additionally, there's an emerging third type known as hybrid cloud, which allows you to manage portions of your resources between secure and less secure environments, optimizing both security and cost.

00:04:49.680 Another aspect of cloud you might consider is whether to use a closed or open cloud. A closed cloud means you can't access the source code and any requests for features would result in a simple comment submission. In contrast, an open cloud, like OpenStack, allows you to view the source code, submit pull requests, and collaborate with the community to enhance functionalities. OpenStack is in fact celebrating its third anniversary today!

00:05:49.680 With OpenStack, you can choose to run it in a private or public environment, download a distribution to install on your own servers, or use a service like Rackspace or HP that offers open cloud solutions, enabling you to move between diverse environments seamlessly.

00:06:21.390 So, that covers a brief overview of cloud concepts. Now, let’s dive in and start manipulating clouds using Ruby. We're going to use the Fog gem, which is considered the number one cloud SDK according to rubytoolbox.com.

00:06:54.960 Fog has over 2 million downloads and boasts around 300, almost 400 contributors, making it a very active project. This project sees 30 to 60 commits weekly, and chances are, even if you're not using Fog directly, you're benefiting from it as it powers many popular gems like Paperclip and CarrierWave, as well as tools like Chef and Puppet Provisioner.

00:07:31.440 Today, I'm also excited to introduce you to RUM, a new utility from Rackspace, which Charles Lowell will discuss later in his session. Since I’m a top contributor to the Fog project, feel free to come talk to me after the presentation if you have questions regarding using Fog with Rackspace, Amazon, or HP.

00:08:11.840 To see the list of providers available in Fog, we can call `fog.providers.keys`. It will return a list of around 36 providers. Some of these are strictly DNS-based, while others like Rackspace, Amazon, HP, and OpenStack provide a full suite of services. For today, we will choose Rackspace, as I know everyone here appreciates open clouds and fanatical support.

00:08:42.559 Next, we'll look at the services offered by Rackspace. This can be done through `fog.services`, which will return a hash. The hash will contain services as keys and their corresponding providers as values. We will examine the services available under the Rackspace provider. Today, we will look at `compute_v2`, load balancers, and also DNS services.

00:09:19.760 For those observant in the audience, you might notice that Rackspace has two compute services: `compute_v2` for our next-gen servers. Unless you're operating legacy servers, you should primarily utilize the `compute_v2` service.

00:09:52.640 We'll create a sample architecture using Fog. We will provision two cloud servers, set up a load balancer, and create a DNS entry that will point to the load balancer. Let's begin with the virtual machines. We'll create a compute service using the command `fog.compute.new` and pass in a hash. The first key-value pair will specify the rackspace provider, ensuring that we get the right implementation.

00:10:22.960 Next, we will include additional key-value pairs for specific configuration parameters of that service provider. For example, we'll specify `v2` as the version to get `compute_v2`, and then provide the credentials while using the Chicago data center for our operations.

00:10:56.560 We need three pieces of information to create a server: the image, the flavor, and the server name. The image could be one of the following operating systems: Ubuntu, FreeBSD, Windows, or Arch Linux. This could even be an image you created from an existing server.

00:11:38.399 The flavor represents the server configuration, combining memory, disk space, and the number of CPUs. This is where much of the cloud server's cost originates. Be mindful that some images, such as Windows or Red Hat, require licenses.

00:12:08.160 The last piece of information is simply a name for your server. Choosing a name is notably one of the hardest tasks in computer science, second to cash invalidation and off-by-one errors. When selecting a name, it's useful to keep it disposable since you'll use the server temporarily rather than forming a deep relationship with it.

00:12:41.760 The name I’m choosing is TK421. With all this information gathered, we can proceed to create the server while providing our image, flavor, and name. The server object returned might have some attributes set to nil because the server creation will be scheduled on the Rackspace cloud but hasn't been completed yet.

00:13:25.040 To check the current status of the server, we can reload it to get the latest data using the command `tk_421.reload`. After doing so, we can observe that some attributes have been populated, showing you the current state of the server.

00:14:05.840 If we want to access specific attributes programmatically, we can use the syntax `server.attributes`, which will return a hash containing all attributes. Fog also enables some meta magic, allowing you to use `tk_421.state` to check its status. At this point, we can also check the server's progress during its creation.

00:14:59.840 Since building cloud resources involves a common pattern of status checks, Fog offers a method called `wait_for`. This method will reload the server state and check if it's ready based on certain conditions agreed upon by the provider. For our specific server, it will determine if it's in an active state, has 100% progress, and is allocated an IP address.

00:15:45.640 If the block for `wait_for` evaluates to false, it will pause for a second or two before repeating the check until the condition is met or a timeout occurs. In this instance, our server was ready in a little over three minutes, at which point we can retrieve the login credentials using `server.username` and `server.password`.

00:16:45.000 When a server is created on the Rackspace cloud, the password attribute will be populated. However, when retrieved subsequently, it will not be available. Hence, it could be helpful to either store it upon creation or change it later. Personally, I prefer using SSH key authentication for improved security and convenience.

00:17:29.440 Fog even provides a bootstrap mechanism that creates a server, waits for it to become ready, and then sets up an authorized key file while locking out password access for the root user. This approach ensures we're using safer practices in setting up our server.

00:18:18.080 Once setup is complete, Fog will generate an attributes and metadata JSON file containing server provisioning details. The syntax for bootstrap is similar to server creation but includes additional attributes for the public and private key paths.

00:18:55.840 Following the completion of the bootstrap, you can start using the server, facilitated by Fog methods like `scp` and `ssh`. For instance, I can upload a setup script using SCP to `root/setup`, then remotely execute it to set up the VM as needed.

00:19:43.840 Moreover, the Rackspace team employs this methodology for load testing the Repose project, which is our reverse proxy positioned in the cloud to handle authentication and rate limiting functionalities. We've merely begun exploring what can be achieved with cloud servers, and I urge you to delve further into the documentation.

00:20:25.760 Now let's talk about the collections within Fog. Earlier I referenced the `service.images` method, which returned a collection of images. This collection inherits from an array, meaning it allows us to utilize array-related methods, which is why we've referred to methods like `first` and `find`.

00:21:09.360 Also, there's a `Fog::Collection` containing the inner workings that aren't especially interesting. The images class, however, includes multiple methods such as `all`, which facilitates lazy loading when accessing data within the collection.

00:21:45.920 If I had known an ID for an image, say the Ubuntu one, I could have retrieved it directly with the `get` method. The class also has a `create` method, emphasizing our architectural integration with RESTful design principles. Using `create` not only creates a local copy of the VM but also the remote server.

00:22:29.040 To explore service collections, we can invoke `service.collections`, which reveals the various collection types available for our compute service. Three of these collections have been used already: servers, flavors, and images, alongside two others: attachments and networks.

00:23:05.760 Now that we've seen how the model layer operates via both compute and load balancer services, we should discuss the request layer, which provides a direct mapping from the cloud to Fog. The request layer offers the most efficient way to interact with the cloud but lacks portability across providers, which is why gems like Paperclip, CarrierWave, and others favor the model layer.

00:23:45.760 You would typically resort to using the request layer only if a specific feature was unavailable through the model abstraction. To see what's available in the request layer, you can employ `service.requests`. For instance, I can extract various requests supported by the storage service from Rackspace.

00:24:36.760 Let's consider the practical scenario of creating a DNS record for our load balancer. We begin by creating a new DNS service instance using `fog.dns.new`, specifying our provider. Similar to other services, we pass in credential information, leading to an increasingly familiar pattern in how we apply services in Fog.

00:25:35.520 For demonstration, we will employ the request layer to create a new DNS record and check the available requests through `dns_service.requests`. From there, we will utilize `list_domains` and `add_records` methods to browse existing domains and subsequently add our desired record.

00:26:30.400 When we retrieve domain details through `service.list_domains`, we can expect an X-Conn response, allowing us to easily manage domains with Ruby objects rather than raw JSON. This transformation is automatic, providing a smooth programming experience.

00:27:31.760 Programming practices are evident here, including the extraction of headers from the response, which can be important when handling various cloud services that utilize data transfer methods, especially within storage services.

00:28:20.720 Let’s proceed by adding our DNS record. We will obtain the desired load balancer's IP address programmatically, then supply the details through `service.add_records`. This will include an A-type record associated with the IP address and set a desired TTL if necessary.

00:29:13.520 To wrap up, this example serves as an illustration rather than a definitive architecture. In a real-world application, we would likely implement additional components like databases, monitoring, and secure networking between VMs.

00:29:55.680 I want to conclude by providing a deeper examination of how Fog handles requests. For instance, I recall the ID of the Ubuntu image we discussed. We can retrieve it using `service.images.get`, verifying that the collections class indeed invokes the request layer.

00:30:51.840 The model layer should never directly access the HTTP layer; maintaining a clean architecture is crucial. When digging deeper into the Get method, we observe that the request layer strategy aligns neatly with the architecture of Fog.

00:31:45.360 By employing the request details obtained from Rackspace, we can conduct proper requests with set headers and expected response parameters. The token received from authentication will remain valid for approximately 24 hours, classifying all our requests.

00:32:39.360 For testing purposes, Fog's integration allows for mock data via commands like `fog.mock!`, or the usage of the VCR gem to record request details for future tests.

00:33:06.000 And that is my presentation! Thank you for attending.

LoneStarRuby Conf 2013