Fog or: How I Learned to Stop Worrying and Love the Cloud

by Wesley Beary

In the video titled 'Fog or: How I Learned to Stop Worrying and Love the Cloud,' Wesley Beary discusses the challenges and solutions related to cloud computing with an emphasis on his project, Fog. Beary, who works for Engine Yard, was motivated to create Fog to provide a simpler way to manage cloud services, addressing the complexities and steep learning curves associated with various cloud providers. The main points covered in his presentation include:

Definition of Cloud Computing: Beary outlines cloud computing as an API-driven service that is available on demand, where users only pay for what they consume, making it more flexible and repeatable than traditional hosting.
Challenges in Cloud Services: He highlights the overwhelming number of options and providers (such as AWS and Rackspace), the expertise required to navigate these services, and the issue of vendor lock-in as significant barriers that newcomers face.
Creation of Fog: In response to these challenges, Beary developed Fog to abstract multiple cloud services into a single framework, allowing users to switch providers easily without deep knowledge of their APIs. Fog simplifies the setup and management of cloud resources through higher-level abstractions and includes features such as mocks for testing purposes.
Use Case - Uptime Monitoring: Beary provides a practical example of using Fog to create an uptime monitoring service, demonstrating how to set up servers across different providers for redundancy and how to manage those resources efficiently.
Advantages of Using Fog: Fog not only eases development efforts but also allows for quick iteration, configuration, and cleanup processes, all crucial for managing costs associated with cloud resources. Beary emphasizes the adaptability of cloud computing and how Fog can facilitate rapid growth in tech solutions.

In conclusion, the video presents Fog as a viable and effective solution for developers looking to engage with cloud computing without the frustration typically associated with multiple providers. Beary encourages viewers to embrace cloud tools as a means to innovate and succeed in their projects, fostering a supportive community around Fog and cloud development.

00:00:12.480 All right, so I'm going to talk about Fog.

00:00:14.160 I work online, which is probably more useful for GitHub. I work for Engine Yard, and they were kind enough to employ me to work on Fog full-time. I've been doing that since the beginning of September, and the amount of progress that I've been able to make in that full-time capacity has been amazing compared to the bits and pieces of time I could scrape together when it was just a side project.

00:00:22.720 As it's been mentioned a couple of times, we are doing a half fest tonight. I will definitely be there, probably for the whole time if I can manage to stay awake that long. It should be a lot of fun and if you have any questions about this stuff, it would be a great place to ask.

00:00:33.680 First off, just to be clear, the term "cloud" is a word that gets thrown around a lot these days. So just to clarify what I mean when I talk about cloud—since it's something I work with a lot—is that it's something that's API-driven and available on demand. I believe those are both key to what cloud computing is. The core things from that framework involve computing, DNS, or storage, but there are also some things that are kind of on the fringes, such as key-value storage, where you might find a few places that offer a service in this kind of way, but for the most part, you're kind of on your own.

00:00:59.760 To dig in a little bit more, since it's on-demand, you only pay for what you use. This is different from traditional hosting, for instance, where you pay for server months instead of server hours or server minutes. This is really nice, especially if you're working on a project over the weekend. For example, you might only want to run your servers during the weekend when working on it and not pay for the server during the week when you didn't get to touch it because you're busy with your day job.

00:01:19.360 It's also flexible; you can add and remove resources in minutes, not days or weeks or months as it used to be before the advent of cloud services, where you had to fill out the forms to get someone to approve the resources. It's repeatable too; you can set up a server, ensure some configuration happens correctly, and you can have confidence that you can do that again tomorrow if you need one more server or the next day if you need three servers, and so on.

00:01:44.720 From that perspective, you can build systems that are more resilient, because if you consider that servers may not always be available, you can design systems that allow for removing resources as needed. So that's great, but then why am I worried, right? This all sounds very promising.

00:02:04.720 Well, the problem is that you quickly run into option overload. There are a lot of different providers and services available, which can be very hard to navigate when you first enter the space. You might wonder, ‘Do I want AWS? Do I want Rackspace? What are the trade-offs?’ It can be difficult to figure out costs and features associated with each provider.

00:02:40.160 Along with that, there's the expertise required; you have to learn a whole different set of knowledge to work with each individual provider, which can be incredibly challenging and involves a lot of upfront effort. It also creates a situation where you're locked in; once you've put in the time to learn AWS, for instance, it would be difficult to switch to another provider after you've invested all that effort.

00:03:03.160 When I was first coming into this, I found that the tools available were vastly different from each other, often completely different from the APIs they were meant to be interfacing with. The quality and maintenance of those tools varied significantly; it could be very challenging to know if they were still actively maintained or not.

00:03:30.640 I decided to start working on my own solution. Another common question I get revolves around standards. Standards could potentially be a way to navigate this space, but I think the standards process has been slow, and often the proposed standards end up being interpreted in such vastly different ways that it can be almost comical.

00:03:50.520 My approach to dealing with these challenges ultimately led to the creation of something called Fog. It started out pretty humbly; I initially worked on a couple of little services simply to learn them better. I did a lot of low-level stuff that wasn't particularly user-friendly and required a lot of expert knowledge.

00:04:06.920 Eventually, I built abstractions on top of that, which improved the usability over time. It got better and better until I recognized that I was on to something significant. I think the key advantages of using Fog are its portability and how it allows you to move between different cloud providers quite easily. If you use the higher-level abstractions, you can shift between providers without having to deal with the nuances of each provider’s API.

00:04:38.400 Using these higher-level abstractions provides flexibility and power. It's quite established—recently, I reached over a thousand followers on GitHub, and I have 85,000 downloads along with a bunch of contributors and forks. I’m working on Fog full-time, which, I hope, indicates that it’s here to stay. There are also mocks built in, so in a lot of cases, you can try out what will happen and work on building out exactly what your provisioning workflow is without having to spend real resources.

00:05:10.760 At some point, you do need to interact with live resources, but mock environments can significantly ease the testing, bootstrapping, and discovery processes. We already have a decent number of users who are utilizing both products and libraries which is quite exciting.

00:05:30.840 Now let's engage in a little interactive part to ensure everyone is awake after the dry definitions. A quick show of hands for those who actively use cloud services as I’ve defined it. It seems like a good number. Now keep your hands up if you’re doing it with Fog. It’s a much smaller number, but at least there are a few of us using it.

00:06:01.080 For most of you, this probably sounds intriguing, but you might feel you lack a use case for it. You don’t need cloud computing on a day-to-day basis. So for the purpose of this presentation, I'm going to lay out a specific use case: we're going to build an uptime monitoring site. We already have a tagline: 'Who wants a busted site?' This will be our next big thing.

00:06:36.440 We’ll work on it over the weekends and hope to charge five bucks a month for it. If we get around 20,000 people, that’s a decent revenue stream. First off, we’ll need to set things up.

00:06:56.440 You can install Bundler or Sudo if you’re not using RVM, and ensure that your system is configured correctly. The first step is to create a connection representing all the different compute providers in an abstract way by passing some credentials. For Rackspace in particular, we’ll provide our credentials and specify what server we want to create.

00:07:21.320 The process involves calling 'create server' with the necessary parameters to initiate it. We’ll have to sit and wait a bit until the server becomes active, as we can’t proceed until it’s ready. After that, we can run SSH commands to place our keys, disable the root password, and start everything up.

00:07:47.600 Setting this up is straightforward; it’s generally easy. However, it can be frustrating because it requires specific knowledge and expertise. Most people may find it challenging to acquire that level of detailed knowledge within a short time, especially when building something for a weekend project.

00:08:16.320 Often, the first 10 weekends might simply be about learning to use cloud services adequately enough to get your project off the ground. The arguments needed for the server creation may seem cryptic and specific to the provider.

00:08:53.920 This experience made me realize that the tooling wasn't achieving much. I was effectively making API calls from scratch with only basic tools like Net::HTTP, which didn't simplify the process at all. Since then, I've made quite a few improvements to help users more effectively manage their cloud resources.

00:09:32.920 Now, instead of dealing with the low-level commands, we can work with a collection of servers provided by Fog. This allows us to bootstrap servers by simply specifying their attributes without deep diving into API specifics. This saves an immense amount of time and complexity.

00:10:05.600 We can use the lazy loading feature to call server collections, get a list of available servers, or fetch a specific one as needed. Adjustments like these greatly improve the ease of managing cloud resources.

00:10:32.480 Now, let’s showcase how we would actually execute a ping request from the created server. This involves connecting via SSH and executing a series of commands to check uptime for various targets.

00:11:03.760 After executing the commands, we would need to process the results. This involves parsing the output and aggregating data which enables users to collect useful metrics.

00:11:32.520 Once done, it’s important to clean up by shutting down and destroying the servers to avoid unnecessary charges, which leads us to the previous examples of creating uptime monitoring across multiple providers.

00:12:02.080 As an uptime monitoring service, we might want to set up servers in various locations, perhaps on both AWS and Rackspace for redundancy. It’s beneficial to manage multiple server locations to ensure uptime even in the face of regional issues.

00:12:40.080 Switching between providers can be seamless if we employ the same code base while changing credentials and attributes as necessary. This ease of transition is particularly evident when leveraging similar functionalities, making the developer experience far superior.

00:13:24.520 Furthermore, we can integrate additional locations by simply adding new configurations without substantial changes to the existing implementation. The subsequent iterations of this geolocation strategy can be smoothly adjusted to new requirements.

00:14:07.280 Now, you may wonder how I strategized knowing which images to use for setting up servers. Fog includes a command that facilitates listing available images, which is very user-friendly. In the context of AWS, users can visit certain listings like elastic.com to find the most recent and relevant images.

00:14:43.720 Exploration can be done effectively alongside mocking features built into Fog, allowing testing without using live resources. Utilizing the mock environment reduces the costs associated with random trial and error.

00:15:24.520 When you want to interact with the cloud but avoid unnecessary expenditure, mocking enables you to simulate actions. This simulation will allow you to validate practices and inspect outputs safely without the fear of accruing costs.

00:16:10.920 Mocking takes the stress out of validating your procedures through simulation, while you still maintain a way to run effective tests and checks.

00:16:34.720 Now pivoting back to the subject of how to work with the data gathered— we don't want all this on our local machines without backup. Cloud storage solutions allow for efficient aggregation and are highly beneficial in storing server states or data outputs.

00:17:00.360 Here, we can create directories and files in the cloud that act similarly to how traditional file systems work. The benefits continue with scalable integration of additional services alongside our existing framework.

00:17:25.480 Through these processes, we can structure data effectively and ensure its public visibility, enabling seamless access for stakeholders. It's essential to understand the differences associated with cloud file management, as some actions might require adjustments or changes at both levels.

00:18:06.000 When crafting premium services, DNS plays a foundational role in creating dedicated subdomains for customers. This allows end-users to view their uptime metrics directly while abstracting the underlying computational complexity that sustains it.

00:18:44.120 Opting for a domain registration setup, every new zone you create connects users to new domains, each with its own namespace for tracking uptime effectively. From these simple principles, we can construct increasingly complex structures, as the world of cloud services enables a remarkable extensibility.

00:19:25.000 For cloud services to be truly effective, it’s paramount to maintain a concise and direct approach, ensuring the clean operation of your infrastructure. This ongoing cleanup prevents resource overuse while upholding accessibility for users seeking uptime reports.

00:20:03.560 Finally, the lesson here is to capitalize on this adaptability of cloud computing. It opens the doors for rapid growth, allowing you to pivot service offerings quickly as customer requirements arise in the market.

00:20:44.960 As you evaluate how to bring your ideas to fruition, consider how cloud computing lowers barriers and encourages innovation to leverage time and resources efficiently. I hope this introduction has sparked your enthusiasm.

00:21:13.960 As we wrap up, I encourage everyone to take these concepts and apply them in their workflows. This isn't just a roadmap; it's a starting point where your potential impacts the broader tech landscape.

00:21:43.080 If you want further insight into Fog, follow the GitHub repository or join the community forums. I also have stickers available, so don’t hesitate to find me after the talk.

00:22:09.960 As we conclude, I want to recognize the opportunity here. My journey was filled with challenges, as I initially struggled to understand these emerging technologies. If my experiences can make this journey smoother for others, then I am more than satisfied.

00:23:00.760 Cloud computing represents innovation and accessibility in the tech world, and I genuinely want to see others succeed in taking full advantage of these tools. Should we work together, I believe we can contribute positively to this ever-evolving landscape.

00:23:34.200 In the grand scheme, I am grateful for the community that has developed around Fog. We can all contribute to creating a more integrated, supportive environment for future developers to thrive.

00:24:07.960 As I mentioned, I enjoy coffee, bourbon, and games, so let’s mingle and build a connection beyond these technical discussions. Shifting perspectives to the artistic side of coding can rekindle inspiration, and that’s something I truly wish for everyone.

00:25:10.760 Thank you once again for being engaged during this presentation. I hope that the insights and methods shared here can translate into tangible action for all of you.

00:25:41.560 Don’t forget about the community around Fog—each contribution builds toward a stronger future. And now, let’s see where these ideas take us!

MountainWest RubyConf 2011