Beehive, Scalable Application Deployment

Talks

Ari Lerner

2 talks

#application-deployment

Beehive, Scalable Application Deployment

by Ari Lerner

The video presentation titled "Beehive, Scalable Application Deployment" by Ari Lerner at the LoneStarRuby Conf 2010 addresses the complexities of deploying applications and introduces Beehive, an open-source framework designed to streamline the deployment process. Lerner begins by outlining the challenges developers face when transitioning from application development to deployment, such as selecting the right HTTP server, scaling beyond a single instance, and managing server resources efficiently. The following are the key points discussed throughout the presentation:

Introduction to Beehive: Beehive is positioned as a solution to overly complex application deployment stacks, emphasizing its features such as fault tolerance, scalability, and multi-tenancy.
Deployment Challenges: The current deployment environment is often inefficient, with resources underutilized and scheduled releases creating additional hurdles for developers.
Simplified Developer Experience: With Beehive, developers can deploy applications using a simple command like "git push" or via HTTP, allowing them to focus on development without getting bogged down by deployment complexities.
How Beehive Works: Beehive enables intelligent routing and holds requests during deployment so that the transition between application versions happens smoothly, boosting uptime to an impressive 99.9%.
Graceful Handling of Failures: Beehive is designed to manage unexpected failures by allowing old application versions to serve requests if the new version isn’t ready, ensuring continuous service to users.
Future of Beehive: Lerner highlights the potential for Beehive to grow with new features and community contributions, enhancing its utility for developers.

In conclusion, Beehive represents a significant step forward in modern application deployment environments, automating various deployment tasks while providing developers with a more user-friendly experience. The presentation encourages further exploration and potential collaboration in enhancing the capabilities of Beehive, making it an invaluable tool for developers looking to improve their deployment processes.

00:00:00.599 Hello everyone.

00:00:10.500 Okay, I have a lot of slides to get through, so I'm just going to start. If you have questions, please hold them until the end.

00:00:15.660 There are two slides that prompt for questions. Did you hear that? It's just a prompt for questions.

00:00:21.900 That's cool. Okay, um, so I'm here to talk about Beehive.

00:00:27.599 The first part of the talk is going to go by kind of fast, and you'll probably wonder what you are actually listening to. That's okay because in the second part of the talk, I'll hopefully reveal that; we just have to get through the first stuff.

00:00:39.120 So just to get started, Beehive: Scalable Application Deployment by Ari Lerner.

00:00:45.000 Yes, that would be me. Oh boy.

00:00:50.160 Okay, so I'm a lead software engineer at AT&T Interactive in Glendale, California.

00:00:55.399 I am soon to be an innovation coach at AT&T. On Wednesday, I moved to the San Francisco Bay Area.

00:01:07.140 I'll be starting a new job.

00:01:12.299 I've been working with Ruby since 2004. Oh man, I feel like I'm getting older.

00:01:17.520 I've been working with Rails since 2005. However, I haven't used Rails in about a year.

00:01:23.759 Right now, I primarily work in distributed computing and systems deployment at AT&T Interactive.

00:01:30.540 I also have a totally rad Mini Cooper; unfortunately, it's not the one in the picture because that's a really cool Mini Cooper.

00:01:37.740 I also have an awesome dog. Don't tell anyone I have that because she's just a little girl.

00:01:44.640 So, I've been working in open source for a long time, primarily in open source.

00:01:50.100 You can see all my open-source projects at GitHub under my name, Ari Lerner.

00:01:55.619 These are just examples of some of my open source work.

00:02:01.140 But let's focus less on me and more on what I'm here to talk about today, which is Beehive.

00:02:06.780 The subtitle is scalable application deployment, as the program suggests.

00:02:12.900 So, what is it? Oh man, raise your hand if you can't deal with the resolution problems.

00:02:20.300 What is it? Beehive is an open-source solution to complex application deployment stacks.

00:02:26.220 It's fault-tolerant, application-agnostic, natively scalable, and event-driven, which are important features. It's also multi-tenant and has intelligent routing.

00:02:41.760 It is a deployment system—let me clear that up—it's not a Ruby framework or a Ruby library. Since I'm at a Ruby conference, I should make that distinction.

00:02:54.000 It's MIT licensed and free. As I mentioned before, it's open source, tested, and currently in use internally in production at AT&T Interactive.

00:03:06.480 It's extensible, and I’ll show you how in a little while. It's distributed, written primarily in Ruby, Bash, C, and Erlang, with three AT&T Interactive developers collaborating on it, as well as open-source contributors from the web, notably one person in Argentina.

00:03:19.099 So, that was a whole bunch of buzzwords, am I right?

00:03:24.420 So, what actually is it? Beehive is a tool that enables developers to deploy and manage their own applications in a scalable, multi-tenant environment without having to deal with deployment directly.

00:03:33.060 Okay, so basically, it's just an intelligent routing mesh. That's a lot easier to understand.

00:03:38.760 So why do we do this? Why even change anything? Because deployment today is really rough. It requires a lot of resources—human resources, computer resources.

00:03:56.040 Most of the time, you end up deploying one application to one computer, so 99.9% of the time, that computer is doing nothing.

00:04:01.680 Unless it’s an AMHadoop mesh, in which case, 88% of the time, it's not doing anything. The other problem with deployment today is that you have to deal with scheduled releases.

00:04:20.220 It’s like you're not going to deploy until Thursday at 2 AM, and then you have to stay up until Thursday at 2 AM.

00:04:26.040 So really, I'm a developer; I'm not a deployment person. Well, that's not entirely true anymore for me, but most of the time, what I want to deal with is only development. I don't want to deal with deployment.

00:04:38.759 Deployment today does not focus on the application nor does it focus on the developer or anyone else involved.

00:04:50.220 I suggest we have an application revolution! Let's focus on the application. Let the server managers manage the servers and let the developers develop.

00:05:06.180 What we want is to make it easy to deploy rapidly, so when you add a change and all your continuous integration server tests pass, you can deploy immediately.

00:05:13.320 There should be no need to wait.

00:05:20.699 Beehive enables rapid deployment. It’s built to scale; your application can grow as needed.

00:05:34.620 For example, if your application is like Twitter and all of a sudden people in Iran are using it to express political dissent, then you have to be prepared to scale.

00:05:46.320 Beehive also handles failures gracefully. It's designed to help streamline the deployment process, so developers don’t have to learn new tools and can easily integrate it into their workflow.

00:06:04.860 We at AT&T Interactive like to use 'git push.' You can also utilize pure HTTP with Beehive.

00:06:11.280 Most of the examples I'll cover today will focus on pure HTTP, which is our case, but there is a Ruby gem being developed. If you want to contribute and don’t know Erlang or C or Bash, you can also create your own HTTP client in Python, for example.

00:06:31.940 So, how do we work with Beehive? First, you need to define an app.

00:06:41.640 You can do that using a curl request or soon the Beehive gem to create an application. Essentially, it takes your application and prepares it for the Beehive environment.

00:06:57.600 How do you deploy an application? You can use either a git post-commit hook, which is what we do at AT&T Interactive, or you just make a curl request.

00:07:16.919 Beehive checks out and bundles the application and performs error checking at every stage. If it can’t clone it because, say, your public key isn't there, it will let you know.

00:07:29.759 If it can't find a dependency during the bundling process, it will also notify you. It’s very communicative.

00:07:42.700 Once everything succeeds, Beehive is designed to hold incoming requests until the application is ready. It starts routing requests to the newly deployed applications.

00:07:53.520 If your application goes down for any reason, the old version will continue to serve until the new version is fully prepared, ensuring 99.9% uptime.

00:08:06.000 This dramatically reduces the time needed for deployment from scheduled releases to mere seconds.

00:08:22.560 The router will temporarily hold any HTTP requests until the application can accept them. This way, all incoming requests are managed without creating a bottleneck.

00:08:42.899 Beehive facilitates intelligent routing to manage heavy traffic efficiently. For example, if there are sudden spikes in user activity, Beehive can scale itself seamlessly without developer intervention.

00:09:01.320 It's equipped to handle these challenges efficiently thanks to its intelligent router. If a server goes down, the connected clients remain unaware of any disruption.

00:09:19.080 The failover process is handled automatically and distributed among active services.

00:09:30.000 But wait, what happens if an HTTP request comes in while you're deploying an app that isn't up? The intelligent router in Beehive holds requests until it can successfully establish a connection with the application.

00:09:55.500 Once the application spools up, requests flow smoothly and as intended.

00:10:01.860 This process can significantly improve deployment times, making it less of a chore and more accessible for developers who want to focus on creating outstanding applications.

00:10:16.740 In the case that a malfunction occurs during a deployment, Beehive’s design maintains maximum uptime. Requests will continue to be processed through the old application until the transition is confirmed successful.

00:10:27.000 In summary, Beehive aims to transform the deployment experience.

00:10:38.520 For a more developer-friendly approach to app delivery, Beehive represents a significant advancement in modern application deployment.

00:10:47.700 The architecture prioritizes application performance while providing robust failover capability.

00:11:03.060 Beehive is designed to help developers focus on writing code instead of worrying about the deployment process.

00:11:17.640 It automates various deployment tasks allowing quick responses to user needs.

00:11:30.000 Whether deploying a new feature or updating an existing one, Beehive's routing allows for speedy transitions.

00:11:46.860 Developer tools like Beehive can enhance collaboration while simplifying the entire deployment experience. There are many exciting opportunities for enhancing Beehive in the future.

00:12:00.000 As the community continues to grow, the potential for Beehive to integrate new features will also diversify, making it a valuable asset to developers everywhere.

00:12:25.500 With that, I’ll keep you posted as we navigate the evolving landscape of deployment. Thank you for your time, and now I'm open to any questions.

00:12:35.640 Feel free to ask anything about Beehive or my experiences with application deployment.

LoneStarRuby Conf 2010