Lightning Talk: Turning Web Servers into Workers

by Janusz Mordarski

In the lightning talk titled "Turning Web Servers into Workers," Janusz Mordarski, a freelancer at Sync, shares valuable insights on efficiently launching web applications, focusing on the use of AWS Elastic Beanstalk. The session emphasizes the importance of scalability and the integration of web servers as background job workers.

Key points discussed in the video include:
- Application Deployment: Mordarski recommends platforms like Heroku and AWS for quick application launches. He highlights AWS Elastic Beanstalk as a user-friendly alternative, particularly for users already within the Amazon ecosystem, noting its ease of setup.
- Integration with AWS Services: He explains how Elastic Beanstalk dovetails with other AWS components such as Elastic Load Balancers, EC2 instances, RDS databases, and SQS, enhancing scalability and ease of management.
- Ruby on Rails Compatibility: The speaker points out that Elastic Beanstalk supports Ruby on Rails using pre-configured Docker images, streamlining deployment and management through the Elastic Beanstalk CLI.
- Utilizing Cron Jobs: A significant focus of the talk is on transforming web servers into dedicated workers that can efficiently handle background jobs by leveraging cron jobs set up through Elastic Beanstalk.
- Distributed Mutex for Job Management: Mordarski addresses the challenge of job concurrency and proposes using a distributed mutex with Redis—specifically, utilizing the Redis Mutex gem. This technique wraps job code in a lock to prevent simultaneous execution across multiple servers, thus eliminating conflicts and ensuring jobs run smoothly.

The main takeaways from the talk include the emphasis on scalability from the outset of development and the importance of proper job management to avoid failures likely caused by overlapping executions. Overall, the insights shared equip developers with practical strategies for using AWS Elastic Beanstalk to effectively manage web applications and their background job processes.

00:00:02.590 Hello, everyone! My name is Janusz Mordarski, and I am currently a freelancer at Sync. I split my time between Hamburg and Krakow, which I highly recommend for weekend trips. It’s a beautiful city.

00:00:14.289 Today, I want to share some insights on quickly launching web applications. Typically, I would recommend using platforms like Heroku or AWS. For those who find themselves tied to the Amazon platform, Elastic Beanstalk is a great alternative. It serves as a Heroku-like solution for AWS, allowing you to pick some pre-configured images from the internet.

00:01:11.710 Elastic Beanstalk works seamlessly with various AWS components, including Elastic Load Balancers, EC2 instances, RDS databases, and SQS. The setup is straightforward; with the click of a button, you can integrate RDS databases or Elastic Cache, making it easily scalable. I recommend using Elastic Load Balancer, which comes with additional scalability to handle any number of users.

00:02:05.020 It works with Ruby on Rails right out of the box, utilizing pre-configured Docker images. This makes management effortless using the Elastic Beanstalk CLI. Deployments and debugging in production can be handled with ease using EB SSH, where you can manage files for cron jobs, for example.

00:02:30.430 The focus of my talk is on transitioning web servers into workers that can execute background jobs quickly. I make use of the Elastic Beanstalk extension to set up cron jobs for both staging and production environments. Scalability is essential, so I advise considering load balancers from the start of your project.

00:03:38.170 Many people ask me how to best set up cron jobs in Elastic Beanstalk. There is a solution that has existed for a while called Mutex. This is a distributed mutex that can be implemented with Redis using a gem like Redis Mutex. This allows you to wrap your job code in a lock to prevent conflicts across multiple servers.

00:04:43.160 By specifying how long this lock should live, the job will run on one server while preventing it from executing on others simultaneously. This ensures that you don’t face issues with jobs overlapping or failing due to multiple executions. That’s all from me today. Thank you!