Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
By Tanner Burson At Tapjoy we process over a million jobs an hour with Ruby. This talk is a discussion of tools, techniques, and interesting problems from the trenches of scaling up over the last two years. Expect to learn a lot about Ruby job queues (beyond Resque/Sidekiq), performance, concurrency and more. Help us caption & translate this video! http://amara.org/v/FG3n/
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
In the talk "Mo' Jobs Mo' Problems" by Tanner Burson at the Big Ruby 2014, the speaker discusses the intricacies and challenges faced when scaling job processing systems at Tapjoy, a mobile ad platform. The session emphasizes the significant growth in traffic handling—from upgrading to Ruby 1.9 to processing millions of jobs per hour, presenting several intriguing problems faced during this scaling phase. Key Points Discussed: - **Job Processing Challenges**: Burson introduces the struggles of managing job queues, particularly during high-traffic periods, where job backups reached tens of thousands, resulting in inefficiencies and data loss during event processing. - **Importance of Scalable Systems**: The necessity for distributed systems and avoiding single points of failure is highlighted. The naive solutions to job processing faced immediate breakdowns, demonstrating the need for robust designs. - **Queue Selection and Trade-offs**: Several queuing options were evaluated, including RabbitMQ, SQS, and Redis, based on factors like durability, availability, and throughput. Each queue had pros and cons, leading to decisions based on specific use cases. - **Building a Custom Job Processor**: Finding existing job processors that fit Tapjoy’s requirements was challenging, leading to the development of a custom solution called Chore that could handle a variety of job types efficiently. - **Concurrency in Ruby**: Burson explains the methods available for concurrency in Ruby, including forking, fibers, and threads, and shares his experiences and challenges in using these methods for effective job processing. - **Debugging Challenges**: The complexities of debugging Ruby code, particularly in high-scale environments, are emphasized, with insights into race conditions and the importance of a deep understanding of the Ruby environment. Conclusions and Takeaways: - Job processing is a non-trivial task, especially at scale. The growth in request volumes brings challenges that require detailed consideration of design and implementation. - Different workloads require tailored solutions; not all jobs can rely on the same processing strategies or systems. - Recognizing the importance of availability and durability over throughput in some contexts leads to more informed decision-making in system design. - Continuous learning from past experiences is crucial for building resilient and effective job processing systems. Overall, Burson’s talk conveys that navigating the realm of job processing at scale in Ruby requires attentiveness to numerous factors, flexibility in solutions, and readiness for unexpected challenges while developing robust systems.
Suggest modifications
Cancel
