Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
Like most web applications, you run important jobs in the background. And today, some of your urgent jobs are running late. Again. No matter how many changes you make to how you enqueue and run your jobs, the problem keeps happening. The good news is you're not alone. Most teams struggle with this problem, try more or less the same solutions, and have roughly the same result. In the end, it all boils down to one thing: keeping latency low. In this talk I will present a latency-focused approach to managing your queues reliably, keeping your jobs flowing and your users happy.
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
In this presentation titled "What does 'high priority' mean? The secret to happy queues," Daniel Magliola discusses the challenges teams face when managing job queues within web applications, emphasizing the importance of maintaining low latency for a smooth operation. He introduces the story of Sally, a lead engineer who continually finds her team's job queues overwhelmed despite attempts to solve the issue with separate high and low priority queues. Key points covered include: - **The Origins of Queue Problems:** Sally's company initially thrived with a small number of developers, but as complexity grew, support tickets began highlighting issues like delayed password reset emails due to a backlog of 40,000 jobs in the queue. - **Ineffectiveness of Priorities:** Joey, another engineer, attempted to create priority queues, but issues persisted as critical jobs got slowed down by longer-running tasks in the same queue. - **Proposed Solution by Marianne:** A senior engineer recognized that organizing by priorities failed because there's no clear understanding of what constitutes 'high' or 'low' priority. Instead, she suggested creating separate queues based on job types to manage latency effectively. - **Understanding Latency:** The crux of the issue lies in accurately measuring and managing latency rather than attempting to set job priorities. Jobs should be categorized by how much latency they can tolerate, and queues must be renamed to reflect those guarantees—like 'within ten minutes'—creating a clear agreement between job timing expectations and performance. - **Implementation of Fixed Latency Contracts:** By grouping jobs into defined latency limits and requiring jobs to finish within those constraints, development teams ensure clarity and accountability in their queue management. This creates a responsive system where alerts trigger if latency expectations are breached, thus preventing issues before they arise. - **Ongoing Maintenance and Improvement:** Teams focused initially on easy wins by categorizing critical jobs, gradually shifting high volume and long-running tasks into designated queues without overwhelming the system. They learned to balance efficiency with performance needs while continually iterating on their system structure. In conclusion, the presentation emphasizes the importance of understanding job latency over vague prioritization, driving home the point that effective queue management is essential to ensure a healthy and efficient system operation, ultimately leading to happier users and developers alike.
Suggest modifications
Cancel
