Suggest modification to this talk

Title

Description

Services, Scale, Backgrounding and WTF is going on here?!??! by: David Copeland

You want to improve the performance of your app, or you want to keep your system composed of small, easy to understand services. You start using background jobs, REST calls, and cron. And then weird things happen.

Designing services from the start can be tricky, but there is guidance out there. Extracting services can be even trickier, and whenever there's a message queue or job processing system, it becomes very difficult to truly understand the order in which things happen in your system. If you're lucky, you've got alerting when things go wrong, but even then, what do you do about it?

This talk will go through an increasingly frustrating set of circumstances that I've seen on a regular basis at LivingSocial as we extracted code from a monolithic app into a set of services. I'll then show the solutions to these issues that make our payment processing system more or less bullet-proof, and generalize these lessons into what you can do when extracting and designing services.

Date

Summarized using AI?

If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.

Show "Summarized using AI" badge on summary page

Summary

Markdown supported

In the talk titled 'Services, Scale, Backgrounding and What is Going on Here?', David Copeland shares insights from his experience transitioning a monolithic Rails application at LivingSocial into a service-oriented architecture. The discussion outlines the complexities encountered during this journey and offers practical solutions to common challenges faced by developers.

**Key Points Discussed:**
- **Transitioning to Service-Oriented Architecture:**  The shift from a monolithic application to smaller services can be challenging, requiring developers to make reasonable decisions amid unexpected complications.
- **The Importance of Background Processing:**  When users sign up, the synchronous task of sending confirmation emails can cause bottlenecks under high load. By implementing background processing with tools like Resque and Redis, email notifications can be sent asynchronously, improving user experience.
- **Database Transactions for Consistency:**  To prevent inconsistency in user states, the implementation of database transactions ensures that both user account creation and email notifications either succeed or fail together, thus avoiding partial failures.
- **Handling Edge Cases and Technical Debt:**  As new functionalities accumulate, issues such as race conditions can arise. To address these, employing retry mechanisms and ensuring proper job processing can help maintain system stability.
- **Growth and Complexity Management:**  With company acquisitions leading to integration challenges, adapting existing logic to handle bulk user activations while preventing technical debt is crucial.
- **Need for Continuous Improvement:**  After facing difficulties with the email service causing duplicate emails, Copeland emphasizes a need for a centralized logging and robust testing protocol to improve reliability.
- **Best Practices and Future Directions:**  Concluding with a call to prioritize clean architecture, monitor background processes, and maintain technical agility, he highlights the importance of being proactive in design to mitigate future frustrations.

The key takeaway is the significance of understanding and carefully managing the complexities of service architecture transitions to enhance performance and user experience.

Cancel