Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
Mostafa Abdelraouf shares the journey of evolving Instacart's Rails application beyond a single Postgres instance. He discusses how they managed the added complexity from adding read replicas, and later vertically and horizontally sharding, and also touched on the topics of query routing, connection pooling, and load balancing at Instacart's scale. #railsatscale #postgres #sharding Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/ Stay tuned: all 2024 Rails World videos will be subtitled in Japanese and Brazilian Portuguese soon thanks to our sponsor Happy Scribe, a transcription service built on Rails. https://www.happyscribe.com/
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
The video titled "Going Beyond a Single Postgres Instance with Rails" features Mostafa Abdelraouf, an engineer from Instacart, who discusses the evolution of their Rails applications from a single Postgres instance to a more complex architecture involving multiple instances and sharding. He emphasizes the significance of understanding query patterns and connection pooling before scaling databases, mentioning that performance improvements can often be achieved without adding complexity. Key points discussed include: - **Initial Setup:** Instacart embarked on its journey with Rails 3.1 and a single Postgres instance, which served its needs initially. - **Scaling Challenges:** As Instacart grew, the need to introduce read replicas, manage replication lag, and implement more complex routing strategies became apparent. - **Connection Management:** Mostafa shared insights gained from using the MAA gem for managing multiple Postgres adapters, which helped in load balancing and automatic query routing until performance issues led them to rewrite their own solution. - **Routing Strategies:** The talk covered the strategies for directing read and write queries either to replicas or the primary instance, emphasizing the importance of managing replication lag and ensuring efficient load on the primary. - **Connection Pooling:** Mostafa stressed the necessity of using connection poolers like PG Bouncer and PG Cat to effectively manage connections and load balance across instances, especially during failover scenarios. - **Sharding Implementation:** The need for sharding, especially during the operational challenges faced during COVID-19, was highlighted. Mostafa detailed both vertical and horizontal sharding approaches and the considerations necessary for effective sharding implementations, such as maintaining data accuracy during transitions. - **Future Considerations:** The discussion emphasized that sharding is an evolving landscape, and it's critical to ensure that the sharding key is effectively incorporated into queries to avoid complications. Overall, the talk provided valuable insights into the complexities of scaling a Rails application with Postgres, the importance of understanding system behaviors, and strategic approaches for efficient database management. Mostafa concluded with a reminder about the ongoing evolution of sharding practices and the need for continuous assessment of database architectures as systems grow.
Suggest modifications
Cancel
