Suggest modification to this talk

Title

Description

RubyConf AU 2013: http://www.rubyconf.org.au

If you've ever done anything in ruby, you've probably used rubygems and to search or install your favorite gem. On October 17, 2012, rubygems.org went down. A Dependency API was built to be used by Bundler 1.1+ to speed up bundle install. Unfortunately, it was a bit too popular and the service caused too much load on the current infrasture. In order to get rubygems.org back up the API had to be disabled. You can watch the post-mortem for details.
Members in the community stepped up and built a compatible Dependency API service called the Bundler API. Working with the rubygems.org team, we were able to bring the API up for everyone within a week. In this talk, I will cover the process we went through of extracting the API out into a separate Sinatra app that now runs on Heroku. I'll go over how the API works and how Bundler itself uses it. Since we don't have direct access to their database, we needed to build a syncing service to keep our local cache up to date. We'll discuss the original sequential implementation of the sync code and how we improved performance through the use of a consumer thread pool. The sync time down was cut from 16 mins to 2-3 mins. Next, we'll talk about the productization steps we took for visibility to make sure the app is performing well.
We're prototyping a replay service, to replay production traffic to different Heroku apps. With this we can compare the performance between the current MRI app in production and the same app running on JRuby. This will also allow us to test any changes/features against real production load. We'll go over how to set this up.

Date

Summarized using AI?

If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.

Show "Summarized using AI" badge on summary page

Summary

Markdown supported

The video titled "bundle install Y U SO SLOW: Server Edition" features Terrence Lee discussing the challenges and improvements associated with Bundler, a dependency manager for Ruby. Presented at RubyConf AU 2013, the talk dives into the need for a robust Dependency API due to performance issues encountered when RubyGems.org went down in 2012.

**Main Points Discussed:**
- **Introduction to Bundler:** 
  - Bundler simplifies the management of Ruby gems, much of which was cumbersome before its implementation, especially prior to Rails 3.
  - It allows users to define gem dependencies in a Gemfile.

- **Performance Issues:** 
  - Bundler's initial dependency resolution was slow; installation times could reach up to 18 seconds due to how it fetched specifications from RubyGems.
  - Bugs and the need for multiple API calls contributed to this slowness.

- **The Need for API Improvements:** 
  - The spike in API usage highlighted scalability issues with RubyGems.org, culminating in its downtime on October 17, 2012, which resulted from heavy load from Bundler API requests.
  - The bundler API's implementation soon required a redesign to manage traffic effectively.

- **Creation of Bundler API Service:** 
  - A community-driven initiative led to the development of the Bundler API, separating it from RubyGems.org to reduce load and improve performance.
  - The prototype was established in under a week and provided a dedicated service to enhance response time and reliability.

- **Performance Enhancements:** 
  - The new syncing code significantly improved update times from 16 minutes to 2-3 minutes through the use of a consumer thread pool.
  - Future directions included implementing a webhook mechanism for instant updates when new gems are pushed.

- **Monitoring and Open Source Contributions:** 
  - Continuous improvements were tracked using logging and alerts for any service interruptions, showcasing a commitment to open-source principles. 
  - The ongoing collaboration within the community plays a vital role in the Bundler API's evolution.

- **Future Prototypes and Features:** 
  - The talk concluded with a preview of a replay service intended to test performance against real-world traffic without impacting service stability. 
  - Insights gathered would help optimize the infrastructure and provide a feedback loop for further enhancements.

**Conclusion:**
- The session emphasizes the importance of community involvement in technology evolution and the necessity for performance-focused solutions in software development.
- Lee encouraged attendees to consider further optimizations and improvements in the Bundler API while highlighting the efficacy of collaboration within the Ruby community.

Cancel