Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
Parallel Ruby: Managing the Memory Monster by Kevin Miller At Flexport, we process a lot of data. One sunny day, we decided to switch from a swarm of single-threaded Ruby processes to a wonderful new threadpool. Threads have way less overhead than processes, after all, so we could run far more. Fast forward a couple of hours and everything is on fire. Let’s talk about what went wrong, why it was the Ruby garbage collector's fault, and we did about it. #confreaks #rubyconf2019
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
In the talk "Parallel Ruby: Managing the Memory Monster," Kevin Miller discusses the challenges of memory management when transitioning from single-threaded to multi-threaded Ruby applications. This transition was made at Flexport to better manage the high number of IO-bound background jobs, which peaked at around 20,000 jobs per minute. The presentation identifies how overloaded memory can impact performance, specifically focusing on the implications of Ruby's garbage collection process and memory fragmentation. Key points discussed include: - **The Transition to Multi-threading**: Flexport moved from a single-process, single-threaded setup to a threadpool to improve resource utilization, despite initial successes, they soon encountered severe performance degradation. - **Performance Challenges**: As job runtimes increased over time after deployments, it was crucial to understand the memory patterns. It was found that although the memory usage increased with garbage collector activity, there was a consistent upward trend indicating deeper issues. - **Memory Management in Ruby**: Miller explained how Ruby manages memory with a Ruby heap, contrasting it with traditional C memory management. The Ruby heap's organization into slots contrasted with the less structured C heap, leading to memory fragmentation. - **Impact of Parallelization**: Multiple threads exacerbate memory fragmentation, leading to inefficient memory use and potential leaks in multi-threaded applications. - **Solutions Implemented**: Miller introduced strategies for mitigating memory issues, including setting the `MALLOC_ARENA_MAX` environment variable to unify thread memory allocation and adopting alternative memory allocators like jemalloc, which reportedly improved memory efficiency significantly. To conclude, Miller emphasizes best practices such as continuous deployment that can help quickly resolve performance issues. Notably, employing specific configurations can lead to improved memory management and performance in Ruby applications, especially those running multiple threads.
Suggest modifications
Cancel
