Ruby Video
Talks
Speakers
Events
Topics
Leaderboard
Sign in
Talks
Speakers
Events
Topics
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
RubyConf 2019 - How to lose 50 Million Records in 5 minutes by Jon Druse Join me in re-living the worst catastrophe of my more than a decade long career as a developer. Enough time has passed that I can laugh about it now and hopefully you will too while being inspired to stop cutting corners. It’s like a game of Clue and all the different parts of the system are characters. Which one was the killer? Spoiler alert, it was me, in the office with the keyboard. #confreaks #rubyconf2019
Date
Summary
Markdown supported
In his talk titled *How to Lose 50 Million Records in 5 Minutes* at RubyConf 2019, Jon Druse recounts a catastrophic experience from his decade-long career as a developer, emphasizing the dangers of cutting corners in software development. He illustrates the complexity of managing data in real estate and shares the journey of building a system named Forge to handle extraction, transformation, and loading (ETL) of over 60 million records sourced from various Multiple Listing Services (MLS). Key Points Discussed: - **Complex System Environment**: Jon explains the diverse data sources and the intricacies of managing them, including handling hundreds of APIs, slow data feeds, and varied data formats. - **The Forge System**: He describes the creation of Forge to streamline the ETL process, highlighting initial missteps such as not maintaining a raw data copy and the lack of a proper staging environment. - **Impact of Poor Decisions**: Critical mistakes made in the development process, including ignoring tests and legacy system issues, lead to a situation where the Elasticsearch cluster experienced catastrophic failure, resulting in the loss of 50 million records after an upgrade attempt. - **Learning from Failure**: Jon advocates for a calm and systematic approach to handling crises. He stresses the importance of taking ownership of projects, adhering to best practices, and not ignoring underlying issues that might cause problems later. - **Preventative Measures**: After the disaster, Jon outlines new strategies implemented in their systems to prevent future occurrences, including better data storage practices, proactive monitoring, and the implementation of redundancy. Significant Examples: - Jon uses the example of a disastrous upgrade to Elasticsearch, which led to the erroneous indexing of records and the eventual loss of data, as a cautionary tale. Conclusions and Takeaways: - Handle crises calmly and thoroughly investigate issues rather than applying quick fixes. - Prioritize creating robust backup strategies and test thoroughly before implementing changes. - Emphasize the need for consistent adherence to development processes, regardless of the perceived size of a change. Overall, Jon’s story serves as a vital lesson on the consequences of engaging in non-compliant practices in software development, encouraging attendees to learn from mistakes to fortify their own systems against similar failures in the future.
Suggest modifications
Cancel