Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
RubyConf 2019 - How to lose 50 Million Records in 5 minutes by Jon Druse Join me in re-living the worst catastrophe of my more than a decade long career as a developer. Enough time has passed that I can laugh about it now and hopefully you will too while being inspired to stop cutting corners. It’s like a game of Clue and all the different parts of the system are characters. Which one was the killer? Spoiler alert, it was me, in the office with the keyboard. #confreaks #rubyconf2019
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
In his talk titled *How to Lose 50 Million Records in 5 Minutes* at RubyConf 2019, Jon Druse recounts a catastrophic experience from his decade-long career as a developer, emphasizing the dangers of cutting corners in software development. He illustrates the complexity of managing data in real estate and shares the journey of building a system named Forge to handle extraction, transformation, and loading (ETL) of over 60 million records sourced from various Multiple Listing Services (MLS). Key Points Discussed: - **Complex System Environment**: Jon explains the diverse data sources and the intricacies of managing them, including handling hundreds of APIs, slow data feeds, and varied data formats. - **The Forge System**: He describes the creation of Forge to streamline the ETL process, highlighting initial missteps such as not maintaining a raw data copy and the lack of a proper staging environment. - **Impact of Poor Decisions**: Critical mistakes made in the development process, including ignoring tests and legacy system issues, lead to a situation where the Elasticsearch cluster experienced catastrophic failure, resulting in the loss of 50 million records after an upgrade attempt. - **Learning from Failure**: Jon advocates for a calm and systematic approach to handling crises. He stresses the importance of taking ownership of projects, adhering to best practices, and not ignoring underlying issues that might cause problems later. - **Preventative Measures**: After the disaster, Jon outlines new strategies implemented in their systems to prevent future occurrences, including better data storage practices, proactive monitoring, and the implementation of redundancy. Significant Examples: - Jon uses the example of a disastrous upgrade to Elasticsearch, which led to the erroneous indexing of records and the eventual loss of data, as a cautionary tale. Conclusions and Takeaways: - Handle crises calmly and thoroughly investigate issues rather than applying quick fixes. - Prioritize creating robust backup strategies and test thoroughly before implementing changes. - Emphasize the need for consistent adherence to development processes, regardless of the perceived size of a change. Overall, Jon’s story serves as a vital lesson on the consequences of engaging in non-compliant practices in software development, encouraging attendees to learn from mistakes to fortify their own systems against similar failures in the future.
Suggest modifications
Cancel
