Suggest modification to this talk

Title

Description

RailsConf 2019 - The 30-Month Migration by Glenn Vanderburg
_______________________________________________________________________________________________
Cloud 66 - Pain Free Rails Deployments
Cloud 66 for Rails acts like your in-house DevOps team to build, deploy and maintain your Rails applications on any cloud or server.

Get $100 Cloud 66 Free Credits with the code: RailsConf-19
($100 Cloud 66 Free Credits, for the new user only, valid till 31st December 2019)

Link to the website: https://cloud66.com/rails?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
Link to sign up: https://app.cloud66.com/users/sign_in?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
_______________________________________________________________________________________________
This talk describes how our team made deep changes to the data model of our production system over a period of 2.5 years.

Changing your data model is hard. Taking care of existing data requires caution. Exploring and testing possible solutions can be slow. Your new data model may require data completeness or correctness that hasn't been enforced for the existing data.

To manage the risk and minimize disruption to the product roadmap, we broke the effort into four stages, each with its own distinct challenges. I'll describe our rationale, process ... and the lessons we learned along the way.

Date

Summarized using AI?

If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.

Show "Summarized using AI" badge on summary page

Summary

Markdown supported

In this video, titled "The 30-Month Migration," Glenn Vanderburg presents a detailed case study on a significant transition in the data model of a production system over the course of 30 months. The talk is centered on the complexities and challenges associated with changing a data model, emphasizing the importance of managing existing data and minimizing disruptions. The effort was divided into four distinct stages, each confronting unique business-driven challenges, while allowing the team to resume feature development between these stages.

**Key Points Discussed:**
- **Challenges of Changing Data Models:** Changing a data model is inherently difficult due to the need for data integrity and the requirement for correctness in existing data.
- **The Four Stages of Migration:** The migration process was segmented into four distinct stages:
  - **Stage 1:** The initial realization that data was spread across two databases (Postgres and Neo4j) led to a decision to migrate data from Neo4j to Postgres while maintaining strict data integrity through well-defined schema and constraints.
  - **Stage 2:** This stage focused on refining the primary key strategies and cleaning up the database schema while managing a balance between adding features and fixing existing problems.
  - **Stage 3:** The goal was to eliminate shared contacts to improve data privacy and reduce complexity. Migrating contacts was done realtor-by-realtor to mitigate risks.
  - **Stage 4:** The final stage looked to remove the unnecessary joint table, streamlining the data structure further using triggers for maintaining consistency in data relationships.
- **Technical Strategies and Solutions:** Throughout the migration, the team applied different strategies relevant to their changing requirements, emphasizing the reliance on database features such as triggers, foreign keys, and migration helpers to ensure data integrity and consistency.

**Significant Examples:**
- Vanderburg highlighted the initial mistakes made, particularly the decision to build a social graph, leading to data management complications. He illustrated specific data migration challenges with examples from their structure in Postgres and Neo4j, showcasing the necessary migration scripts and schema design changes made.

**Conclusions and Takeaways:**
- The gradual, iterative approach taken allowed the team to continue feature development despite the ongoing migration work.
- The necessity for flexibility in technical solutions and database utilization was emphasized, advocating for an incremental strategy to manage risk effectively.
- Vanderburg strongly recommended avoiding UUIDs for primary keys in certain situations, suggesting that integer primary keys are preferable unless specific conditions dictate otherwise.
- The talk concluded by emphasizing that careful planning, execution, and leveraging database strengths can facilitate substantial changes, even when risks are involved.

Cancel