Search
Juanito Fatas
Data Migration With Confidence
Edit
Summarized using AI

Data Migration With Confidence

Juanito Fatas • June 22, 2017 • Singapore • Talk

In the presentation "Data Migration With Confidence" at the Red Dot Ruby Conference 2017, Juanito Fatas, a Software Engineer at Cookpad, discusses the complexities and strategies involved in migrating data to a new system, particularly when transitioning from non-Rails platforms to Rails applications. He highlights the importance of data migration as a critical component of software development and outlines several steps and considerations to ensure a successful transition.

Key points discussed include:
- Understanding Data Migration: Juanito differentiates between schema migration and data migration, emphasizing that data migration involves transferring data from one system to another, such as moving legacy data to a new Rails-based platform.
- Preparing for Migration: Before initiating the migration, it’s essential to gather the data effectively, either through an API or a data dump. He shares his experience in managing large datasets efficiently without overwhelming the database.
- Handling Large Datasets: Juanito explains the challenges of migrating large volumes of data, such as images and records, while ensuring the site remains operational. He details strategies for breaking down the data into manageable chunks and incorporating delays to avoid database saturation.
- Data Integrity and Testing: Emphasizing the need to maintain data integrity, he discusses wrapping migration processes in transactions and the significance of testing the migration code to resolve potential issues proactively.
- Performance Considerations: The speaker provides tips for enhancing the performance of migration code, such as preloading associations, managing workers efficiently, and optimizing database interactions.
- Real-World Examples: Juanito shares practical anecdotes, including the migration of millions of records and how he tackled the complexities surrounding user passwords and image handling during migration.
- Post-Migration Steps: He highlights the importance of conducting thorough post-migration checks, such as updating counter caches and ensuring accurate data representation before fully launching the new system.

In conclusion, Fatas stresses that maintaining clarity in your migration code and prioritizing data accuracy over speed are crucial. He advocates for using small, manageable code structures to simplify future migrations and ensure reliability in the processes. The session ends with an emphasis on thorough testing and continuous monitoring throughout the migration to uphold a high standard of data integrity and performance.

Data Migration With Confidence
Juanito Fatas • June 22, 2017 • Singapore • Talk

Speaker: Juanito Fatas, Software Engineer, Cookpad

Finally convinced your new client to switch from X to Rails? Did your company acquire a non-rails site? Then you probably need to migrate their data to your existing system. In this talk, I will share some false starts, lessons, tips, optimisations and decisions from a recent data migration I performed. How to migrate large amount of photos and records? What tools will you need and how to test the data migration. What do you need to do before & after the data migration. What I tried and how I migrated large amounts of data while kept the site up and running.

Speaker's Bio

Juanito is a developer at Cookpad, based in Tokyo, Japan. He spends most of his time programming in Ruby, choosing which emoji to use, and seeking for good ramen.

Event Page: http://www.reddotrubyconf.com/

Produced by Engineers.SG

Help us caption & translate this video!

http://amara.org/v/8HYR/

Red Dot Ruby Conference 2017

00:00:04.800 hello everyone my name is bonito photos
00:00:10.270 and I'm working at cookware ramen specialist and I have a Spanish name but
00:00:19.540 actually I'm from Taiwan but live in Tokyo so recently I became a salaryman
00:00:26.619 so if you can find me in this picture I can give you a special present after the
00:00:32.379 presentation so I work at a crooked global team and we have another team
00:00:40.300 Japan team's biggest monoliths of the world and I work for a global team and
00:00:46.870 this slide is made in Japan so it's trustworthy you can rely on the
00:00:53.079 content and the other day I was preparing the slides and I pull this
00:01:00.760 slice to you know think about local jobs then a girl behind me just asked me what
00:01:07.479 is your joke then I told her I don't know I need to think about my joke so I
00:01:13.030 still don't figure my joke so I pull the slide here so today I'm here to talk
00:01:21.610 about data migration and so what is a data migration so in Wales we do schema
00:01:30.040 migration and the other one is data migration but the normal migration we
00:01:36.970 talk about is schema migration which will change the database schemas over time while the data migration is you
00:01:44.470 translate data from 68 to system B so and there are two types two types of
00:01:52.450 data migration tool why is existing data migration in salt for example stripe
00:01:57.850 they need to migrate earlier data in the system to a new model so this is
00:02:04.180 existing your migration you can check out this talk and today I'm going to talk about you my guerrilla external
00:02:11.200 data to your existing system for example you buy a company or you need to reload the site for trying
00:02:18.420 so this is my talk today and so first
00:02:23.760 why we need to do they have migration for example you convince your client to
00:02:31.140 switch from the great PHP C website to Rails or other you know you have a new
00:02:40.409 panel joining your company for example you have to your company share the same
00:02:45.780 value and they join your company so you need to migrate it out to your website so you need to do some data migration
00:02:53.159 and there's a single goal is to get all data to our system but is very hard to
00:02:59.939 achieve map saying this morning so the single goal is to get all the data we
00:03:07.319 only need to do these four things and this is very simple so the first thing to do is to get the actual data you need
00:03:15.120 to migrate and there are two ways to gather data first way to if you are
00:03:20.669 provided can provide you API so you can actually solar data or
00:03:26.280 you can just get the data dump from the database and Cookeville we already did
00:03:33.030 many migrations so if they have developed resource to build an API we
00:03:38.669 already have a generic migration code to make all the things happen automatically so today I'm talking about the migration
00:03:47.250 for data dump which is impossible to automate so let's see how to do it so we
00:03:55.530 can start with sync already tossed as easy as you just run rails coming and
00:04:02.310 hopefully everything will be migrated to your system and the Lemaitre you can
00:04:11.069 just put whatever you need to microwave and start to write a code to make it happen but first we need to import a
00:04:19.829 data to your system so you my model can connect to the database you want to
00:04:25.289 migrate and it's as simple as one comment you just import to my sequel but in
00:04:33.149 cookware we support currently is about 62 countries with 13 million users so
00:04:39.479 our production site has a lot of users 24/7 across every time zone so we cannot
00:04:46.860 just import psycho Dom because the database will be saturated is too fast so I need to add some delay to the
00:04:55.050 psycho dump so I'm thinking just add some sleep statement before insert it is
00:05:02.849 true it is but the synchro dump file is very huge I try all the editors below
00:05:11.279 and all the editors doesn't work I have
00:05:17.069 to use this one call hex friends you can actually edit a few gigabyte files but
00:05:22.919 still not perfect because I'm a made a mistake so instead I write a single rupee program which used this amazing
00:05:31.919 feature is called lazy that every time you only need to put all 2000 lines into
00:05:41.279 memory so it's a fast and very easy to use and one of the friends say if you
00:05:49.529 want to get better at Ruby programming you just read about a new level and read it again so yeah so this is how I edit
00:05:59.399 delay to the psycho dump and so you can just connect the database through an
00:06:05.729 environment variable and you sell a environment variable accordingly for
00:06:11.429 development and staging and production then you can start to model in your
00:06:16.499 database so the data you get from the provider is all kinds of formats and you
00:06:23.429 just need to map them to your 16 cistern in the case of cookies user and recipes
00:06:29.399 and other things and with all these five methods from rails basically you can
00:06:35.399 model anything just these five methods is amazing and so let's see example to
00:06:42.089 map the data to our current system so suppose recipe has many steps and sometimes the data is
00:06:51.120 as easy you just specify the foreign key and create this one line of code and it
00:06:57.330 automatically works but sometimes is as hard as the association's stores plan
00:07:06.060 HTML in the recipes table so you need to write some number give a parser to pass
00:07:12.210 all the HTML into the Association again so it is also not so hard you just need
00:07:19.530 to write a parser that make all the data aligned with your current associations
00:07:25.949 so modeling the database and we can also set up the test suite for the migration
00:07:33.560 for example you just need to tell your test database to run against the
00:07:39.210 different database and you just require all the files not being read also loaded
00:07:45.120 and in the test you just require this special helper and you skip the test on
00:07:52.289 the CI because you may not want to import your confidential valuable data
00:07:58.050 to leave CI service but you may ask why do we need test because the migration
00:08:04.349 code only used once because we we can write better code sort of boring test or
00:08:09.719 we can use TDD to get things done so now you get a test so you can do remodeling
00:08:17.219 and write tests and repeat this process so you can model all the data of only
00:08:23.340 provider into your system so you just sold a PHP site will not work like rails
00:08:29.009 application then you can start to a real migration and for the migration you who
00:08:35.969 you need to create a record or update records you use all the method I will raise exception like safe ban or update
00:08:43.380 bed or create ban because we want to fail fast to find all the errors that could be happen before a real migration
00:08:51.209 happens so for example we can look at how to migrate recipes
00:08:56.730 so I have this structure that I made a lot of migrators and each migrator in
00:09:04.950 charge of my guerrilla entities so for example to my grave recipe you just need
00:09:12.690 to write a Migrator for recipe and in the recipe model you just tell what
00:09:21.000 attributes should be migrated and in this migraineurs class
00:09:27.960 you just find final recipe from the
00:09:34.040 provider data and create a recipe in your current system and just update all
00:09:40.860 the attributes then it can it will migrate to your system is very simple
00:09:47.760 and you just keep implementing this migrators and they all respond to the
00:09:53.400 same interface so I always keep my color simple and stupid because I don't know
00:09:59.970 how to do later programming so and you just keep explore migrators then your
00:10:07.470 migration is some but I want to talk about how you ensure data integrity so
00:10:14.760 the first thing you can do is to wrap all your migration code in transaction
00:10:20.070 you also make your call it'll be faster because in transaction there will be less coming to your database
00:10:26.340 I also you need to make your caller ID more potent it's very hard to pronounce
00:10:33.570 and someone put it in your tweets but you do the same thing but produce the
00:10:40.080 same result because we want to run the migration many times but produce the same result
00:10:45.780 it's like FX equal to FX so basically you need to have something called
00:10:52.290 observe is when the record is it you update when it recognizes you insert in
00:11:00.720 my sequel or Postgres secret you have this two sequel that can do absurd or
00:11:06.420 you can use this gem but I keep it simple just you is simple REST API to implement it
00:11:14.060 beautiful we first want to make it right and make it fast later so this is absurd implementing
00:11:21.920 simple rules and then our accuracy how can we ensure data accuracy first I just
00:11:30.600 do some manual trick for a few data I migrated the this doesn't scale
00:11:36.720 so I think about how to automate all the tricks so I can have more time to you
00:11:42.570 know enjoy I don't need to work so for example you let's see how to check user
00:11:50.760 with most recipes in some simple Ruby code first you just loop all the users
00:11:58.920 has a lot of recipes and pass them to a user track object and in this user check
00:12:06.209 object you just check about the recipes and the jars recipe then the recipe
00:12:15.150 checker it looks like this so I just check if the recipe for a provider I'm
00:12:21.720 going to migrate the count is in line with the recipes by grady to prepare or
00:12:28.740 not so and follow geoff part you just change the method to use draft you see
00:12:36.930 these two class didn't change much this i want to be simple so and the checker
00:12:46.020 just gives you a log message to recall the information so you can implement
00:12:53.850 your check like this in some simple movie class and you just keep adding more checkers because for example you
00:13:00.690 need to check about my grade follows or comments or other entities so you use
00:13:07.620 many small objects to compose what do you want to do and i believe everyone's
00:13:15.390 code base you have every everywhere she has a lot of objects but it will become
00:13:21.779 messy very soon so i coming to check about dry Abbey or Roma beats for better object design or trap
00:13:29.560 laser so now we can make our coat a big
00:13:36.190 one job because now we make sure it already works and for how many workers
00:13:42.490 for how many CPU cause then you can have how many workers and you design your job
00:13:48.640 to queuing a different cue so you can
00:13:54.720 distinguish from the regular jobs and the job is the job is just you just call
00:14:03.490 another class that will microwave a record and you migrate a recurring this
00:14:12.160 class so and in the deck on job you need
00:14:17.470 to lock every unexpected arrow so you just in the bass drop add a
00:14:23.440 logger and let's queue every arrow every
00:14:28.540 possible arrow then you can fix it but I'm not sure if this is good you can check out the next talk to handle the
00:14:36.340 arrows better and you need to run your
00:14:42.630 billing job against all your data to be migrated then you can find out all the
00:14:48.280 arrows and you can fix them before the real migration and you will also need
00:14:55.510 some tools during the migration for example retry mechanism and in my sequel
00:15:01.930 if you have a foreign key or constraint when you do an insert or update you will
00:15:08.680 have locks and this kind of and because you have many workers working to migrate
00:15:15.580 data and they may try to actually send record so the lock will result in my
00:15:22.720 sequel dialog and in rails you can just rescue from this exception and
00:15:30.420 automatically that if we try after like two minutes and
00:15:35.820 so after we try we will work or you will
00:15:41.190 just retry again but sometimes and you need to make sure what you want to retry
00:15:48.360 so the first time you don't need to want to rescue everything just see how the arrow is and to understand if this need
00:15:56.640 to be automatically try or not and rails provided to API we try on in this car
00:16:03.330 you can do anything you can imagine you can retry later or just discolored
00:16:10.590 exception or retry is potentially longer is only two ABI and you have everything
00:16:18.230 so but sometimes you cannot automatically to try and you need to
00:16:23.580 look at the arrow in the fail view and in rescue you can implement a simple
00:16:30.870 retried object that can retry our arrow
00:16:36.210 found from a design later q and this
00:16:42.560 retry rescue object just fine hourly failures and we in keulen and remove
00:16:50.940 length so you can retry the arrows in the two you want to retry so in
00:16:58.530 migration you also need to see some status reporting because you want to
00:17:04.110 know how much data you still need to migrate and you can implement a simple
00:17:12.800 Ruby to us called progress and you just loop through all the models and call the
00:17:20.430 progress method only model and each progress method only model is just check
00:17:27.810 how many did how you need to migrate and divided by total data you need to my
00:17:34.500 grave then you can know the progress and you can just wrap another loop be swift
00:17:42.150 to make it report every minute so you can know the migration progress every
00:17:49.290 minute it and you also need to monitor the CPU usage and in our company we use graph
00:17:57.520 Anna I'm not sure if you heard about but this is a good service to monitor your
00:18:04.530 CPU usage or performance and requests okay so the next I want to talk about
00:18:11.169 performance how to make your migration coal run faster but first performance is
00:18:17.380 a ready halt because I spend so much time and every chance you make to the
00:18:25.720 migration code you need to run through every record again because you need to make sure you works first then you can
00:18:32.440 guarantee it is faster so some performance tip I found is you can
00:18:40.840 preload the associations or you can minimize the scope of transaction and you can also
00:18:48.250 kill the transaction isolation levels to make a faster and you can also avoid
00:18:56.250 unnecessary codecs for example you create a user or recipe and you can just
00:19:03.669 skip the moderation because it is finally migration you know it doesn't
00:19:08.860 need to be moderate or you can use no touching so the ko will not touch the Association
00:19:19.150 so you will be faster because you can touch them all after the migration and
00:19:24.659 another tip is to process much more records in one single job and it is also
00:19:33.690 very easy you just use each slice and you process 100 users in one job and you
00:19:41.980 will be faster and so if you process 100 records and some of the records may be
00:19:49.059 already created and you can cash them in memory or later on I try to cash them in
00:19:55.570 Redis so it's even faster and to make it faster you can also migrate the
00:20:03.490 important things first for example only my grader first want to use 10,000 users
00:20:10.390 with most recipes or but so my guerrilla important things first make your
00:20:16.330 migration : faster but later on I run into a problem ease IO bond so ruby is
00:20:24.580 actually super fast but my database has reached the limit of IO you can perform
00:20:31.780 so I'll oh I need to fill up the database or I find out just decrease the
00:20:38.800 workers actually make the IO Bunga better because if you have too many
00:20:46.179 workers working on the same database your i/o will soon to fill up so you
00:20:52.690 instead you decrease the worker actually can do more than more workers or you can
00:20:58.510 do something like pack ether or pack up set to insert or absurd many records in
00:21:07.510 one go so every time you make it fast
00:21:13.059 you need to run the whole migration again and when you run a migration you
00:21:20.410 need to look out your CPU usage to keep it and max 75% from my experience so
00:21:28.480 your site one went down and after migration you can update all the things
00:21:36.100 you need to update like counter cache or statistics or you need to touch some
00:21:42.280 associations and you can also do a redirect because you want to read the
00:21:48.309 real old data from the outside to your new site and first we do something like
00:21:55.300 we generate a redirect tables and we hand to the provider and they design a
00:22:00.850 redirect programs and their server start to redirect but this comes with a cost
00:22:07.030 because the migration provider doesn't always have developers so recently we
00:22:13.990 are working on my reservation service open-source to do these simple things
00:22:21.990 that doesn't require developers from the provider okay so I'm going to share some
00:22:30.220 stories about data migrations I've done for the email you better remove the
00:22:38.169 duplicate emails before the migration or remove the invalid emails before the
00:22:44.440 migration because this far more expensive to handle and after you
00:22:50.799 migrate data to your site and you need to make sure you can taste all the
00:22:56.169 emails because it will cost you a lot of troubles and another story is how to get
00:23:03.730 the site dump so one of the provider the other site is 100 gigabytes ec2 and ec2
00:23:11.919 has a bandwidth limit so if you do a SCP or a sink or other things you will take
00:23:19.690 days that if nothing fell we sing the days or you can you know buy a more
00:23:27.040 expensive easy to machine so the solution was just used pho to deliver
00:23:33.870 encrypted prefix to Tokyo is actually faster than SVP and another stories
00:23:44.790 migrated millions of records so when I need to migrate millions of records I
00:23:50.679 look into all these solutions so the first one is active lurker with transaction or back insert or absurd or
00:23:59.130 faster one active record import or even the fastest load data in file is my
00:24:06.490 sequel command you can load data in file lattice so fast but it still takes like
00:24:13.390 weeks or one month to migrate all these millions of datas so instead I think we
00:24:21.940 just need to run this migration in very low priority job and when the user signs
00:24:30.320 we actually migrate their data in a high priority queue so in this way I don't
00:24:35.570 need to migrate all the millions of records to our system which is slowly
00:24:41.120 migrating and if they do signing we migrate layer bookmarks or some other
00:24:48.740 important data so it's simple as just check if user need to migrate then you
00:24:55.100 run this migration bookmarks so you don't need to migrate millions of data beforehand you just need to migrate and
00:25:02.000 slowly and when user really needs you migrating a high priority to another
00:25:10.730 stories migrate 100k photos so first
00:25:15.890 let's see how our imagery work so every time when we upload an image we will
00:25:24.140 generate a unique image ID then use this
00:25:29.510 ID a CDN to access our photo so but this
00:25:36.200 this process takes like five second because all the images from the external
00:25:42.860 sites you need to you know open a HTTP connection and upload to your server the
00:25:49.610 generator unique hash so this will slow down your migration in very great scale
00:25:57.200 so instead I desire way that it always
00:26:02.810 produce the same hash but yet it's still unique so in the migration I only need
00:26:10.340 to set this previously designed hash during a migration so I don't need to
00:26:16.280 upload a image during a migration I can upload our images beforehand then I make
00:26:22.670 benchmark how long to finish only image upload so it's like 10 days and I just
00:26:29.900 microwave all the data ten days before in low priority because in the reality
00:26:35.470 99% of the photos will not change at all so you can just do the things beforehand
00:26:43.580 so it won't slow down your migration and another story is to migrate a user
00:26:49.960 password to secure authentication because layer authentication may be like
00:26:57.980 use md5 or something simple so first you need to figure out what what is the
00:27:04.999 algorithm they use to encrypt the password and when the migrated user
00:27:10.999 signing and they enter the password and you will fail in your system because
00:27:17.029 their password is encrypted in md5 or something not send us your current
00:27:24.499 systems password encryption so when the when you fail you fall back to the
00:27:29.779 legacy authentication and in vitro it looks like this and when you fail you just use the
00:27:38.480 legacy authentication to check if they enter the correct password or not and
00:27:43.489 when they enter the correct password you just set this password through your
00:27:48.759 existing secure password screen so you can make make their password secure
00:27:57.169 again by doing this simple change and for the future of recent migration I
00:28:05.600 think if I ever need to talk again I will look into my guerrilla data in a
00:28:11.570 ghost table so I just write into another table then when it is ready I swap the
00:28:18.320 table or I were looking to make my migration called more generic so I only
00:28:25.879 need to model all the data base then I can just run the migration and everything will work I can have more
00:28:32.749 holidays so some takeaways rails provides shop tools thanks to the dress
00:28:39.409 coding and you use small objects to make your code more reliable and maintainable
00:28:45.739 and my friends say abstraction is the god of programming and you always
00:28:52.369 remember schedule is more important than fast and accuracy is more important than schedule
00:28:59.120 and then our migration sands up but if you keep it simple it can make it easy
00:29:06.110 to the simplest thing with my previous spot to instance eight and thank you and
00:29:13.010 enjoy your tea breaks after my top thank you all stop
00:29:23.120 you have any questions about the immigration for joining yes hi have you
00:29:31.940 ever run into a situation where a very important data that is not valid under the new system yes so we run a migration
00:29:39.740 call you will find a lot of records okay now migrate to your system because you have holy validations or their fear
00:29:48.169 is too long this kind of things so you can either increase either soft your
00:29:55.580 validation or just update them we saw
00:30:00.980 running these validations like update color and then you're just keeping all
00:30:06.470 the data in the database yes until with it later somehow and do you have any
00:30:12.519 strategy to deal with it afterwards so basically before the migration I will
00:30:19.309 run through all the data make sure everything works so I didn't have much
00:30:25.580 to do after the migration okay thank hi
00:30:34.330 okay we got to so I think I noticed a
00:30:39.409 part there where you were migrating data and then conditionally migrating some
00:30:44.450 more data based upon the user like the bookmarks if the users like wanted that with that all beforehand or was that
00:30:53.059 based upon input like live from the user so a user finally logged in and then you
00:30:58.880 in only then you lazily migrate a different part okay sorry just about and
00:31:03.980 if so do you see yourself moving more towards that style of my raishin of like kind of lazily migrating
00:31:10.639 things along so in our system we have some important data like recipes so we
00:31:18.740 only migrate important things first and things not so important like bookmarks
00:31:24.679 and there are many of them so we make really kind of large data in lazy
00:31:31.399 fashion yeah that's how I try to did it
00:31:38.139 thank you we won't question you hey um
00:31:44.360 so how do you how do you manage your foreign keys if you go if you've got
00:31:50.629 users and recipes and you're uploading them in parallels you were kind of showing what if you're trying to put a
00:31:57.740 recipe and that doesn't yet have a user that's been created ah so when I migrate a user know
00:32:04.610 migrated recipe I will actually migrate user associated first then migrate your
00:32:10.549 recipes it starts going through parallel
00:32:17.090 ah thread hmm how does the recipe migrator know about
00:32:23.690 the user if it's come from different data or doesn't so every so even it's
00:32:31.070 the job using different workers every worker will need to check first if the
00:32:36.679 recipe has a user migrated to our system or not so we always migrator user first
00:32:42.649 so it's kind of more slow but integrity
00:32:48.169 is secure thank you thank you yeah
00:32:56.070 so do I understand correctly that when you migrate from the outside to the new side the news there is no downtime there
00:33:03.820 is like they enable your site immediately and your data is still not
00:33:09.489 there you your you will be loading in right or there is some scheduled maintenance that the owner of the old
00:33:14.710 site is K in the three days we come back or how does it look like from users perspective thank you
00:33:19.779 I didn't cover lease so when the migration happens we make sure the
00:33:24.929 provider website change to read-only and so everyone cannot signing they can just
00:33:31.450 read and we start a migration uncooked pad like few hours then when we ready we
00:33:38.019 shut down the outside and redirect all the things to our new site it depends on
00:33:45.729 the size the short one is like four hours it can also hours depends on how much
00:33:53.349 data you have more questions are the
00:34:03.460 back
00:34:08.919 so just one also um I don't know database to that so as to migrate the
00:34:15.049 data excuse me could you say again I don't know like database to do we need
00:34:21.409 to actually create I think we of course could be the general and which you can do anyone but are they know like
00:34:28.669 database way to dump vitally important everybody's so your question is how this
00:34:37.309 will be talk to other database no like a attorney for example ms SQL Oracle right
00:34:46.719 they also provide data tons and data transformation ready to maybe for
00:34:53.599 example to report another database so we will consider using those tools on
00:34:59.150 something ah you mean use the database tool to do the migration oh yeah yeah I
00:35:05.450 have been considering doing that but I'm not very good at the database tools so I
00:35:13.910 used to be and he still works so okay thank you I think this Aaron's
00:35:24.589 that sometimes you need to do some transformations or validations that you do in your Ruby code and or maybe the
00:35:31.160 schema your transform yo you're migrating from is not the same thing so you need to do some transformation bits
00:35:36.890 that the database tooling design quite support it depends yeah so yeah I think we can take one
00:35:43.190 more question before I open the break one of them okay
00:35:52.650 just one asked about that the lazy migration so when I use a lot in then
00:35:57.940 only you kill data in a high priority queues in migrated data so what happens if the user never locks in so you don't care about the data anymore ah so I
00:36:05.110 actually has a low priority queue running for everything that would take
00:36:10.810 like few weeks but it's late saying during these few weeks layer things will
00:36:16.210 be migrating in high priority move that low priority queue and move them up to the high priority queue not its
00:36:22.840 duplicated so it's duplicated ok yeah but your migration Cori's there
00:36:28.480 most of them yeah you can run multiple times and it's still the same so no problem ok cool thank you very much
Juanito Fatas
@JuanitoFatas
Explore all talks recorded at Red Dot Ruby Conference 2017
+12