00:00:11.809
close out we have this amazing panel on performance so I'm going to introduce Sam who's the moderator Sam saffron is a
00:00:19.830
co-founder of discourse creator of the mini profiler memory profiler mini meme
00:00:25.019
and mini racer gems he's written extensively about various performance topics on Sam saffron calm Sam loves
00:00:32.610
making sure discourse keeps running fast enjoy hey is everybody still awake okay
00:00:51.629
I'm just going to have the panel introduce themselves and then my I've got a bunch of questions prepared but
00:00:58.590
also we're accepting questions and there's a little chat thingy so huge
00:01:03.780
login and type your question and I'll triage us while stuff is going on and if anything really interesting pops up or
00:01:10.320
uninteresting falafel or vet crashes it's back to traditional way anyway
00:01:16.500
let's start my name is Nate Birkbeck I'm
00:01:21.750
a independent performance consultant I worked on people's rails apps to make
00:01:27.210
them faster and I blogged about it online at speed shop Co I am Raphael
00:01:35.610
Fresa I work at chop I as a production hearing my job day is to make sure that
00:01:42.270
shop virus is mostly in our producers decisions and I also read member of the
00:01:49.170
vehicle team hi I'm Eileen you should tell I work at github
00:01:54.780
my job is sometimes performance but not always but I've given a few talks on
00:02:00.240
performance on speeding up integration tests and other stuff in rails and I'm also on the rails core team with Rafael
00:02:08.899
everybody my name is Richard schneemann I go by Eames on the internet I work for a start-up in San Francisco
00:02:15.560
called Heroku they were kind of up-and-coming some of you might have
00:02:21.320
experienced issues with them in the last couple of minutes here so uh yeah I I do
00:02:29.540
performance work like when customers come with an issue then it's like okay
00:02:35.000
well how can we make this a little bit faster and I'll take a look at their apps I wrote derailed benchmarks I don't
00:02:40.670
know if any of you all have used that one as well as I blog I'm on the
00:02:48.320
blogosphere so yeah that's me thank you cool so I think we should I'd like to
00:02:54.800
get started talking a bit about how to get involved in this whole performance thing so I guess the question to the
00:03:02.270
panel is you know if I haven't done anything publicly before and I haven't done any performance work where where do
00:03:08.630
I start we're going to do this like down
00:03:14.780
the line every time yeah all right so I roll fortune what your arm-wrestle for
00:03:20.239
it yeah we can do Rio all right I lose oh well I just talked though like two
00:03:26.390
seconds ago so you got it so I mean I got started in performance work because I used to work at an e-commerce company
00:03:32.590
and it was clear to me then and it's
00:03:38.420
still clear to me now that fast sites are sites that make money and sites that
00:03:44.299
people enjoy using so I've kind of always had that very like customer
00:03:49.760
focused approach of like you know the the speed of the website is a primary primary component of how your users or
00:03:56.930
customers interact with it so I mean I got interested in it for that reason
00:04:02.989
because I wanted people to buy more stuff on this website and you know it's
00:04:08.989
it once you start digging into it it just like became like it's not often in programming that you can get like these
00:04:16.940
very discreet problems which when you fix them it's like oh I made this 10:01
00:04:23.090
that's really good right like it used to take ten seconds now take one second like that is better you know
00:04:29.960
like if maybe that's just like the one route that someone hits like once every blue moon but like it's faster for that
00:04:37.490
dude now so that's cool so that concrete impact of like performance where it
00:04:43.669
became very addicting to me and then you know you just keep doing it and keep doing more of it I guess and you just
00:04:50.270
sort of start picking up the skills I guess so so I mentioned I work for
00:04:58.430
Heroku one of the things that I do is when a ticket comes in for Ruby someone
00:05:04.550
is using Ruby and it can't be handled by our our support staff it'll get
00:05:10.010
escalated and it'll end up on on my desk and people occasionally will say hey I'm
00:05:16.789
using Heroku and like I'm not getting the performance I want and generally whenever I consider I consider
00:05:22.669
performance issues a bug and like generally when somebody is hitting a bug of any kind on Heroku they're like ah
00:05:28.280
rook who's breaking all of my stuff well so first of all 99% of the time unless it's like right this very moment that's
00:05:34.190
not true and so instead so I mean I do
00:05:40.070
spend a good amount of time helping customers get to the right solution but also a good amount of my job is also proving essentially that it's not that
00:05:48.889
it's it's you know it's not me it's you and so that's that's actually what
00:05:55.880
inspired derailed Dench marks is I was was getting a couple of performance
00:06:01.550
issues that I just could not could not debug on Heroku and this was before we
00:06:07.970
had so we have something called PS exact right now where you can actually SSH into a dyno and get you know some extra
00:06:14.840
metrics and stuff that way but before we could do that the only way to debug a
00:06:20.180
performance issue was to reproduce it and reproduce it locally which also happens to give the benefit of now I can
00:06:30.650
say oh the performance issue also happens locally like not my issue
00:06:37.050
but no and then I got into this great community met a bunch of other people who care about performance and it's just
00:06:42.479
like even even today people will come to me and say hey this I'm experiencing this behavior on my app like why is it
00:06:48.720
happening like this and I was like I don't know that's a really like you know that's like telling Sherlock Holmes
00:06:53.960
somebody I don't know stole a cookie out of the cookie jar and you know it's just
00:06:59.370
like I've I have to dig in and get deeper yeah is this is this something
00:07:07.410
like only advanced developers can do I'll just move sorry
00:07:12.560
it's like because I remember you got you got started like a lot with the performance work just you picked the
00:07:19.160
mini profiler gem back in the day and you worked on this giant commit so so
00:07:24.539
how does it like how do you what made you decide to just I guess get started
00:07:30.240
with this and like get out there and do this kind of work
00:07:35.419
well the first time I got involved with performance was because I crashed a staging database I was trying to insert
00:07:43.470
like a ton of data for the sales team to use for load their demos and I didn't
00:07:48.539
really know I just knew I was like oh well active record knows how to do it I'll just shove data into the database
00:07:54.060
and then I used so much memory that it crashed the MySQL instance of the staging database so I had to like figure
00:08:01.440
out well why why is it slowing is it rails fault is it my fault is it - Wells fault and I used that to figure out ways
00:08:08.639
to insert or update or delete data better so we didn't have a whole memory
00:08:14.400
flow problem so for like the insert record I used like direct MySQL batch
00:08:20.190
insert versus when I was deleting stuff I use delete all instead of destroy all
00:08:25.259
because I just needed to like blow the database away that led to my first talk
00:08:30.780
on active record which is all about really how active record is not it's not it's not active records fault because it
00:08:36.029
can't be like oh by the way you should use this instead and I'm going to do it for you because then your call backs
00:08:41.339
aren't run and like other things happen to your life that make you sad that are not just performance so you don't you
00:08:47.190
know you don't want a tracker to take over for you and then like because of that getting me involved in open source stuff
00:08:52.290
and why your make the mini test I'm not mention sub-recoil filer just
00:08:58.140
there was like this big problem it wasn't like a problem but it was I don't
00:09:03.570
even remember what I what I worked on it was just something that you were like I need help and it was like the one open
00:09:09.750
source project that said I need help across the top of it like in big bold letters and I was like cool this guy's desperate enough to level
00:09:16.380
such as Kobe Rafael so as I started
00:09:27.150
we've performance work as I was working as a consultant in Brazil and one of my
00:09:36.290
one of the things that my job required me to do was to hear other people
00:09:41.670
problems and usually other crises that we had had major performance problems
00:09:48.470
because of any reasons like sometimes was database and time loss front-end
00:09:54.630
performance and nobody my team wanted to work on those problems so I kind of felt
00:10:01.350
my plate and I enjoy it because like Nate said is the kind of problem that you actually see the value being
00:10:09.540
delivered to the users right away because you can see that your machine -
00:10:14.580
so that was when I started I can say
00:10:19.740
that I did not had any kind of contact with this kind of work before doing that
00:10:26.970
so I had to learn how Disney worked for me a lot of different tools and yeah I
00:10:35.460
don't think that you need to be I experienced developer to perform a
00:10:41.940
series you can just know how the system
00:10:47.040
works like let's say that you you're working a rubric application that if you know Ruby well enough to write the
00:10:53.610
application Ruby it's possible to you to also know how to do performance or Ruby
00:10:59.310
application I just I just wanted to add that like a our experienced developer like I don't
00:11:05.550
think you have to be an experienced developer to do performance work but I do think you have to be curious because
00:11:10.850
performance work can involve so many different areas of the stack most of
00:11:18.090
which you probably have no experience with you know if a web request takes
00:11:23.250
five seconds right if that's all the information that we have there's literally like 20 layers right that that
00:11:30.450
are involved in that five seconds it's like the network the application server your application code the Ruby VM the
00:11:40.370
kernel like there's there's there's and there's so many shades in between of all those different layers right so like I
00:11:46.680
found that this work has made my knowledge pool like much I'd like a very
00:11:52.170
wide and shallow knowledge pool now because when you're doing performance work the requirement is known you know
00:11:58.260
it's this thing is slow and I have to make it faster but often times it's like that solution could be anywhere so you
00:12:06.480
just have to be curious and use it to not be intimidated I think oh yes sir so
00:12:13.320
uh one one just generic piece of advice that I've really taken to which when
00:12:19.950
somebody asks me like oh how do I do think this thing well there's a term there's a saying that says writers write
00:12:26.750
if you want to be a writer you should go right like how do I become a writer well I'm right if I want to do performance
00:12:32.400
work well you need to do performance work and just like being generally
00:12:38.100
interested in it if you don't know where to start like you can see what other people are doing or other people are
00:12:43.440
talking about like what tools exist out there like I I got my my even my first
00:12:50.850
what things could possibly even be optimized like I got from like adequate
00:12:56.280
record from a keynote and it was like oh hey there's this thing called like memory allocation when you like create a
00:13:02.310
new object and like if you do that a bunch like that's like bad and there's cases where we're like at the library
00:13:07.950
level maybe we can optimize that a little bit and that doesn't have to be the case for everyone but like I also
00:13:13.320
went to Nate had a front-end workshop and it was like here are screech examples of you know here's a
00:13:19.880
problem and then here's a here's a series of potential solutions and it's just kind of a iterative process of
00:13:27.589
doing the same things like over and over and over and like like he said exploring and going deeper cool
00:13:34.519
the chatroom is actually all alive and full of questions so I can go off my question list to the chat questions one
00:13:44.390
question that I'd like to actually talk about here is is tuning the Ruby GC still needed in 2x
00:13:50.269
I remember one 9x default did not work well with rails a question by Toni and
00:13:56.079
I'll start answering it in kind of a non obvious answer is but if that's the
00:14:03.050
wrong question to be asking because what you should be doing is measuring and if
00:14:08.630
you are measuring and you have some sort of graph that you're trying to make the line go down and you twiddle with
00:14:15.440
something and the line goes down and you know that there's success but if I tell you yeah you need to edit this kind of
00:14:22.490
GC setting and you go and edit it and actually does damage to your performance
00:14:27.860
and you don't know then you're in a very serious situation at this course we do
00:14:34.100
edit some GC settings that the ones that we do edit are all based on feedback
00:14:39.199
that we got from grass but actually showed that changing that setting helped
00:14:44.779
do you regret publishing that blog posts that listed all those GC settings are used because I think I wish I knew the amount of people
00:14:52.399
that have just copy/paste it does I'm sorry I regret I don't know no I don't
00:15:02.930
regret information is good I I don't regret information being out there yes
00:15:08.050
if you brought up something very important about performance that I think a lot of like when you first see a
00:15:13.279
performance problem like your first reaction is I want to fix this but you can't fix anything with performance
00:15:18.860
until you measure it just like you can't fix a bug without actually knowing what's causing the bug you can't just
00:15:23.899
like you could just shove a rescue in your controller you're like sweet I fix the bug but you didn't repeat
00:15:29.710
oh yes it doesn't throw an error anymore but it's still there and so that's the biggest thing with performance is making
00:15:35.200
sure that you have a measurement before and after you fix something because otherwise you don't know you actually fixed it and there was an incident there
00:15:42.010
was one time when Patterson and I were trying to fix a performance issue and we luckily were measuring because the
00:15:48.990
profiler said it was fixed but the benchmarks said it wasn't and so we had
00:15:54.610
just gone with the profiler like speed that that number is no longer high after we benchmarked it and figured out it was
00:16:00.850
still the same speed and slow so that's why you have to measure before and after or else you don't know you actually
00:16:06.880
changed anything for the better any other GC setting recommendation so so on
00:16:14.140
what was that conversation we had earlier about I think the three of us you're talking about we were talking about setting GC the max
00:16:23.050
lot growth limit yeah so there's a I think it's like GC growth it's got the
00:16:29.560
words growth and factor in it oh yeah so it's a it pays to know what changing one
00:16:36.790
of those settings is actually doing like you don't want it just like this is a
00:16:42.490
blog post that's been sitting in my queue for a while I want to go through each GC tuning variable and explain exactly what it does oh that's the
00:16:48.790
important one though one of the important ones that we're talking about is so each Ruby object gets a slot in
00:16:57.130
the Ruby VM this is an R value C structure and the there's three
00:17:06.339
variables I think which controls how this slot the CD slots grow well one is
00:17:14.050
a factor so each time we run out of slots we multiply the amount of thoughts we have by this factor and that's the
00:17:20.559
new amount of slots that we have then there's two very two other variables which can put a max and a minimum on
00:17:27.699
that number so it by default we realized it was one point eight I thought it was less than that so and if I have a
00:17:34.960
thousand heap slots and I decide I need more I will grow the heap to 1,800 and the problem is that most people's
00:17:41.169
applications have a million keeps Lots when they're in a steady state you know after running for six
00:17:48.260
hours and whatever and then you run out of slots at some point major GC happens and now you have one point eight million
00:17:54.710
slots which is probably eight hundred thousand more than you actually needed what you really needed was another hundred thousand slots so setting those
00:18:03.520
can sometimes reduce the amount of free
00:18:08.810
slots that are in the Ruby VM you would know if this is a problem if in for
00:18:15.590
example if you have new relic new relics Ruby VM tab will show you how many used
00:18:21.050
and free slots you have at any given time and if you have these huge numbers of free slots the yellow part of the
00:18:28.760
graph on new relics and maybe you have a problem with this setting and you can also a lot of these things you want to
00:18:35.530
like you don't want to just start optimizing GC until you you yeah you venture marked and you say okay yes this
00:18:42.020
is the thing this is the thing I'm having problems with you could maybe tell if you have this problem if you look at your mess to your memory graph
00:18:48.560
and it's you know it's kind of going up and going up and then just all of the sudden you see this giant wall of memory
00:18:55.430
and you're just like okay well you know what after your servers been running for like whatever six hours or something and
00:19:01.310
changing and changing this value change this growth factor will help with that I've got a recommendation and the actual
00:19:07.610
name of the growth factor the actual name of the GC setting on an article on
00:19:12.800
the dev center for Heroku under if you like search boop memory use Heroku I
00:19:18.230
think it's our our 14 maybe I have a little section on that with a recommendation I think it's like 1.0 one
00:19:24.410
so it grows by like 1% instead of like 80% so yeah I mean if you're looking for
00:19:31.400
specifics discourse is actually open source and anybody can download it and
00:19:37.460
install it and we have an official docker image that controls the whole installation so you can just look at the
00:19:43.940
source there and see what we're using and it's pretty well-documented why I
00:19:49.000
change whatever GC settings I change so there's only one really but I play with
00:19:54.020
which is capping the growth at instead of it going exponentially
00:19:59.280
making it just grow by a hundred a hundred thousand-plus at a time there
00:20:05.220
are more questions here what does great
00:20:10.770
performance mean to you is high RPS fantastic CPU resource utilization sub
00:20:17.040
100 millisecond requests by Danny what is fantastic performance for you I guess
00:20:23.790
yes yes that is fantastic it's when I
00:20:31.620
when I think about performance I think about the end result so a lot of times
00:20:37.230
we like to think that you know the server says that it's able to render stuff to generate stuff in hundred
00:20:44.100
milliseconds everything is good and like we can go home now because our work is done when in fact like we have all of
00:20:51.570
these little JavaScript snippets that we've collected from all these places and we put that we didn't think about
00:20:56.880
client performance at all and every user that goes to US side actually experiences a five-second delay because
00:21:02.670
we didn't think about a CDN we didn't think about deferring our JavaScript we didn't think about minifying javascript
00:21:09.350
possibly and so on so for me like yeah I mean there's this hundred millisecond
00:21:15.330
thing that you know Amazon strive for but there's also you know when when I
00:21:20.580
think about performance I don't just look at the server side when I think of
00:21:27.270
great performance I think of Wes Anderson movies that is good yeah those are those are good so that was a joke
00:21:40.370
Thank You Evan thank you i lo so I focused on the server side of things and
00:21:47.280
and I consider good low request variants and so what that means is that it's even
00:21:55.170
if your average is like oh we're serving requests in like 10 milliseconds but one out of every like you know that means
00:22:01.980
maybe most of your requests are like one millisecond and then one is like 10 seconds or 20 seconds or 30 seconds like
00:22:09.030
that's really high request and that that ends up having a lot of
00:22:15.000
impact I mean yes that one person has to wait for that ten-second thing but if you can also get this like this cueing
00:22:24.000
effect where slow request will end up behind that hour fast request will end
00:22:29.430
up behind that slow request and then it'll feel a lot slower and like a lot of these things like tuning you know GC
00:22:35.820
or like you know modifying their server settings or something like it kind of doesn't matter if if your end code is
00:22:42.540
just slow and so like my big my approach when I end up with when I'm working on
00:22:48.930
the app that is slow is I want to know these slowest end points and that's where I start I mean the slowest end
00:22:54.960
points that have a lot of traffic and then getting getting those down and generally you not only do those become
00:23:01.620
faster it actually speeds up the little bit of the rest of your server as well
00:23:07.550
yeah it's just like queuing theory 101 that it's way easier to load balance something where the response the the
00:23:14.670
time in the system is has low variance than something that has high variance
00:23:23.810
now there are jokes in the chatroom as well if I need to filter them out now
00:23:33.990
there's a good question yeah Laura said do you ever find yourself preemptively over optimizing your code for
00:23:40.710
performance before there is need to consider it yes well it's like
00:23:47.480
performance I think this is kind of why sometimes measurement will go by the wayside is that or improperly like
00:23:54.330
French marking and profiling is like I think I don't know if you guys feel like this but like the the solutions are like
00:24:01.200
such like candy to me like getting to implement you know let's look the latest
00:24:06.390
like little trick or whatever like I love that so like yeah sometimes we'll just do things to do it and and whether
00:24:13.650
or not I really needed to so you know as
00:24:18.780
long as it you're not actually spending that much time doing it it's not really that but when you get like you know 24 hours
00:24:26.919
deep on a yak shave to like you know make whatever the new thing cool hot anything was work with your app it's
00:24:32.529
like that was a waste of time good going and I was like I think you're
00:24:42.429
done oh hi mal request expired yeah I
00:24:53.349
figured out okay so when you start learning like things that you can improve like for in your app performance
00:24:58.539
wise there's you're going to have those just do that every single time because like oh I know it's faster but if you're
00:25:03.999
doing that push that upstream to Rails or to a gem or to Ruby because if you're doing it
00:25:11.499
over and over again chances are someone else is doing it over and over and over again - and so then everyone can benefit
00:25:17.289
from it when you push it upstream to rouse or whatever you know if you do something we had of someone change fine
00:25:23.559
and batches recently on Rails something you could add loaded limits or yeah yeah
00:25:28.659
and so like now you know maybe somebody was doing them their app a lot so now everyone can use it because it's Newton
00:25:33.669
Grails go Thank You Benjamin so you said you
00:25:43.980
said push it up so I guess this is sort of Elsa like there what a thousand open pr's Rail five and like how do i how do
00:25:52.320
I manage yeah right my PR now Interrail five two and my performance sticks into
00:25:59.370
there what can I do to make it more likely that the change that I submitted
00:26:04.980
will be reviewed by the team and actually merged in what what steps can I
00:26:11.190
take a fig there the first thing that you need to do is actually prove that
00:26:17.730
there is a problem in different work so if you are measuring your application and you put that data in the issue or
00:26:24.900
the pull request description is very likely to you to have these PR bases
00:26:32.700
because we know that someone is having that problem because what we try to
00:26:38.400
avoid in rails is premature optimizations like they're we're a bunch of PR frozen strings everywhere
00:26:46.470
different work but we don't want that like we only want to sort of be upon us
00:26:52.710
not to change out the put the baits to be slightly faster but no really
00:26:58.890
so say saying if I overall I cut down on ten allocations in a rare case and it's
00:27:04.560
less likely to be accepted for me really cutting down 50,000 allocations in the general like hi yeah exactly and that is
00:27:12.900
what you're looking for and I'll also say different patches are every single
00:27:19.470
pull request has an overhead even if you're just adding dot freeze to like a string on one line well now suddenly if
00:27:27.150
I'm blaming doing get blame then that you know that might mess up that it's you know it requires overhead from from
00:27:33.330
us I've done I've done a bunch of rails performance stuff in the past a lot of those freezes are my fault I'm sorry but
00:27:42.090
when I did it I submitted I said here's uh here's the benchmarks here's how to reproduce here's the steps I took like
00:27:49.110
and not not only that but not only does that keep me honest it also gives other people eyes and to look
00:27:57.990
at it and then previously I said if you're interested in doing performance stuff then find other people doing
00:28:03.480
performance stuff and hopefully if you hero somebody so and so like yesterday there's a camel got three three times
00:28:10.020
faster right that's that's a thing whoo thank you
00:28:17.040
and and the first thing I did was was be like okay well how I messaged him that
00:28:23.070
and misspelled it horribly and then I think a bunch of people have saved it and I and but in your pull request if
00:28:32.220
you have here's my benchmarks then you you can go and try and and do that I AK
00:28:38.480
and like I've actually had a case where people have done that and not gotten the
00:28:44.670
same results and then and actually it turns out that you know they didn't push
00:28:51.930
the fix or whatever and it's like oh that was actually really helpful to know that not everybody got the same thing so
00:28:58.320
yeah benchmarks cool I have a correction
00:29:04.020
some smiley face huggy face koala saying they're only 700 issues in
00:29:09.060
rails that are open now just apologizing I mean so anyway okay there's a question
00:29:17.490
by Michael should you refactor before trying to optimize a code path about like the tension between fast code and
00:29:25.200
pretty code so where do you where does the panel see themselves in this I guess
00:29:32.480
continuum I'll start a fast code which
00:29:39.690
that's actually two separate questions I think in general you just you know every time I touch something I try to make it
00:29:45.840
better so if I'm touching it for a performance reason I'm also going to try to rewrite it to make it you know more
00:29:52.050
readable or whatever like we just we literally just had a thing on Puma where we have an issue where Puma processes
00:30:01.140
are accepting too many requests and the like conditional and the code
00:30:09.490
that was like causing this behavior it was like really confusing and we actually like reading through it with
00:30:15.789
Evan we're like what does this until less than and like couldn't even like process at all at once so we rewrote it
00:30:23.200
at the same time like the actual change was like just removing and unless from one line but then we rewrote the
00:30:28.840
conditionals we actually understood what the change was doing so I'm gonna do any
00:30:34.059
tension there I think you're kind of getting that like I mean the we don't
00:30:39.340
really do this in Ruby but like when people like in line stuff that you would like in your like unroll or loops or
00:30:45.370
whatever like we don't really do that kind of stuff I don't do you feel I don't feel like we have a tension between pint kinda so like on the other
00:30:52.840
on the other question which was I don't even remember it because it's the one that you just said no about what was
00:30:59.470
that question the tension between pretty code and Fusco's so like knowing a bunch of
00:31:06.429
performance tips like a bunch of micro performance optimizations sometimes that micro performance optimization is not as
00:31:13.270
pretty as what you would normally do in in Ruby like you know recently there's a
00:31:22.090
thing on like don't use case statements there's like a blog post like stop using case statements very large grandiose you
00:31:28.870
know thing and it's like coming from a like performance work it's like a case statements are actually optimized by the
00:31:36.179
by like the the Ruby VM in some cases and like yes it might actually be harder
00:31:42.400
to work with and not as pretty but there there might be case there might be some
00:31:47.740
times when you just want to make that thing faster or it's like oh I can move
00:31:53.039
move this object allocation outside of the loop but it's you know it's not as
00:31:58.510
like it's not as nice it's not as pretty and so I mean I get that but I tend to
00:32:05.140
only optimize for one thing at a time ideally I want it to I want it to be
00:32:11.440
correct and then I want it to be pretty and then I want it to be fast like that's kind of like make it correct make
00:32:18.400
you know good to work with and then and then make it fast and generally when I
00:32:24.250
do it in that order I'm fairly happy with the end results I think I think
00:32:30.370
breeze is a good example of this because adding freeze everywhere makes your code ugly but then it sometimes makes it
00:32:36.220
faster so it's usually like you have to and that's why you have to measure is the performance impact so much better
00:32:41.470
you know the method in rails is called only once like ever on boot there's no
00:32:46.540
point in freezing it because it's not going to actually it's not like all bolts multiple times to build up your allocations so you don't necessarily
00:32:51.820
need that in the same way if you're like calling it all the time and you want to
00:32:57.010
keep your allocations down another
00:33:03.400
question what might you do to tackle a major performance degradation from a
00:33:08.920
framework library language upgrade that you only find with live traffic and actually FL has been going through a big
00:33:16.150
upgrade at Shopify and like you how do
00:33:21.430
you how do you tackle any performance regressions that you get some a major upgrade so just to give it like letting
00:33:32.530
clear like the rails thing you want to know those who ratio so the first thing you have to do is like OPI issue please
00:33:39.370
we want to fix that as soon as possible because what we want to avoid like this
00:33:46.150
kind of regressions and the only way that two we have right now to know that
00:33:51.310
is when people get with your life's traffic we have some some projects like
00:33:56.950
they hope benchmarks websites that runs the rails some rails benchmarks to see
00:34:03.400
if you had a regressions or not but that don't get other kids that's happening so
00:34:09.310
if you find our measure equations please support to the framework so we can work
00:34:14.800
together to fix those relations i so i discussed there's a lot of those reports
00:34:20.890
like I remember when we believe in research ooh he did a bunch of reports
00:34:27.010
saying that we were allocating more objects or some good peppers slower than before in
00:34:33.389
we fix all of them yeah that's my I agree reporting reporting is the key but
00:34:39.780
like how did you pick up how do you pick up that you've got a problem for example like you've just upgraded it did you
00:34:48.300
know before did you measure everything before did you deploy and then look at
00:34:53.340
monitoring how do you good supply so we have a we have some monitors working and
00:34:59.850
we also every single new version of different work we do is always hold out
00:35:05.700
so we put the new vaishu only a small subset of servers and we measure the
00:35:12.270
performance difference between them and if anything is in of the graphs hires
00:35:18.360
because of the new pressure we usually use some profiling tools like strike
00:35:24.390
proof to see what's the difference in the code path like what better have been
00:35:30.780
college now that was not to be college before and we get that report we see
00:35:36.300
first the change back in the service and we get the reporting we try to fix the
00:35:41.730
button so i kinda want to answer a subset of that question just see like
00:35:47.820
how do you deal with a performance problem if it like only happens in production so most of the time I'm able
00:35:56.040
to isolate it a little bit like say okay this endpoint is slow like this one
00:36:01.920
specific endpoint is low and then if I can't reproduce it locally like well what is different between my server and
00:36:09.480
and locally like one big one is data right like if you if you don't have data
00:36:16.770
locally that is you might be doing like a select star and without a limit and if
00:36:22.500
you only have five elements in your database it'll be really fast if you've got a million elements you know it's
00:36:27.600
it's like it's really painful and so like your database you read us your data
00:36:32.730
stores and and then may it might be the conditions it might not just be that endpoint and with that in your database
00:36:38.970
it might be that all of those things but only when one specific user is logged in
00:36:44.250
and so that's when having like a little bit of logging where if you do something like paper trail or something and you
00:36:50.670
say oh yes here here this one request like this very specific request is
00:36:57.000
itself slow I'm going to try and reproduce my local environment as much as to simulate that request as possible
00:37:04.950
and that's gotten me a really really long way granted you sometimes you spend all that effort and it still can't reproduce it
00:37:12.450
locally but nine times out of ten like I find some really low hanging fruit that way if I had a nickel for every time a
00:37:19.080
developer said it's fast on my machine like that yeah so there's a ton of
00:37:26.160
different ways to make sure that your experience is experiencing the same things that your users do the big one
00:37:32.430
that most rails developers and shops don't do is work in development with production like data you probably can't
00:37:39.330
do that if your Shopify or github but most of us aren't on Shopify or github so just making sure that when you know
00:37:47.450
you call user dot all that in development that might return ten rows but in production it's going to return
00:37:53.730
ten thousand so just exposing yourself as much as possible to the same
00:38:01.770
conditions that will occur in production on front-end this is always Network and
00:38:07.230
CPU we all have really nice MacBook Airs probably some fat fiber connections at
00:38:12.270
our offices or at home chrome dev tools lets you rot all your network connection there's also I mean
00:38:18.180
there's a ton of tools for throttling network connection but throttling your network connection slowing down your CPU chrome will do that for you and and
00:38:26.160
making sure you experience your website the same way your users do I just wanted
00:38:32.790
to make another comment about like major framework upgrades something that both Eileen and Rafael do is they're able to
00:38:41.130
run at the same time both an old version of rails and a new version of rails so
00:38:47.100
they have the ability to deploy it into production with the new major framework change see if it works but doesn't work
00:38:53.760
well we'll just use the old version for now and do you think this is something that
00:39:00.430
more I guess shops should a practice more shops should follow with major
00:39:07.180
rails upgrades or is this something that is very niche to github and Shopify well
00:39:14.650
one of the problems with github is were like really far behind so we kind of
00:39:20.440
have to do that like have the multiple versions because it's going to take too long to do an upgrade so you actually get to the point where we can deploy it
00:39:27.310
without having two different versions in the codebase most times actually in fact you theoretically could have more than
00:39:33.460
that we just don't do that more than two at once I think that if if
00:39:39.849
you have a big codebase it makes sense to have multiple Rails versions going just because of the whole how how long
00:39:46.150
the upgrade process is going to take because otherwise you're just going to constantly be falling behind and and there's no way you don't want to work on
00:39:53.890
a branch like by yourself that you're constantly rebasing because when you're fixing other people's conflicts all the time and then you're going to go like
00:39:59.530
extra crazy not just rails upgrade crazy but you're going to get rebased crazy
00:40:04.800
and so that's one of the things that we do in at github is we have a conditional
00:40:09.819
debt if and and this is how I recommend to do it the version that you're currently on is your if statement so
00:40:15.430
that's like the special case and then everything else should fall through so that way you keep moving the code that's special up to the if case and everything
00:40:21.940
else should fall through and then you know when it's broken for that new version and that it's not like you're not confused about well when am i
00:40:27.760
running rails for code or when am i running rails 3 codes like it's always your top conditional and and falls through so so yeah I believe that doing
00:40:39.640
this kind of dual booting as we call is valid to upgrade because it's make easy
00:40:47.589
to you to the point but actually find bugs but also I think it's good practice
00:40:54.040
for the entire community to be able to do that early like we always try to
00:41:00.760
release rails in some kind of good piece like every six months but we release
00:41:08.140
release candidly a lot of different things everyone is try risk and date usually in small
00:41:13.779
applications but the real process only happens when people upgrade the major
00:41:19.799
applications like we shall fight discuss github or big applications aggregated
00:41:25.150
because usually they get all the big corny case of the frameworks so if we
00:41:32.979
always do that the other applications that we always are able to use to defeat
00:41:40.089
speciality frameworks that would be better for the entire community here also for the application there any
00:41:46.150
resources that people can go to figure out how to do this because they know now
00:41:51.400
that it can be done but what what what magic do they need to do to make this
00:41:56.440
happen yeah so github has about four so really really odd bulk push about that but I
00:42:02.949
think it's a lot they get a blog but is somewhere someone from the github made
00:42:08.289
that bulk post and I fiction Phi also has a blog post two years ago baby and I
00:42:15.849
going to release my slightly also a new blog post next week excellent I'll just
00:42:25.029
I'll just upload the question is what do you do with the gem file and the gem
00:42:30.099
file lock file so we have a condition right now what we are doing we have a
00:42:35.979
multi page on bundler inside origin file because the files are Ruby coding became
00:42:42.640
occupied side the GFI itself that's not so great so I try to change that to use
00:42:50.829
a different approach like the band rarity told me that there
00:42:56.859
is a loop shall call it evolving file that you can use to share different to
00:43:02.140
different GFI O's and I'm going to use that in an
00:43:08.380
expert so we have two log files but only one is a file right now
00:43:13.709
yeah we have two log files that our regular gem file has a conditional in it so if the if it's github where else for
00:43:21.940
or whatever like it although with the gems for that and then otherwise it loads the three to four
00:43:28.260
times help one more like which jump all that lot no so when we yeah some unifier
00:43:36.640
me higher you hack button bundle basically everything has to run with the environment better and variable in front
00:43:42.490
of it so we do rails for equals 1 and then it knows which gem file locked to load so it loads those four but I don't
00:43:49.120
I don't know if we hacked bundler because I I wasn't there for that I
00:43:55.510
think we're running very very long time I think we're already five minutes over so I'm just web still here you can come