00:00:10.360
uh first we're going to start this
00:00:11.480
everybody in the uh the front row I want
00:00:13.759
you to all say hi to my mom because she
00:00:15.799
needs a picture everybody ready three
00:00:18.720
two
00:00:20.000
one there we go thank you all right
00:00:23.160
mom's going to be happy all right my I'm
00:00:25.320
me I'm my name is David hner um I'm
00:00:28.599
going to be talking about performance
00:00:29.679
today
00:00:30.920
um and a lot of people have asked me in
00:00:32.759
the last couple days well this Tech talk
00:00:36.040
more geared towards uh junior people or
00:00:38.280
more senior people I've been telling
00:00:40.039
people like uh it's probably more Junior
00:00:42.719
but now that I think about it I've seen
00:00:44.000
a lot of your senior code out there too
00:00:46.199
and you might not you might know what I
00:00:51.039
am going to present but you don't
00:00:52.840
probably do what I present so it might
00:00:55.320
be a reminder for those senior people
00:00:57.280
but oh I need the uh clicker and and so
00:01:00.600
when I started this talk also
00:01:03.559
um I wanted to show that my face was the
00:01:07.479
same size as errand and that ear is the
00:01:09.520
saying by the way but uh and I wanted to
00:01:12.119
start with a whole bunch of little jokes
00:01:14.000
but then I realized oh I don't have 45
00:01:16.200
minutes to talk I have 30 minutes to
00:01:18.360
talk oh no I'm not going to
00:01:20.920
actually want to get kicked off stage
00:01:22.880
because I think somebody would come up
00:01:24.119
here and maybe uh well tackle my
00:01:28.439
ass anyways so I'm going to start by
00:01:31.159
giving you a little history of zenes
00:01:33.399
zenes has been around about 15 years or
00:01:35.560
so we've had the first 12 years almost
00:01:38.560
no dedicated performance team is
00:01:40.640
actually working on the code so you can
00:01:42.159
imagine there's a lot of low hanging
00:01:43.600
fruit and one of the things I need to
00:01:45.680
talk about uh zenes is our end points
00:01:48.240
aren't what uh I'm guessing they're
00:01:50.240
dirty like some of yours also where our
00:01:53.680
restful one points aren't really restful
00:01:56.039
and by that for example for our uh code
00:01:59.039
base we have an includes user in the
00:02:03.439
ticketing endpoint or includes comments
00:02:06.119
so our actual endpoint will change the
00:02:09.800
response depending what the C user asked
00:02:12.040
for so it's technically not a restful
00:02:14.560
response there's many different
00:02:16.040
responses that you might get depending
00:02:18.040
on what the includes are that's going to
00:02:20.200
become really important at the end of
00:02:21.800
the talk because all of a sudden we came
00:02:23.400
up with ways of caching that craziness
00:02:27.200
at the end um
00:02:30.319
anything else yeah and we've diverged a
00:02:31.760
lot from Ruby on rail so upgrading is a
00:02:33.599
pain but here's the work that my team
00:02:36.280
can do and this is just three people um
00:02:38.560
and this is P99 data which means it's
00:02:41.239
the worst custo or the customers the
00:02:42.840
biggest customers are and you can see
00:02:45.360
all these Trends you can see when we
00:02:47.959
released code every one of
00:02:50.640
these big staggering uh code changes are
00:02:54.200
actually making a huge difference on our
00:02:55.879
infrastructure cost don't believe I'm
00:02:57.599
allowed to tell you how much our
00:02:58.800
infrastructure costs uh was benefited
00:03:02.080
but think of it more on the tens of
00:03:03.680
millions as opposed to 50,000 or
00:03:06.120
whatever not we so we saved three people
00:03:09.360
and with the help of the infrastructure
00:03:11.440
team you got huge benefits here uh so
00:03:14.680
archiving um a lot of uh people just uh
00:03:18.599
they always have a small app in the
00:03:19.720
beginning they're like okay I'm just
00:03:21.440
going to keep all my data in the main
00:03:24.440
data store whatever that is usually my
00:03:26.000
SQL and they never even think that uh
00:03:29.000
this data is stale after a certain
00:03:31.280
amount of time you should probably have
00:03:33.319
an archiving strategy and in fact uh
00:03:36.239
like a complete removal strategy of data
00:03:39.080
and bring that up when you start your
00:03:41.280
applications because bringing it up
00:03:43.120
later is harder and basically be able to
00:03:44.760
tell your customers hey we're not going
00:03:46.200
to keep your data let's say five years
00:03:48.159
and then for archiving for the Zen desk
00:03:50.799
after a ticket is closed most people
00:03:52.680
don't care about the ticket after a
00:03:54.000
certain amount of time so after a
00:03:55.640
certain amount of time we'll say you
00:03:57.280
know what we're not going to keep that
00:03:58.280
in the MySQL data store we're going to
00:04:00.280
uh put that we do it in Dynamo but
00:04:02.319
whatever store you might want to put and
00:04:04.239
then that will save your application
00:04:06.319
from having to make SQL queres on just a
00:04:08.400
huge amount of data
00:04:10.120
set um so how did we get here we did not
00:04:15.120
have a PM when we started uh this team
00:04:17.239
just three years ago which actually
00:04:19.959
probably was for the better because we
00:04:22.400
let the data lead us as opposed to let's
00:04:24.320
have this Grand project and there's just
00:04:26.160
so many little things you can do within
00:04:27.840
your app to actually get this your app
00:04:29.759
app to actually perform well but the
00:04:32.199
first thing you really need to do in
00:04:34.479
that effort is ADD observability at
00:04:37.680
zendesk we use data dog I'm sure many of
00:04:40.120
you actually use data dog and uh
00:04:42.840
probably are complaining about the cost
00:04:44.360
but um so here's what a typical Trace
00:04:47.160
that we had before we started the team
00:04:49.080
would look like and we actually added a
00:04:51.120
lot of little details in here any place
00:04:52.880
that you see uh anything yellow below
00:04:56.360
the purple um is stuff that we added
00:05:00.080
custom but the problem with this data is
00:05:03.000
it's still left a lot of empty spaces
00:05:05.720
but to fill in all that data would just
00:05:07.520
cost a lot of money because data dog
00:05:10.479
charges for the amount of uh the the
00:05:13.400
amount of traces you actually put in
00:05:14.800
there so we need some flexibility and
00:05:17.120
I'll go into the flexibility that we
00:05:18.800
added and then um essentially these data
00:05:22.240
dog traces that we actually do do add
00:05:24.520
are work in
00:05:27.120
progress new features will come up you
00:05:29.759
need to add more um traces new areas
00:05:34.720
that become slow in the application for
00:05:36.840
some P99 customer add more traces so you
00:05:39.600
never can say that you're done but the
00:05:41.600
adding this uh observability was
00:05:43.720
absolutely crucial to find the problems
00:05:45.199
that we were doing so how did we do that
00:05:48.440
here's an example method it's actually
00:05:50.120
not a real method in our application but
00:05:51.800
you'd have a new ticket method for
00:05:53.360
example and this is just a syntax I'm
00:05:55.960
really showing you here but the syntax
00:05:57.840
is essentially you have a a method
00:05:59.960
called rapy dogging Trace you pass in
00:06:03.080
new ticket and new ticket is the uh
00:06:05.680
method name and then the second new
00:06:07.639
ticket uh parameter is actually what you
00:06:09.800
want to call it that Trace in data dog
00:06:12.199
so you might not want the same um Trace
00:06:15.479
name and the last one's really just a
00:06:17.120
color coding we mostly use the same
00:06:19.360
color coding for anything custom that
00:06:20.759
we're doing but uh you can add some
00:06:22.800
color coding so here's the syntax for
00:06:26.199
the block notation that we have and
00:06:29.960
exactly what you see it's just basically
00:06:31.479
a block where you and you add the low
00:06:34.080
method in between the problem with this
00:06:35.840
method is that it clutters up your code
00:06:37.560
you don't want blocks all over your code
00:06:38.880
so the other me uh uh way to actually
00:06:42.919
add traces is better but then let's
00:06:44.840
actually dive more into the actual
00:06:47.160
implementation of this so here's here
00:06:49.800
what we have is the block notation uh
00:06:53.240
implementation so what what you do is
00:06:56.080
the first couple lines there is a
00:06:58.599
software switch we call our software
00:07:00.360
switch uros none of you call your
00:07:02.319
software switch uros anyways essentially
00:07:04.879
you see if you have the feature if you
00:07:06.680
have the feature then essentially what
00:07:08.240
you're doing here is wrapping a trace
00:07:09.639
around your implementation if you don't
00:07:11.960
then you just yield to the block and it
00:07:13.199
just renders like normal and you it's
00:07:14.720
almost like the block is not even
00:07:18.960
there likewise um for WAP de debugging
00:07:22.960
traces which was the implementation that
00:07:24.960
I showed can I go fast yeah right here
00:07:28.720
for RDW uh debugging traces this is what
00:07:31.759
we do essentially what we doing is you
00:07:34.479
redefine the method that you're actually
00:07:36.360
wrapping the the block around so you
00:07:38.520
have the same code that the the block
00:07:40.840
notation in within this and you redefine
00:07:43.720
the method that you're trying to
00:07:44.960
actually uh put traces in and then you
00:07:47.520
call that method within this block with
00:07:49.720
a syntax here
00:07:52.000
um so what you get before is traces that
00:07:55.919
look some somewhat like this and I need
00:07:58.280
a drink CU my water
00:08:03.639
and uh which is gives you a good amount
00:08:05.800
of data but then after you get something
00:08:07.919
like this where you can dive right down
00:08:09.960
into the actual problem you can find the
00:08:11.840
actual method that if you zoom into this
00:08:14.159
you'll see something like this where all
00:08:15.960
of a sudden you can go into merge ticket
00:08:18.000
data in model that's an actual method in
00:08:19.720
our implementation and like uh purple
00:08:22.879
thing on the bottom we know that's a
00:08:24.560
query that query is happening uh each
00:08:26.520
time you call that is it n plus1 do you
00:08:28.479
want to cach that data do you is it an
00:08:30.680
okay query that you expect well now we
00:08:33.839
could have ways to debug every query
00:08:36.360
know exactly where they're coming from
00:08:37.599
and know how to actually remove
00:08:41.080
it the other thing you have to know is
00:08:43.399
as I said earlier these traces are not
00:08:45.600
free this block on the bottom is not one
00:08:47.640
Trace it's actually 5,000 it's just
00:08:50.680
they're so damn close together it looks
00:08:52.160
like one now ironically enough the
00:08:55.120
latency added to 5,000 extra traces with
00:08:57.959
data dog data dog is amazing it does not
00:09:00.160
actually give you much extra latency
00:09:01.880
that is even noticeable that being said
00:09:04.519
I've released code that I'm not so proud
00:09:08.079
of that added 200,000 extra traces it's
00:09:11.839
only a little while we only turn the
00:09:13.200
software switch on for a little while
00:09:15.079
but uh that was code a loop within a
00:09:17.680
loop within a loop I think they call it
00:09:19.160
n cubed or something like that where we
00:09:22.279
got 15 seconds extra response times and
00:09:25.320
we turned it off quickly but uh and it
00:09:27.680
was easy to find the problem at that
00:09:28.920
point because like oh
00:09:31.760
but yeah don't do that um I'm going
00:09:35.720
to move on to some testing that we do um
00:09:38.800
we one of the things I don't see in a
00:09:40.920
lot of applications is testing for the
00:09:42.440
number of queries we have a very
00:09:44.279
specific sent uh syntax at zenes but
00:09:47.079
there's a whole bunch of Open Source
00:09:48.959
gems that you can actually test the
00:09:50.440
number of queries that you have in your
00:09:51.880
application it's really important for
00:09:53.880
things that you especially expect no
00:09:55.399
queries from but whoa uh but could have
00:09:59.399
queries and the syntax is something like
00:10:01.680
this where you have assert SQL queries
00:10:04.240
you pass in the number of SQL queries
00:10:05.680
you want and then you pass it to Rex it
00:10:08.120
works really really well but we actually
00:10:10.240
do this for memorization also put this
00:10:12.880
in your memorization code you say hey
00:10:14.760
the first time I expect a query the
00:10:16.079
second time it better not have a query
00:10:17.640
and if somebody else changes your code
00:10:19.440
like 10 years from now or what something
00:10:21.200
like that um they're going to actually
00:10:23.200
have a failing test and yes that does
00:10:26.040
happen um and I'm going to I'm going to
00:10:29.640
keep moving cycling through a whole
00:10:31.079
bunch of topics this is another one
00:10:33.279
where a lot of you go oh I want to I
00:10:35.800
want to have do some feature that has
00:10:38.480
some dramatic differences in data dog so
00:10:41.279
I can show my boss what I did for work
00:10:43.680
yeah that's nice but you know what
00:10:45.480
there's end points that we have in our
00:10:47.279
application that get 87 billion I think
00:10:49.839
it's more like 10 and something billion
00:10:52.000
requests per year Well if you you
00:10:54.240
multiply 87 billion Time 1 millisecond
00:10:57.200
you're saving a, days worth of sequel on
00:11:01.360
your machines sure it's not going to
00:11:04.000
look really great to your uh uh boss
00:11:07.000
because he's going to be like one
00:11:07.959
millisecond really but to your
00:11:10.519
infrastructure team they're going to say
00:11:12.160
God bless
00:11:13.920
you
00:11:15.519
um here's something that we do at zenes
00:11:17.959
that I don't think we most people have
00:11:19.519
seen before I don't know but uh anyways
00:11:22.120
we have something our account object is
00:11:24.240
actually what I would call a god object
00:11:26.200
you can call it a Singleton but we don't
00:11:27.839
follow the exact Singleton patterns so I
00:11:29.920
call it a god object because it is God
00:11:32.480
what that means is if you call for the
00:11:35.279
account any place in the uh zenes
00:11:38.839
whether it's user data account ticket
00:11:40.959
data account current account in your um
00:11:44.480
controllers wherever you call it the
00:11:47.320
object is the exact same object it's not
00:11:50.440
just account. ID is the same it's the
00:11:53.320
same object ID which allows you to
00:11:55.760
leverage that and take that object and
00:11:58.279
now you can use that object object
00:11:59.839
anywhere and memorize on top of that
00:12:02.279
object instead of the object that you're
00:12:03.639
getting from someplace else that there's
00:12:05.120
a Code implementation way over here and
00:12:06.920
then there's a Code implementation way
00:12:08.440
over here but you can't like figure out
00:12:10.920
how to actually memorize the the two
00:12:13.760
because they're completely different uh
00:12:15.279
pieces of the code um anyways I'm going
00:12:18.240
to be going into this quite a bit if
00:12:19.760
you're not uh familiar with memorization
00:12:21.560
like this the syntax looks like this if
00:12:23.880
you need help on that you're probably
00:12:25.320
totally confused so I hope that uh you
00:12:28.040
understand how to memorize very simply
00:12:30.320
um here is an imp implementation of um
00:12:34.880
like that we don't uh we didn't uh do
00:12:38.880
any type of uh memorization where we
00:12:41.120
just Loop through a whole bunch of
00:12:42.480
audits we'd call audit. author and then
00:12:46.279
author. user emails most of the time the
00:12:48.480
author was the same author not the same
00:12:50.519
object ID but the same I like the same
00:12:52.320
exact author we called user emails on it
00:12:55.320
and we' get anywhere from one to three
00:12:57.079
queries every time we called this even
00:12:59.560
though it was the same damn author every
00:13:01.079
time our implementation is a lot
00:13:03.160
different than this but this is the
00:13:04.279
condensed
00:13:05.639
version well if you actually leverage
00:13:08.199
the fact that you have that account
00:13:09.600
object that is the same account and you
00:13:12.000
put the user emails method on the
00:13:14.880
account object and just pass in the user
00:13:17.680
well now all of a sudden you can
00:13:19.079
memorize that on the account and if you
00:13:21.079
actually have to use user emails any
00:13:23.639
place in your request cycle you actually
00:13:26.360
get a memorized result all after the
00:13:28.839
first
00:13:32.040
time changing topics again a lot of
00:13:35.160
people in Ruby and rails they're always
00:13:36.880
getting back arrays from their data sets
00:13:40.199
makes sense great but the problem is
00:13:43.720
sometimes an array is not the right
00:13:45.240
thing to use at what for what you're
00:13:47.880
doing and yes in this case there's only
00:13:50.880
three things in the array but what Let's
00:13:52.600
Pretend This is not three uh uh object
00:13:55.600
in your array Let's Pretend There's
00:13:57.560
20,000 then you do some C calculation
00:14:00.519
and then you call Unique well unique is
00:14:02.959
not a free uh uh method when you have a
00:14:06.639
huge array so this is going to create
00:14:08.440
some time in certain some in certain
00:14:10.480
cases in fact in ours it did cause let's
00:14:13.120
say 100 milliseconds of time I don't
00:14:14.480
know what it caused but anyways if you
00:14:16.959
change this to go create a set and then
00:14:20.639
do that same sum calculation but now
00:14:23.600
actually keep on adding to the set every
00:14:26.720
you're guaranteed uniqueness the end of
00:14:29.560
that this set by definition is going to
00:14:31.720
be unique so now instead of actually uh
00:14:34.839
calling the unique method and spending
00:14:36.600
time 100 milliseconds on that you're
00:14:38.440
just iterating through and you get
00:14:40.480
unique values and you don't have to even
00:14:42.000
call Unique at all likewise uh with
00:14:45.680
leveraging a hash I think this is more
00:14:47.519
obvious but it's something that I just
00:14:48.880
want to go through is basically an array
00:14:51.519
you can use a hash and instead just look
00:14:54.079
for the key you have an O of N1 uh uh
00:14:58.240
operation or L of one operation actually
00:15:00.480
constant time to look up for the three
00:15:02.680
and I just have nil here but you could
00:15:04.160
actually have some calculation three
00:15:05.560
equals something and uh your operation
00:15:08.720
is just going to be so much faster
00:15:10.120
because you're leveraging a
00:15:11.680
hash uh I'm adding this also uh flat map
00:15:15.880
versus m flat flatten it's literally
00:15:18.279
because every application I've ever
00:15:20.480
opened up uses map. flatten all over the
00:15:24.440
place when you could use flat map I'm
00:15:27.079
not going to go into the the differences
00:15:28.639
of the to but you do have to know that
00:15:30.079
there is differences but every uh
00:15:32.000
Implement every implementation of map.
00:15:35.120
flatten that I've ever seen can be
00:15:37.920
replaced with flat map I've never seen a
00:15:40.040
case where it hasn't at least in my
00:15:42.519
applications more
00:15:48.120
water so at zendesk we also we obvious
00:15:52.319
well anybody knows zenes is we get a ton
00:15:55.160
of emails coming in for customer tickets
00:15:58.160
in those emails
00:15:59.440
that data
00:16:01.959
is if you've ever looked at emails it
00:16:04.519
can be and you get you get in the
00:16:08.160
emails and there's one customer this
00:16:12.759
whoever did this was just stupid but
00:16:14.639
anyways One customer they had an email
00:16:17.240
signature and at the end of their email
00:16:19.360
signature was 36 at least
00:16:22.160
36,000 empty
00:16:24.240
spaces and our re XX is for whatever
00:16:27.759
reason were run on the out outbound and
00:16:29.319
it made sense to run out the x's on the
00:16:31.399
outbound the problem is our R XS were
00:16:33.480
extremely slow with
00:16:35.639
spaces and well this customer is
00:16:37.759
complaining why am I getting bad slow
00:16:40.040
responses well Jesus Christ
00:16:43.519
what anyways this is not our
00:16:46.160
implementation exactly because we had to
00:16:48.000
take care of uh tabs we had to take care
00:16:50.240
of uh new characters we had to take care
00:16:53.440
of returns um but essentially you can
00:16:57.480
just run a simple sanitizer to some
00:17:00.120
degree whatever your data you're trying
00:17:01.440
to sanitize run it on the inbound of the
00:17:04.880
actual data coming in rather than the
00:17:06.559
outbound and it saves
00:17:09.760
time uh give a pluck so let's picture
00:17:13.240
you have a form that form has a whole
00:17:15.760
bunch of fields and some of those fields
00:17:17.720
are drop- down fields and all those drop
00:17:19.760
down Fields have options so Le let's say
00:17:23.520
oh so they all have options typically a
00:17:26.439
drop down field should have let's say a
00:17:28.120
100 or less in your options but we give
00:17:31.520
our customers a tool and when you give a
00:17:34.720
customer a tool some customers want to
00:17:36.520
shoot themselves in the foot and yes we
00:17:39.000
do have limits in this but at the end of
00:17:40.600
the day some people go way over the
00:17:42.240
limits and I'm talking like 20,000
00:17:44.520
fields in your actual um dropdowns our
00:17:48.799
problem is originally when we took got
00:17:50.799
the data from uh the the options we
00:17:53.600
grabbed the data and we just
00:17:56.159
instantiated 20,000 active record uh
00:17:58.880
objects what can go wrong well a lot and
00:18:02.120
essentially what we did did to change
00:18:04.480
this is we changed this and we actually
00:18:06.880
used the pluck this isn't exact pent be
00:18:09.159
too long for the field but what we did
00:18:10.720
is uh got that data in pluck used to
00:18:13.799
convert it to a hash well when you
00:18:15.600
convert it to a hash that's just so much
00:18:17.600
faster than instantiating 20,000 active
00:18:20.080
record uh objects and then the presenter
00:18:24.400
before it looked like this it was very
00:18:26.320
readable it was active record after it
00:18:28.840
looks like this it's not as readable but
00:18:31.360
it's pretty damn readable and if you're
00:18:33.640
going to save let's say 500 milliseconds
00:18:36.240
on some of the responses it's hella
00:18:39.240
worth it um one of the things you're
00:18:42.159
probably not asking but when I gave this
00:18:44.120
talk and internally they were asking uh
00:18:47.000
what is this ticket field store so in
00:18:49.400
ticket the ticket field store in uh our
00:18:52.400
application well it looks like just like
00:18:54.679
an object that we store ticket fields in
00:18:57.080
well one of the things you got to know
00:18:59.039
zendesk our ticket life or request cycle
00:19:02.559
you go in and you let's say you have in
00:19:04.640
the beginning uh some validations in the
00:19:08.000
middle you do something that we call
00:19:10.159
triggers and in the end you have a
00:19:12.600
presenter well each one of them needs a
00:19:14.600
different subset of uh ticket Fields the
00:19:18.200
beginning you need maybe all the ticket
00:19:20.360
fields in the middle you need a subset
00:19:22.039
of ticket fields in the end you might
00:19:23.480
need all of them again or some some
00:19:25.159
combination of the above well what we
00:19:27.440
did is hey let's create a ticket for
00:19:30.559
store we know the the account object is
00:19:33.400
this SC object that I was talking about
00:19:35.240
we can memorize it and we can reuse
00:19:37.679
these this data throughout the request
00:19:39.480
cycle so what you have in the beginning
00:19:42.520
is that that you're using one subset of
00:19:46.360
the data another subset of the data
00:19:49.000
another subset of the data and at the
00:19:50.760
end of the day like you're making one
00:19:53.039
query for all that data instead of 1 2 3
00:19:57.840
instantiating them objects because that
00:19:59.520
ain't free either and you're saving a
00:20:01.280
hell of a lot of time and if you want to
00:20:03.760
take that one step further you can
00:20:06.360
actually leverage active support memory
00:20:08.080
store I wish I actually had time to go
00:20:10.600
with the implementation of this but
00:20:12.440
active support memory store what you can
00:20:14.440
do is now when you make that uh initial
00:20:17.440
call you actually check to see if you
00:20:19.600
cach this up already the very because we
00:20:21.960
never change or very rarely change
00:20:23.760
ticket fields we can actually store that
00:20:25.559
in memory and if you store that from
00:20:27.480
memory instead of actually making a call
00:20:29.520
in the beginning in the middle in the
00:20:31.440
end we never make a call we just get it
00:20:33.520
from memory every time we actually make
00:20:35.320
the request no requests throughout the
00:20:36.720
whole life life cycle for um many of our
00:20:39.720
implementations of ticket
00:20:42.559
fields that is technically the code I
00:20:45.320
don't have time to go through that but
00:20:47.039
and it wouldn't make sense unless I
00:20:48.280
actually added a heck a lot more code if
00:20:50.159
it was a 45 minute talk
00:20:52.320
though anybody in a t anyways um I'm
00:20:56.240
going to move on to max age if you're
00:20:58.960
you may or may not be familiar with max
00:21:00.600
age but the syntax in the Ruby code is
00:21:02.960
something like this where you say
00:21:04.120
expires in 60 seconds if you don't know
00:21:06.200
what this is this is really important
00:21:09.360
because I'm going to go back here this
00:21:12.400
is the amount of traffic we reduced
00:21:14.559
after implementing uh max age what max
00:21:18.159
age does is it's if your uh browser
00:21:21.240
makes a request let's say it request the
00:21:23.520
tickets and then you send back the
00:21:25.600
tickets and expires in 60 Seconds uh is
00:21:30.080
set in rails when you make the call the
00:21:32.400
second time your browser stops the call
00:21:34.440
and says hey YY yo you asked for this 60
00:21:36.640
second be within 60 seconds I'm just
00:21:38.760
going to give you the same data back so
00:21:40.919
the fastest traffic you can ever have in
00:21:42.679
your application is the traffic that
00:21:44.919
never hits your application so that was
00:21:48.520
a big win um last and oh my god I've
00:21:52.559
went so freaking fast did you add
00:21:56.240
time either that I'm just stocking SP
00:21:58.760
fast but anyways um stale stale in rails
00:22:02.679
is amazing if you're not uh if you don't
00:22:05.840
know what it is well I guess I have time
00:22:08.360
to explain it but anyways um stale
00:22:11.120
unfortunately because of the reasons
00:22:13.559
what I was talking before uh does not
00:22:15.840
work on our application because what
00:22:17.279
happens is stale is supposed to see if
00:22:19.400
the E tag that you that's sent in is
00:22:21.919
stale and if it is if if it's if you're
00:22:25.640
using the same an e tag that is not
00:22:28.600
stale you don't render what you do is
00:22:31.760
active record takes or the uh the
00:22:34.799
implementation um takes Brands and says
00:22:37.880
oh you know what this is uh not stale
00:22:40.159
data so it doesn't render that it goes
00:22:42.159
back to the middleware and then uh
00:22:45.919
converts B I'm saying this wrong anyways
00:22:49.440
at the end of the day in the middleware
00:22:51.480
what happens is the 200 that response
00:22:54.840
returned here is changed to a 304 and
00:22:57.520
what what 304 means is no content is
00:22:59.799
sent back so you didn't need to actually
00:23:01.200
render anything to the customer which
00:23:03.240
saves you a lot of time unfortunately
00:23:05.559
that does not work at zenes um I kind of
00:23:08.559
gave a little bit of a history but
00:23:10.120
anyways um so the history of zenes the
00:23:13.279
beginning uh or essentially this is what
00:23:16.120
middleware looks like and your
00:23:17.400
controller looks like typically the
00:23:19.279
stale method is called here which means
00:23:21.360
all that at time at the end does not
00:23:23.960
have to be actually used you return
00:23:26.200
really fast from your responses
00:23:27.880
unfortunately
00:23:29.000
at zendesk we have to calculate the eag
00:23:31.440
value at the very end of the response
00:23:34.159
because depending on the parameters that
00:23:36.720
were passed in we have no idea what the
00:23:39.320
actual output will be so we're looking
00:23:42.240
at the output and seeing what the eag
00:23:44.000
would value be but all we spent all this
00:23:46.200
time yeah sure you get less traffic in
00:23:49.159
the um um
00:23:55.240
water you get less traffic over the wire
00:23:57.679
going back to your customers but you're
00:23:59.200
not taking the advantage of the redu
00:24:00.880
time you can do here so what we do oh
00:24:03.840
yeah and by the way this is not because
00:24:05.200
of complete mistakes there's a lot of
00:24:07.000
times your product managers will go in
00:24:09.200
and say Hey
00:24:11.080
listen I need this and I need it
00:24:13.559
tomorrow because we're going to lose a
00:24:15.080
customer or something like that and by
00:24:17.320
the way when zenes began there was no
00:24:19.120
such thing there might have been such
00:24:20.640
thing as eex but it wasn't in rails so
00:24:23.279
we had to create our own implementation
00:24:25.000
well before rails actually started we
00:24:26.880
didn't create it correctly but hey it it
00:24:31.320
worked and it was a heck of a lot better
00:24:32.919
than it was there before so wow uh I
00:24:37.720
just described that by the way I
00:24:39.640
actually sent these to this uh anyways
00:24:43.640
good so the real solution here is for us
00:24:45.840
to actually go in and start using rails
00:24:48.080
the right way unfortunately um that is
00:24:51.159
hard it takes years our customers are
00:24:53.039
used to the our apis the way they are
00:24:55.360
today we can't just go in there and say
00:24:57.399
hey here's a new API you can't use a
00:24:59.120
thing that you us 15 years breaking
00:25:01.440
changes are dreadfully hard when you
00:25:04.559
have a quarter of a million
00:25:06.720
customers um so what we did is we cached
00:25:12.480
that eag at the very end right oh here
00:25:17.640
we cash that value and
00:25:21.120
then uh we took that value and we saved
00:25:26.039
it in cash and then when we had a new
00:25:27.960
request Quest we would take a look at
00:25:30.000
the cach D tags that we had actually and
00:25:33.480
we by the way we have phenomenal cash
00:25:35.399
busting at zenes it's a very personal
00:25:37.559
thing to every customer or to every uh
00:25:39.880
implementation so your implementation of
00:25:42.320
how you cash bust is going to be very
00:25:44.640
personal to you but we have phenomenal
00:25:46.120
cash busting so anyways we check the E
00:25:50.279
tags that have came in and we compare it
00:25:52.600
to the eag that's coming in from the
00:25:54.679
request if it matches we know right at
00:25:57.440
the beginning of the life cycle that we
00:25:59.880
don't have
00:26:01.080
to look at the presenter all of the data
00:26:04.640
is fresh or is good or not stale so what
00:26:10.080
we do is we set the headers to the tag
00:26:12.360
that's appropriate and then we have we
00:26:14.840
turn 304 not modified and I was saying
00:26:17.360
in rails it's actually the middleware
00:26:19.679
that kind of takes care of switching
00:26:21.200
from 200 to a 304 and then sending no
00:26:25.080
content back but in our implementation
00:26:27.320
we kind of do it much more at the
00:26:28.600
controller
00:26:30.120
level and again this is my last slide
00:26:33.120
kind of um where uh your caching
00:26:37.120
strategy is very important got to talk
00:26:39.480
to probably everybody on your team get
00:26:41.480
get a good caching strategy and you're
00:26:43.279
going to be able to uh leverage a lot of
00:26:47.080
just great work and make it make make
00:26:49.279
your caching strategy easy I can't it's
00:26:52.480
a whole new talk and probably many talks
00:26:54.919
on how to create a good CRA caching
00:26:56.760
strategy unfortunately
00:26:58.720
um I'm supposed to stop here but I do
00:27:01.840
have extra 5 minutes and I had planned
00:27:04.960
to make this a joke slide where somebody
00:27:06.720
was going to come and tackle me off this
00:27:09.039
stage but I guess I can actually talk
00:27:11.080
about this also um I don't know if I
00:27:13.640
have time for both all of these but um
00:27:15.919
as
00:27:16.960
if when you cash a lot of times you cash
00:27:20.520
on a single object and you go okay I
00:27:23.360
cash that and then you like hey I cash I
00:27:26.080
did a really good great job I cashed
00:27:27.880
this user for example or something like
00:27:30.120
that but then in your implementation
00:27:31.880
you're in the index uh action and you
00:27:34.720
call your cash again and again and again
00:27:38.799
and you make a 100 calls to uh to cash
00:27:41.399
well guess what if you make a 100 calls
00:27:43.279
to your cash you're actually
00:27:46.480
getting um you're basically have an N
00:27:49.240
plus1 but it's not in my SQL it's now in
00:27:51.480
your caching layer but reing leveraging
00:27:54.360
read multi you can actually get all that
00:27:56.640
cash at one time
00:27:58.880
so this is really important and if
00:28:02.039
you're look actually looking at your
00:28:03.200
data dog slides or however you monitor
00:28:06.000
your stuff uh you will save a lot of
00:28:09.000
time by uh grabbing data using read
00:28:13.000
multi instead of uh uh however else
00:28:16.000
you're doing it and last thing and I
00:28:18.960
since I do have time I'll talk about
00:28:20.440
forcing indexes uh damn this was my code
00:28:24.519
I was wishing I I wasn't supposed to
00:28:26.120
actually present this I I went a little
00:28:28.159
little fast anyways um yeah let's say
00:28:33.440
seven years ago there was code in our B
00:28:36.360
uh there's code that uh we implemented
00:28:40.320
and uh we forced indexes because at the
00:28:43.240
time it made sense
00:28:46.320
well as time goes on new features are
00:28:49.080
added you go to Aurora 2 to Aurora 3 or
00:28:53.039
my SQL 5 to8 I think that it was but
00:28:55.440
anyways you change a lot of things in
00:28:57.240
your application
00:28:58.559
well you got to keep on going back to
00:29:00.640
your places that you force the indexes
00:29:02.519
because sometimes they don't make sense
00:29:04.480
the biggest gain that I had in the past
00:29:06.559
year was removing a force index on our
00:29:09.640
ticketing end points it was so sad
00:29:12.720
because I implemented the code to force
00:29:14.760
the code and then seven years ago I'm
00:29:16.919
like who the
00:29:18.519
oh
00:29:20.559
sorry and anyways does anybody want to
00:29:23.240
tackle me off
00:29:25.399
screen that's it
