00:00:00.900
foreign
00:00:12.120
I think we have 1 30 on the clock so I
00:00:15.360
should get this show underway thanks for
00:00:17.220
joining us all today for bringing your
00:00:19.320
ales monolith as your business grows and
00:00:23.039
while I'm up here talking I'm Carrick
00:00:25.980
Rogers the director of engineering for
00:00:28.320
our web applications team this is very
00:00:31.500
much a group presentation that is made
00:00:34.200
possible by all the hard work of all the
00:00:36.480
engineers that have contributed to
00:00:37.920
various portions of our code base and
00:00:40.140
they're down in the front row so
00:00:41.780
unbeknownst to them they may have to
00:00:43.680
help answer some questions especially in
00:00:45.660
areas that I'm not known on sorry Lily I
00:00:49.260
got you in the end
00:00:51.780
so um you know why um who is who is
00:00:55.980
ontra what do we do to start with we
00:00:58.079
process digitize and analyze private
00:01:00.420
Market legal contracts
00:01:02.820
um the the simplest way to think of this
00:01:04.680
is investment Banks like to sell a lot
00:01:06.360
of things to each other they have legal
00:01:08.040
obligations that come out of them and
00:01:10.500
they need to get those contracts through
00:01:12.240
they need to understand what do I need
00:01:13.619
to report to my investors what I need to
00:01:15.960
report to the SEC what I need to report
00:01:18.119
to them six months from now we also do
00:01:20.100
all of that with unions and pensions
00:01:22.200
so we have a lot of AI enabled software
00:01:25.140
that helps specialist lawyers process
00:01:26.759
these contracts and pull out the key
00:01:28.860
terms that they're going to have to use
00:01:30.840
to power their reports
00:01:33.200
and we are a Ruby on Rails monolith the
00:01:36.780
Ruby rails part I'd imagine is quite
00:01:38.460
unsurprising with an ember JS front end
00:01:42.720
and we've been around since 2016 when
00:01:45.420
the Ruby application launched I think we
00:01:48.479
actually I'm told soft launched in late
00:01:51.000
of 2015 our CTO wanted to say he was
00:01:54.720
responsible for doing the entire first
00:01:57.000
round of the application so he pulled a
00:01:58.680
bunch of all-nighters before our
00:01:59.939
principal engineer Jerry joined to
00:02:02.820
avoiding having to share the initial
00:02:04.979
credit and we've been in the same code
00:02:09.360
base in mono repo since Inception and we
00:02:12.000
have about a hundred uh people you know
00:02:14.400
we're headed towards having worked on it
00:02:17.239
and so then why this talk one of the
00:02:20.700
things we found especially is we've been
00:02:22.140
going out you know hiring talking to
00:02:24.300
potential vendors contractors everything
00:02:26.040
is um we get variations on what is your
00:02:28.739
code base like why haven't you moved off
00:02:31.080
monolith slash mono repo you know
00:02:33.300
microservices are very much a thing and
00:02:35.940
then it's all leading up to the big
00:02:37.500
question of are you still on the
00:02:39.000
monoliths because you want to be or
00:02:40.620
because you have to be often asked in
00:02:43.379
the vein of is your monolith just too
00:02:45.120
hard for you to shred and escape from so
00:02:48.060
you're stuck on it or are you actually
00:02:49.379
happy there and we've thought about it
00:02:53.340
and we're actually quite happy there
00:02:54.900
both in terms of our ability to function
00:02:57.060
at scale and in terms of our ability to
00:03:00.360
actually have a code base that you have
00:03:01.560
a lot of Engineers committing into one
00:03:03.060
repo and not be stepping on each other's
00:03:05.459
toes not be constantly running into
00:03:08.099
problems and with the frequency that
00:03:10.620
we've been asked this question has kind
00:03:12.000
of gotten us thinking um
00:03:14.760
you know like how did we end up happy
00:03:17.760
um we can think back we can to some
00:03:19.680
certain um decisions that we've made
00:03:21.840
over the years and those led to specific
00:03:25.019
um high points or low points with regard
00:03:27.060
to our happiness those um paid off or
00:03:29.879
didn't pay off but at the same time we
00:03:32.340
realized we never really had a full-on
00:03:34.379
retrospective of how did we make all of
00:03:37.980
this work how did we actually end up at
00:03:40.860
a point where when we tell somebody
00:03:42.420
we're talking to that we're happy we're
00:03:43.860
not just you know uh lying to them as
00:03:46.200
part of the pitch process but we're
00:03:47.640
really happy there and
00:03:51.060
um so we kind of started to think of
00:03:52.680
that and we came up with this talk and
00:03:54.120
this is a talk where if we had a time
00:03:55.680
machine and we could go back and we
00:03:56.819
could talk to ourselves four or five
00:03:58.379
years ago you know we would tell
00:04:00.180
ourselves um you know hey this is this
00:04:02.819
is great keep doing this or yeah this
00:04:05.220
was not such a great idea you shouldn't
00:04:07.200
have done something this way and
00:04:08.700
hopefully for folks out there that might
00:04:10.860
be looking at you know thinking about
00:04:12.840
keeping their monolith alive and
00:04:14.400
functional as they grow they can also um
00:04:18.479
you know derive some value from that
00:04:22.380
so before we um you know jump in to um
00:04:25.919
where we are I want to kind of just you
00:04:27.240
know set the stage for how big of a
00:04:28.800
monolith we are
00:04:30.240
um so we're at about a I'll say I'm sure
00:04:33.419
we're over 4 500 commits by now since I
00:04:35.940
pulled these slides together a couple
00:04:37.199
weeks ago and I'm pretty sure people
00:04:39.060
have been working since then so I
00:04:40.680
believe we've um broken through the 4
00:04:42.540
500 barrier we have um 48 different code
00:04:46.020
committers
00:04:47.400
um we have um about 62 000 lines of Ruby
00:04:51.180
code spread
00:04:52.680
um you know spread across app live and
00:04:54.720
config and about 112 in our spec
00:04:58.800
directory right now because tests are
00:05:00.479
very important and then on the Ember
00:05:02.699
side of the world we have a little over
00:05:04.259
61k of JavaScript and a little over 33k
00:05:08.220
of handlebars we have all The Usual
00:05:11.280
Suspects in our uh you know Ruby on
00:05:13.800
Rails tooling active record redis
00:05:16.259
sidekick
00:05:17.600
what you kind of expect off a standard
00:05:20.160
models especially one that kind of came
00:05:21.720
out in the
00:05:23.100
you know mid-20 teams or
00:05:25.860
so and then
00:05:28.320
um just to prove we have project Olive
00:05:30.979
Scorpius off of this is not a real
00:05:33.300
project but off of kind of our demo
00:05:36.180
environment showing um yeah we are
00:05:38.280
actually a real app we can show you
00:05:40.080
things about documents and attorneys
00:05:42.120
processing documents and
00:05:44.479
track through multiple versions of the
00:05:46.800
file which we'll all touch on as we get
00:05:49.139
into this more
00:05:51.539
um
00:05:52.620
so the first thing when we were going
00:05:54.419
back and looking through kind of um our
00:05:57.960
decisions on what did we do right what
00:05:59.880
did we do poorly
00:06:01.259
um
00:06:01.860
the fact that we um came out with those
00:06:04.500
hundred ten thousand lines of Ruby
00:06:07.560
testing within our spec um you know
00:06:10.020
proved to be um
00:06:12.180
very useful to us and then we started to
00:06:15.120
think
00:06:16.199
um back early on we were three engineers
00:06:18.720
and we got up to five we were
00:06:21.600
um you want to generate Revenue you want
00:06:23.340
to prove you have product Market fit
00:06:25.020
you're don't have any concerns about
00:06:26.759
operating at scale you're just concerns
00:06:28.919
about am I going to exist in a year type
00:06:30.660
of situation and how did we come up with
00:06:34.680
the space to actually come through and
00:06:37.500
have decent testing patterns and what we
00:06:39.960
realized was we were actually in
00:06:41.880
retrospect fairly pragmatic about this
00:06:44.360
and very much unintentionally so so
00:06:46.979
hopefully some other folks can be more
00:06:49.020
intentional about this are when you look
00:06:51.360
through kind of our git history and I
00:06:53.100
talked to some of the older hands
00:06:54.860
initially we only had our spec tests for
00:06:57.419
the rails API
00:06:59.400
um you know so does if you call this
00:07:02.160
controller does Json appear does the
00:07:04.080
Json you um expect appear and then
00:07:07.380
obviously you're inferring that your
00:07:09.060
front end is going to take that
00:07:10.259
correctly and you're really hoping that
00:07:12.720
things are also ending up in your
00:07:14.460
database correctly
00:07:16.440
so
00:07:17.720
that came out though that actually let
00:07:19.800
us get out the door pretty quickly and
00:07:21.539
be pretty stable we brought in you know
00:07:23.699
copy Bara tests came later and then
00:07:27.000
um Ember CLI and simple Cove came in
00:07:30.360
even later for actually testing test
00:07:33.360
regression today we're working on we're
00:07:37.199
using our spec tests for a lot of kind
00:07:39.479
of full-on feature tests where we're
00:07:41.099
checking to make sure things are
00:07:42.720
rendered on a page and that's why I have
00:07:45.360
this lovely code snippet here which we
00:07:47.400
did not come up with so I made sure to
00:07:49.500
um call out Ryan big whose blog post we
00:07:53.039
stole this off of whenever you're trying
00:07:55.199
to work with Ajax copy bar doesn't like
00:07:57.780
to work for it or at least the version
00:07:59.580
of copy we have doesn't so we have a
00:08:03.180
great little counter system here where
00:08:05.639
um basically check to see if the Dom is
00:08:10.139
in the ready State and reporting that
00:08:11.460
everything is returned if not go ahead
00:08:13.800
and sleep for a little bit and we'll um
00:08:16.500
you know keep sleeping for up to five
00:08:18.780
seconds unless you manually override
00:08:20.520
that which obviously is um
00:08:23.940
going to have some long-term
00:08:25.259
implications as we go to scale by
00:08:27.300
building in mandatory wait times
00:08:32.039
you know and now today as we come into
00:08:34.260
six years of existence we're at the
00:08:36.180
point where we're working on forming an
00:08:37.740
automated testing team we still want our
00:08:39.539
Engineers to write automated tests but
00:08:41.820
we're actually kind of we survived we've
00:08:43.440
gotten to the size where we can have
00:08:44.820
full-on automated test Engineers
00:08:46.920
automated test manager and all of that
00:08:49.500
and but overall we've made it to 97
00:08:52.380
percent
00:08:53.399
um
00:08:54.060
coverage when we sit there and we run
00:08:56.160
simple carve and check for mislines and
00:08:58.740
a lot of that I think again comes back
00:09:00.300
to pragmatic early on and then as we had
00:09:03.240
resources going through and actually
00:09:05.580
doing it right
00:09:08.160
um
00:09:08.940
I also want to you know test talk about
00:09:11.459
some other tools we've brought in over
00:09:13.860
the years no we did not bring in a
00:09:16.440
Leatherman Multi-Tool to our code base
00:09:17.880
but being in Portland I figure we should
00:09:19.800
give a I should give a shout out to
00:09:21.300
Leatherman they did make our booth
00:09:23.279
possible by letting us open all our
00:09:25.560
boxes with the various knives
00:09:27.660
but um you know very very important but
00:09:31.500
um you know we didn't start out with um
00:09:33.720
much just beyond the standard Ruby on
00:09:35.760
Rails stack and um I think one of the
00:09:38.940
really big things we've had is we've
00:09:41.640
added in um break man for vulnerability
00:09:44.040
scanners we've added in bundle audit for
00:09:46.560
spotting gems that need to get updated
00:09:48.300
and of course we've added in Robocop and
00:09:51.899
I would call out RoboCop as another area
00:09:54.120
where we had our similar pattern of
00:09:56.160
start out very small we just started out
00:09:57.839
with vanilla rubocop and now we actually
00:10:00.060
write our own RoboCop rules and it was
00:10:03.000
something that we didn't get into until
00:10:04.140
a couple years in
00:10:06.779
and I've come off the slide
00:10:09.600
and now what's really coming to
00:10:12.060
important is kind of the usage of those
00:10:13.380
tools our CTO from day one lane had a
00:10:16.980
really strong focus on developer
00:10:18.959
ergonomics and that extends just beyond
00:10:21.360
oh everybody gets a nice laptop and a
00:10:23.459
nice
00:10:24.300
um monitor he was always an advocate for
00:10:27.720
you know having Tools around testing
00:10:29.339
linting and vulnerability scanning
00:10:32.779
and fundamentally what we've realized is
00:10:36.420
well by having the advocate for those
00:10:38.760
tools by having those we kind of
00:10:42.180
unintentionally again locked ourselves
00:10:44.579
on to a lot of community practices we
00:10:47.519
brought in Robocop and we just did
00:10:49.260
whatever kind when you know robocops are
00:10:51.120
reporting violations so we just we just
00:10:53.279
fixed them with RoboCop Dash a
00:10:55.700
Brakeman came in and started spotting
00:10:58.800
injection risks so we fix them
00:11:02.240
and bundle started coming in and telling
00:11:04.920
us best practices on the gems and
00:11:07.620
basically rather than trying to come up
00:11:09.420
with our own patterns for the monolith
00:11:11.100
we were just accepting the community
00:11:12.600
patterns and that really paid off
00:11:14.220
dividends by limiting the number of
00:11:17.579
patterns that could evolve and that's
00:11:19.320
going to be a theme I'm going to keep
00:11:21.120
hitting on as we um
00:11:23.220
kind of continue to run through this
00:11:25.980
talk
00:11:28.140
um
00:11:29.160
that of course you know we have our
00:11:31.260
initial code base we've had all these
00:11:32.820
tools and you know it's great we have
00:11:35.459
these
00:11:36.680
tools you know we can pass all automated
00:11:39.120
quality checks but we also need to go
00:11:40.860
ahead and meet the needs of the business
00:11:42.180
and we've seen a lot of you know growth
00:11:44.760
in terms of activity from the users on
00:11:47.040
the site that prevents its standard
00:11:49.019
scale problems all of those pesky n plus
00:11:51.959
ones we failed to catch have come back
00:11:55.200
and bitten us at various points in our
00:11:57.060
life but what I really what I want to
00:11:59.279
dwell on here I think has been really
00:12:00.600
critical is we've had a lot of
00:12:03.240
complexity increase in terms of the
00:12:05.040
Persona of the person using the site
00:12:07.980
um
00:12:08.880
ultimately we're allowing lawyers to
00:12:11.160
process documents for companies so you
00:12:13.920
have the lawyer coming in and trying to
00:12:16.260
process the document you have your
00:12:18.300
Banker that would like their document
00:12:19.980
but we've had other people come in we
00:12:22.800
have say the compliance departments that
00:12:25.380
some of these Banks would like to come
00:12:26.579
in and get Consolidated reports of all
00:12:28.440
of their obligations so now that's a new
00:12:30.600
user Persona we've picked up initially
00:12:33.180
we thought this would just we would just
00:12:35.279
kind of have independent lawyers that
00:12:36.839
wanted to go out the one I always like
00:12:38.579
to talk about is she lives in Cancun
00:12:41.459
Works about 20 hours a week for us which
00:12:43.740
pays for a beachfront place in Cancun
00:12:45.959
with all the surfing you can do when you
00:12:49.079
you know get to build what a lawyer gets
00:12:50.760
to Bill out we thought that would kind
00:12:52.079
of be the person we'd have in reality
00:12:54.300
we've discovered a lot of lawyers are
00:12:55.980
starting to set up full on law firms and
00:12:58.860
they like our business because it allows
00:13:00.779
them to have some guaranteed input you
00:13:04.920
know well they're out trying to get
00:13:05.820
cases for their local Court they can
00:13:07.620
commit to XI hours of not as exciting
00:13:10.920
document processing per week so we've
00:13:13.980
had their legal Staffing team start to
00:13:15.720
come on and we've had to expand out of
00:13:17.820
the business logic of our code base
00:13:19.500
we've had a lot of those Ruby lines come
00:13:21.779
out of that and a lot of what we've
00:13:23.940
realized is thinking back is we were
00:13:25.740
actually really lucky for keeping a lot
00:13:27.300
of our models clean
00:13:29.040
um we also have more
00:13:31.860
complexity for internal customers we
00:13:33.660
didn't we definitely have a data science
00:13:34.860
team When We Were Young early on you
00:13:37.079
wanted to report you talked to an
00:13:38.339
engineer and they dumped something out
00:13:40.620
probably in a CSV file and that was
00:13:42.540
called the data warehouse back into good
00:13:44.820
old day and then of course there's just
00:13:47.100
new customer features like hey I want to
00:13:49.500
integrate with your API versus some sit
00:13:52.200
there and look into web app and as we've
00:13:54.660
gone through and had to address all of
00:13:56.220
these I've pulled out a couple case
00:13:57.600
studies that I think are interesting for
00:14:01.139
kind of this pattern a pattern that I
00:14:03.420
think is really um come through with us
00:14:05.399
initially we were all active model
00:14:07.620
serializer for Ruby on Rails everything
00:14:10.860
we did you know amsed right up to the
00:14:13.740
right up to Ember and
00:14:16.800
then we started to run into scaling and
00:14:19.260
performance problems there
00:14:21.540
and what we came out we looked towards
00:14:24.180
the community again and
00:14:26.339
we actually came towards Json API
00:14:29.100
serializer which comes out of
00:14:32.160
you may have heard of it used to be
00:14:33.779
supported by Netflix as fast Json API
00:14:36.600
they have deprecated support as kind of
00:14:40.440
a ironic aside our current VP is from
00:14:42.660
Netflix and he was telling me hey I wish
00:14:44.880
I had some code to work on and I was
00:14:46.380
really excited because I was about to
00:14:47.760
give them all the tickets we want on the
00:14:49.620
Json API serializer since it was the
00:14:51.480
next Netflix code base only discovered
00:14:53.579
he never contributed to that code base
00:14:55.139
and that dashed all my hopes there but
00:14:58.380
um
00:14:58.920
the process we kind of unthinkingly
00:15:00.959
evolved here that I want to um advocate
00:15:03.420
for is right now we have 117 active
00:15:07.139
model serializers and 106 Json API
00:15:10.620
serializers and we brought in Json API
00:15:13.199
because we were running into the
00:15:14.820
performance problems Pages were loading
00:15:16.620
slow we had people connect directly to
00:15:18.420
our API and they were running into
00:15:20.760
timeouts and there was that initial urge
00:15:23.940
that we all had for great Json API is
00:15:27.060
new it's the exciting new thing let's
00:15:29.279
just rip out all of the AMS serializers
00:15:32.279
in one go which would have been a lift
00:15:35.220
of about 200 at the time I know the
00:15:37.500
number is higher than that now but
00:15:38.699
that's because we've added more and
00:15:40.920
there was no way we were ever going to
00:15:42.839
get approval from product pragmatically
00:15:45.000
to do all of that work
00:15:46.800
and so we've kind of instead as an
00:15:49.680
engineering team we came up with rules
00:15:52.279
first off you have to fix the
00:15:55.320
serializers that are too slow and the
00:15:57.540
customers are complaining about you can
00:15:59.940
we also brought in datadog as a tool by
00:16:01.920
this point so we can start to see some
00:16:03.420
of our P95 and P99 times for slow loads
00:16:07.380
so we said let's go after those as well
00:16:09.839
and we started to build a case for
00:16:11.820
replacing those for um everything else
00:16:15.060
so we just left it and we evolved two
00:16:17.820
very firm rules if you're going to do a
00:16:20.579
significant refactor to a controller or
00:16:24.420
a model you're also expected that's
00:16:27.060
probably going to mean you're going to
00:16:27.899
touch the serializer at that point you
00:16:29.940
should convert to serializer over to
00:16:31.680
Json API and then anything new you write
00:16:35.399
also Json API serializer and that
00:16:39.720
actually that actually let us address
00:16:41.279
the tech debt in a what I would as a
00:16:43.860
relatively scalable way where we've you
00:16:46.620
know we're about 50 percent converted we
00:16:49.139
brought Json API in about 14 months ago
00:16:52.199
and a lot of the AMS ones we have are
00:16:54.899
kind of in dark corners for internal
00:16:56.639
admin tooling and their existence
00:16:58.440
doesn't really hurt us it's in a fairly
00:17:00.480
stable section of the code base and
00:17:03.180
compared to some of the other places
00:17:04.740
I've
00:17:07.079
you know been at I've been very happy
00:17:09.120
with like how it's um it hasn't just
00:17:11.640
been something that's let us solve our
00:17:12.959
problem it's kind of preserved our
00:17:14.339
relationship with product in terms of
00:17:16.919
where we can articulate what we're
00:17:19.380
delivering and why we're delivering it
00:17:22.559
oops and I need to get my mouse back
00:17:24.179
over here
00:17:25.740
the other one I like to talk about is
00:17:28.340
enums and this kind of shows another
00:17:30.660
thing you see when you look at this
00:17:32.400
whole monolithic code base if you our
00:17:35.520
initial way for doing enums was the
00:17:38.160
simple enum Gem and that was I think I
00:17:41.100
would say probably the Ruby Community
00:17:42.299
standard back in the day right up until
00:17:45.799
rails 4.1 came out which introduced our
00:17:49.440
active record enum
00:17:51.799
and we were on active record enum up
00:17:55.440
until last summer active record enum was
00:17:57.960
scaling great no problems we had the
00:18:00.539
same method where the older simple
00:18:02.640
enumes hadn't been converted anything
00:18:05.460
that we were doing now was active record
00:18:09.179
enum and we started to hire a data
00:18:10.799
science team and this was a case where
00:18:12.720
we had an issue with internal customers
00:18:15.179
where I started to get questions like
00:18:17.640
hey what does um email frequency to mean
00:18:21.419
what does
00:18:23.340
um you know user content user preferred
00:18:25.500
contact method three mean to a data
00:18:28.500
science team pulling that into a
00:18:29.940
warehouse that was um you know very
00:18:33.600
um
00:18:34.280
not useful
00:18:37.200
um so initially there was kind of that
00:18:39.720
push we all had where it was Oh What if
00:18:42.240
I just uh give you a dictionary where
00:18:43.799
you can map those integers into
00:18:45.720
something meaningful that um got pushed
00:18:48.539
back on as the data science team did uh
00:18:50.520
you know rightfully so and
00:18:53.520
so we started we built our actually our
00:18:55.620
own enum gem in-house that we call ICC
00:18:57.840
enume because we also used to be called
00:19:00.179
in Cloud Council we've since renamed
00:19:02.340
ourselves but uh refactoring all those
00:19:04.500
iccs out of the code base that's going
00:19:06.299
to be a
00:19:07.380
you know hole on the thing and
00:19:11.760
um it's been it's been very successful
00:19:13.620
but what it's been interesting in is
00:19:15.900
it's again the same pattern where we've
00:19:17.700
stayed under rails Community patterns
00:19:19.200
and just followed them up we hadn't
00:19:21.059
ripped the simple enums out we still
00:19:23.520
have about 90 of them floating around we
00:19:26.400
did end up going ripping out all of the
00:19:28.380
active record he names because we only
00:19:29.640
had about 20 at the time that data
00:19:31.919
science
00:19:33.360
um spoke up so it was like well we can
00:19:34.919
just go in and clean those out really
00:19:36.360
fast and get ourselves down to two
00:19:38.820
patterns to maintain
00:19:45.840
oh
00:19:47.640
so
00:19:49.799
um fundamentally the the point we're
00:19:51.900
trying to I keep trying to hammer home
00:19:53.700
here is we've kind of capped ourselves
00:19:55.620
with the number of patterns we have by
00:19:57.299
staying on the rails Community patterns
00:20:00.360
you go up you um you know it's been
00:20:02.520
really great
00:20:03.960
um you know you look at them you go look
00:20:05.820
through our code base like oh yeah
00:20:06.960
that's how you did it back when rail
00:20:08.100
three was a thing oh yeah that's how you
00:20:10.080
did it when rails 4 was a thing and the
00:20:11.580
fact that we didn't get creative really
00:20:13.500
served us well and no we can't claim
00:20:15.240
that we had the foresight to sync that
00:20:17.760
through but we're pretty happy with that
00:20:20.700
um that being said there's also you know
00:20:22.320
what would we um
00:20:24.360
improve kind of coming through here and
00:20:26.520
what did we what were we too slow to do
00:20:28.020
is a mid-sized company that was
00:20:30.059
converting out
00:20:31.500
um
00:20:33.000
so the biggest one we had when we came
00:20:35.880
back is the lack of an architectural
00:20:38.220
decision uh um process we um very early
00:20:42.780
on as we rightfully they said you know
00:20:45.240
small team of Engineers every engineer
00:20:47.100
has to be a guardian of our architecture
00:20:50.160
um that worked really great when we were
00:20:51.780
probably one team when we were two teams
00:20:53.580
with very SEP with very strong
00:20:54.960
separations of concerns you know did a
00:20:58.200
um really great job
00:21:00.600
um
00:21:03.480
whoops
00:21:05.340
it's like One clicking but we um you
00:21:09.179
know really
00:21:10.679
um didn't do a great job we started
00:21:12.120
thinking about the holistic code base
00:21:13.860
people had strong patterns for solving
00:21:15.480
their um internal problems for example
00:21:18.480
we were really bad with State machines
00:21:21.900
um we have a fun lunch topic of how many
00:21:24.480
state machines do we have do we have two
00:21:27.000
do we have three do we have two and a
00:21:28.799
half and arguably when you're at the
00:21:31.140
point where you're debating how many
00:21:32.400
state machines you have in your
00:21:33.720
application you've um you know it's kind
00:21:36.120
of academic that something has gone
00:21:37.620
wrong at that point
00:21:39.240
um a lot of our state machines are based
00:21:41.100
off of um kind we we track the versions
00:21:44.400
of our documents we track the status of
00:21:46.919
their documents and we track other kinds
00:21:49.860
if you've used rails a lot you've seen
00:21:51.240
kind CD which is a a new for tracking
00:21:55.380
out kind with an integer number and then
00:21:57.539
from there we kind of in first date but
00:22:00.120
at no point did we ever build anything
00:22:02.400
where you can just call thing.state
00:22:05.600
thing.states or the one I really missed
00:22:08.340
from previous code bases was passed in a
00:22:11.580
state with an as of time stamp that
00:22:13.380
tells you what state it was and at a
00:22:14.940
certain time for an audit State and in
00:22:18.840
hindsight that's something you know
00:22:20.159
especially dealing with a lot of
00:22:21.419
business logic we wish we'd pushed for
00:22:23.820
early on it was always tricky because
00:22:25.919
when we go back and we look at it the
00:22:28.200
products back was build a thing to do X
00:22:30.780
that thing had some states and then
00:22:33.600
going back and having to talk about
00:22:34.919
let's build a whole state machine up
00:22:37.020
from nothing would have been a extra
00:22:40.860
talk to product but in hindsight we
00:22:42.539
would have gone back and told ourselves
00:22:43.799
to fight that battle much earlier on and
00:22:46.919
push for some of that internal tooling
00:22:48.419
and unification rather than Let each
00:22:50.580
team solve their tracking problems
00:22:52.140
independently we also have a bunch of
00:22:54.179
other auditing gems that add in
00:22:58.799
so
00:23:00.539
um ultimately we've come to the point
00:23:02.640
where we have our you know we say we
00:23:04.559
have we would say we have strong
00:23:05.580
internal abstractions and really good
00:23:07.799
patterns each team we challenge them to
00:23:10.020
be a guardian of you know their
00:23:11.760
architecture they did a really good job
00:23:13.919
things like more we never had a team
00:23:16.799
whose mandate was a universal thing like
00:23:18.720
a state machine and we've suffered from
00:23:20.460
that we're also very tightly covered
00:23:22.679
with coupled with external vendors I
00:23:25.559
have a big lift on my hands the day that
00:23:27.360
a MailChimp finally shuts Mandrell down
00:23:29.760
for good that's gonna not that's going
00:23:31.980
to be a fun one to go through
00:23:35.120
what we've done and wish we'd done
00:23:37.380
earlier is we've started to form an
00:23:38.760
architecture team to focus more on
00:23:40.440
abstract inner
00:23:42.360
um Integrations
00:23:44.480
and in hindsight we wish we'd done this
00:23:47.940
earlier some of this comes back it steps
00:23:50.520
Beyond coding but I mean staff Engineers
00:23:52.980
principal Engineers they're expensive
00:23:54.600
going back and asking someone for the
00:23:56.460
money to hire one
00:23:58.620
um you know is it's a whole thing asking
00:24:01.919
plus you'll have to have some other
00:24:03.240
seniors pulled off onto the team they're
00:24:04.980
not going to be delivering immediate
00:24:07.140
value the company in terms of building a
00:24:08.880
feature but we wish we'd done it we
00:24:11.940
encourage everybody else out there when
00:24:13.679
you're coming up in that size you know
00:24:15.299
start thinking about having a team that
00:24:17.760
doesn't have a mandate to just deliver a
00:24:19.919
product feature think about having
00:24:21.360
somebody that has a mandate to deliver
00:24:23.100
tools internally in a very clean manner
00:24:28.500
um
00:24:29.900
we um you know
00:24:33.360
our pragmatic approach of refactoring
00:24:35.460
when needed served as well we were
00:24:37.020
always able to bring in new patterns
00:24:38.820
based off the community get them in
00:24:40.559
quickly whereas we never had to go out
00:24:43.380
and like sell a full-on lift and shift
00:24:45.539
of all 60 000 lines of the code to some
00:24:48.539
new pattern the product which made it a
00:24:50.700
lot easier to get Space very happy with
00:24:53.280
you know all of that sticking with
00:24:55.380
Community patterns versus having
00:24:56.880
in-house ones it's capped the maximum
00:24:59.760
potential number of patterns we've faced
00:25:01.500
it's also made onboarding really good
00:25:03.659
because all that documentation from
00:25:05.820
rails three rails four it's still up
00:25:08.460
there so when I have a new hire come
00:25:10.200
across um one of the dark Corners that
00:25:12.360
we haven't gotten back and fixed up yet
00:25:14.820
we can still find something to link to
00:25:16.559
them that's actually useful versus
00:25:18.840
having to keep four versions of our own
00:25:21.179
documentation up for patterns and as my
00:25:24.600
final bit before we get into q a I'll
00:25:26.100
get myself ejected from rails conference
00:25:29.100
but um our spec test served us very well
00:25:32.039
early on
00:25:34.559
um but we should have gotten into Ember
00:25:36.120
tests a lot sooner we were trying to use
00:25:39.480
um that one hammer for every single nail
00:25:41.460
it's really slowed our test Suite down
00:25:43.320
as I mentioned earlier with that weight
00:25:45.480
for Ajax you have mandatory you're
00:25:47.940
pretty much guaranteed to be waiting a
00:25:49.559
minimum of a second for something to
00:25:51.240
happen we really should have pushed to
00:25:53.700
write Ember tests earlier in speed to
00:25:55.440
test Suite up it's a big shift and um
00:25:58.140
yeah love our spec don't um we'll
00:26:01.740
continue to use our spec but we
00:26:03.240
shouldn't have just said oh we're a rail
00:26:04.919
shop so let's just use our spec to you
00:26:07.799
know make sure things are rendering on
00:26:09.120
our front end pages
00:26:10.799
and with that we have a few minutes left
00:26:13.320
so I want to thank everybody and ask any
00:26:14.880
questions on for folks might have
00:26:18.779
hello
00:26:21.299
um
00:26:22.020
I find one of the hardest parts of
00:26:23.700
maintaining a majestic you know monolith
00:26:25.860
is the upgrade process which you never
00:26:27.900
touched on so I'm very curious what was
00:26:30.600
your upgrade process like and um would
00:26:33.059
you do something differently yeah that's
00:26:34.559
a good call that's a great call I should
00:26:36.779
um so we've um we've stayed very highly
00:26:39.419
aggressive on upgrading the minute
00:26:41.340
appoint release of rails comes out we
00:26:44.340
task one of our teams onto it I actually
00:26:48.120
um when I joined I joined during Peak
00:26:50.460
pandemic
00:26:51.919
and we were fairly behind on our upgrade
00:26:55.679
so myself and the principal just sat
00:26:57.900
there for those two weeks around
00:26:59.520
Christmas and New Years when everybody
00:27:01.799
else was off and the code base was
00:27:03.480
stable and got ourselves up and just
00:27:05.400
said we're never going to let ourselves
00:27:06.779
fall behind again and that served us
00:27:10.080
really well in these past couple years
00:27:11.400
but before then we were very guilty of
00:27:14.039
letting things sit for a while until we
00:27:16.679
got to end of life and I would say only
00:27:19.080
just the minute something drops get an
00:27:21.059
MR up see see what breaks and hopefully
00:27:23.700
what breaks is going to be fresh in
00:27:25.260
people's minds because they've been
00:27:27.480
working on it recently and tag your
00:27:29.520
whole team and just to get DMR through
00:27:30.960
within a day or two of it coming out
00:27:32.400
well thank you everybody you've been a
00:27:33.960
wonderful audience and thanks for your
00:27:35.520
time and attention