00:00:00.900
foreign
00:00:12.799
well uh looks like this is a pretty popular talk we could wait a few minutes for more people to trickle in but I
00:00:18.840
think we'll go ahead and get started um thank you so much everyone for coming to my talk uh I know there's a lot of
00:00:25.320
really great presentations especially in this time slot and I really appreciate you guys taking the time and uh coming
00:00:30.960
in having this conversation with me the talks titled you have two seconds to respond
00:00:37.020
um and I know what you're all thinking oh that's that guy that had that absurdly large mask walking around
00:00:43.020
and now you know why it's because I'm deathly afraid of my beard catching coven if it has to isolate for two weeks
00:00:49.260
it's really going to damage our working relationship I don't know if any of you got to check
00:00:54.960
out the lightning talks they were they were downstairs for a good portion of the day yesterday um
00:01:00.780
there was a fellow towards the beginning that had some like really great ideas for for sale like he was selling ideas
00:01:06.119
on like how to give your talk and stuff he uh you know I had some fun ones like uh
00:01:11.640
you know I'm not gonna buy I didn't buy any of them but he had some fun ones like having some plants in the audience like yell stuff out during your talk
00:01:24.299
if anybody knows who that was I need to go find him I guess I owe him something so I'm speaking on behalf of a to B uh
00:01:31.320
it's pronounced A to B as in we're allowing trucks the ability to go from point A to point B
00:01:37.259
uh and A to B we're building the financial infrastructure for commercial fleets
00:01:43.979
um to get a little more specific our Flagship product is a fuel card for Trucking fleets
00:01:50.700
um so that as these trucks are going down the road if they need to stop and get gas they swipe our fuel card uh to
00:01:57.180
purchase their fuel and there's a few reasons why this is needed in the industry uh Legacy fuel cards uh don't
00:02:03.720
work at most gas stations you have to go to very specific gas stations um that very limited tooling they're
00:02:09.300
kind of unreliable most truckers I've spoken to really despise their legacy fuel cards and so we're kind of
00:02:15.300
Reinventing it and we're Reinventing it on the Visa Network so that it'll work at any gas station not just specific gas
00:02:20.700
stations uh we also uh don't charge any fees and
00:02:26.220
not only do we not charge any fees we also provide discounts so they're actually saving more money when they use our card
00:02:31.260
and we offer a lot more tooling and we can get a lot more sophisticated about how our customers can use our fuel cards
00:02:38.040
and they can use it for other expenses such as maintenance uh repairs stuff like that
00:02:43.739
we've also launched a payroll product um it's a great way for uh you know our
00:02:49.860
truckers who many of which are underbanked they don't have access they have small banks in rural areas or they
00:02:56.040
don't have banks at all and so our payroll product allows them to get paid directly while they're on the road
00:03:03.000
without having to deal with paper checks or maybe getting mailed back to their home so that's a really great product as well and we're we're continuously adding
00:03:10.920
more financial products to our our portfolio um we've had quite a journey A to B has
00:03:16.440
only been around for about a year now and we already have over 25 000 Transportation businesses that are using our products uh and we've got about 20
00:03:24.300
to 25 month-over-month growth uh we raised a series a and series B
00:03:30.180
um with over 100 million dollars in Venture Capital rates so far also we're hiring
00:03:36.780
uh just a brief bit about me I'm Justin Powers I think depending on where you saw the talk and where was printed or
00:03:43.440
online you might have seen like three different names I'm Justin Powers I'm giving the talk today I've been working
00:03:49.260
remotely in the southern Sierras for the last decade or so in fact I enjoyed it
00:03:54.540
so much having being in this like outdoor playground that I created a co-working space so other people could work where they where they're playing so
00:04:00.180
feel free to come join me in Kernville spend all the time you'd like
00:04:05.599
let's change gears and talk about the the title of the talk so uh we called it you have two seconds to respond and what
00:04:11.519
I'm what we're referring to when we say that is when the customer is at the gas station they swipe their card in the uh
00:04:17.940
at the fuel pump uh Visa gives us two seconds to respond with an approval a
00:04:23.639
denial in some cases how much we approve it for um you imagine if they haven't actually
00:04:29.400
pumped the gas yet uh we don't want them to pump an unlimited amount depending on how much credit they have available we
00:04:35.100
can uh we can authorize a very specific amount um and this one endpoint this endpoint
00:04:42.660
that Visa hits when uh they're looking for that approval or denial is unquestionably the most critical part
00:04:48.780
of our infrastructure you can imagine that if we if we don't respond within two seconds what's going
00:04:53.880
to happen uh the way we have it set up is it'll automatically prove the transaction uh that opens us up to a lot
00:04:59.940
of risk uh opens us up to a lot of fraud so like we just can't have that we can't uh let these transactions Auto approve
00:05:08.120
if we make some mistake and we're approving transactions that shouldn't that we shouldn't be approving then that
00:05:13.680
opens us up to a lot more risk as well opens us up to fraud um and what's even scarier to me is if
00:05:19.860
we decline transactions that we should be approving and what that can lead to is a trucker that's stuck on the side of
00:05:25.800
the road without a way to get to the next gas station because they can't get any fuel so
00:05:31.080
um that one in particular is my personal nightmare I don't want to when I'm pushing out changes and and and working
00:05:37.440
on this infrastructure it it keeps me up a little bit like thinking like man what would happen if I mess something up and some poor person's
00:05:44.100
out in the middle of the night at 3am you know trying to just get across the country and they can't because our our fuel card's not responding
00:05:51.180
so this endpoint can't be blocked for any reason um and what that means is it creates
00:05:58.020
some very specific constraints around this endpoint in a lot of different ways not just the time constraint there's
00:06:03.840
other constraints that we have to be aware of as well and what this talk is about is how do you build
00:06:09.660
um how when you have constraints how does that change the way that you build and the way that you deliver new
00:06:15.180
functionality some of the constraints we have to think about is constraints about the inputs
00:06:20.580
and and how they can vary or how they might be highly constrained about the outputs of whatever functionality you're
00:06:27.360
making sometimes you have very specific constraints around around that sometimes you have time constraints
00:06:33.020
oftentimes we're kind of Lucky and uh you know we we have uh large uh
00:06:40.199
leeway in that but in some cases we do have you know in this case we have a very specific time constraints and then
00:06:45.419
correctness which is the one that we usually focus on you know what are the consequences if something goes wrong is that going to directly impact your
00:06:51.180
business and how do you build things if that is a very uh you know tight important constraint
00:06:57.419
so talking about the inputs um for this card swipe that we're
00:07:04.199
talking about for this event somebody swipes the card and we get a request from Visa um there's actually two different kinds
00:07:10.500
of inputs that we're concerned about uh Visa gives us some input they say hey uh
00:07:16.199
there was a we have an authorization request that's coming from a merchant with this name it might have a merchant
00:07:21.780
category code that tells us what kind of merchant it is because as I mentioned before we can our cards work for fuel
00:07:27.300
fuel stations for maintenance repairs but like you can't go and buy something off Amazon with the cards they're
00:07:33.120
they're highly restricted based on the merchant category um sometimes it comes with a dollar
00:07:39.060
amount sometimes it doesn't sometimes it comes with a dollar amount but uh Visa wants us to actually give
00:07:44.699
them a proper dollar amount for how much we would authorize um but then other sources of inputs uh
00:07:50.699
are the actual state of the customer and and their account so do they have certain Merchant category uh categories
00:07:57.539
that we've enabled for them do they have you know how much available credit do they have
00:08:02.940
um and that part's a bit more complicated than you might expect you you might think it's a very simple calculation to just look at how much
00:08:08.639
they've spent this month and how much they have left but uh our payment terms we're on a what's called a net seven
00:08:15.240
so at the end of the week we look at what your balance was how much you spent during the week and you make a payment
00:08:22.500
you have a week to make a payment on that which changes how much credit you have for the next week and so it's a little bit complicated and it's not
00:08:29.400
super straightforward and there's a lot of edge cases in there that really change what your available credit might be
00:08:37.320
um we're talking about outputs in our situation uh
00:08:42.599
it's there's not as much of uh of a consideration outputs aren't quite as constrained we do want to pass back some
00:08:49.500
metadata to visa when we're approving these transactions and if we push incorrect metadata or if
00:08:57.180
it's formatted incorrectly it'll raise an error in that and and the transaction gets declined so we we have to be very
00:09:03.959
very careful about any change we're making making sure that we're not changing something accidentally that's going to
00:09:09.720
cause those transactions to get declined um but also of course with the outputs
00:09:15.180
you have to worry about it they're correct we've got a large variability in inputs we've got a large variability in all the different edge cases and and uh
00:09:22.200
in our customers accounts and how much they paid and when they paid and extra payments they've made to try to raise their their credit limit
00:09:28.680
um so there's kind of a lot more than kind of your your you know typical test
00:09:33.839
cases can can easily cover there's a lot of edge cases that we're not not aware of until we actually see them in production
00:09:40.800
um and as I alluded to we have a hard constraint of two seconds um first I want to address the elephant
00:09:46.320
in the room in the in the world of web apis two seconds is like forever right
00:09:51.899
um especially if you go to uh you know any other conference other than railsconf if you tell them that uh you know we're talking about how to keep an
00:09:57.720
API endpoint under two seconds they'd probably laugh you off the stage I'm not saying that we're uh we're doing anything
00:10:03.360
um you know Innovative Innovative by consistently being under two seconds uh but what I do want to talk about is is
00:10:09.060
how those constraints actually change how you design your system and and how you uh how you deliver that functionality
00:10:15.779
so um I mentioned this is the most critical part of our infrastructure but it's also
00:10:20.880
the part of our infrastructure that we we want to invest the most time in we want to make the most changes in we want
00:10:27.060
to really um innovate in the way that we uh that we approve or deny these transactions
00:10:33.660
um for example um we want to add a feature and we have that at a feature where if your truck's
00:10:39.540
in California but the car is swiped in Tennessee we can say oh well that wasn't your uh
00:10:44.579
you know that wasn't you must have been like filling up your uh your personal vehicle or something like that and we can do that because every truck by law
00:10:51.300
in the United States has to have a tracking device in it and we can integrate with those tracking devices and so we can actually get down to the
00:10:57.480
level of making sure that the truck is within two miles of the gas station where the car was swiped
00:11:02.579
um but there's a lot of complexity there's a lot of edge cases there's a lot of situations where maybe the the
00:11:08.760
location of the merchant is like miscoded uh from uh from visa and and they're actually somewhere else across
00:11:15.060
the country there's a lot of uh you know weird situations there uh we have a feature where um companies
00:11:20.820
are a little more security they can unlock they can send an SMS to A to B to unlock the card for the next 30 minutes
00:11:27.540
uh that's a great protection against fraud we have other features like allowing each driver to have an individual pin
00:11:34.500
that's specific to the card um and we're we're integrating things like machine learning and and automated
00:11:41.279
fraud detection into our authorization uh into this endpoint so
00:11:46.800
as we're adding all these features if we are terrified to make changes because it might leave a driver stranded on the
00:11:53.519
side of the road how do we how do we change the way that we build things so that we're a lot more confident in that
00:12:11.640
so there's a number of things that we can do to address
00:12:16.700
being able to uh to be able to make changes safely and
00:12:22.980
with a lot of confidence some things you want to kind of look out for when you're building and some things
00:12:28.500
that that we've been looking at is isolation so we don't want to allow a failure one part of our system to take
00:12:33.839
out the rest of the system if we have a if we have one part of our system that is being really hard on the
00:12:39.959
database or is you know has a memory leak or something like that you don't want it to impact uh the uh the part of
00:12:47.040
your system that's most critical there's a lot of strategies for this there's entire talks about this you can do things like
00:12:53.040
adding redundancy adding caching you can split up your application to have your most critical components run on separate
00:12:58.680
Hardware you can do all these things regardless of whether you're working on a monolith or working in microservices there's a lot of strategies for that
00:13:04.740
we're not really going to get into them in this talk though something else that just makes it really
00:13:10.320
easy to be very confident in the changes you make is
00:13:15.360
is be very very devoted to the single responsibility principle build your
00:13:21.180
classes build your code in such a way that your code's only doing one thing um
00:13:26.820
one one piece of advice I would definitely give try not to mix um your business logic with your models
00:13:34.079
or your database layer if you you want to be able to have specific inputs
00:13:39.300
and specific outputs that you can test without having to worry what the state of the database is in the middle there
00:13:45.000
it makes a lot of things a lot easier including some of the strategies that we're going to be getting at towards the end of the talk
00:13:52.200
um I could spend a whole talk on just designing your classes to make them easy to test but I think there's been some
00:13:58.019
other talks this week that have covered that pretty well um and if you do this it makes testing
00:14:03.720
things in isolation a lot easier makes a lot of the things we're going to be talking about later a lot easier and
00:14:09.060
there's actually another talk going on at the same time that I believe probably addresses us pretty well about kind of a
00:14:14.279
functional programming style that would make this a little bit easier as well
00:14:19.440
um another thing that you can think about is um uh is making sure that for those
00:14:25.260
business critical portions only do the work that you actually have to do right at that time if there's anything that being done later use some sort of async
00:14:32.820
process there's a lot of tools out there for this I'm personally like to fawn
00:14:38.639
over a sidekick Mike perham's here he's been a great part of our community great part of railsconf and it's a really
00:14:45.540
great library to make sure that like for example in our uh in our system when you
00:14:50.820
swipe your card I mentioned you get discounts there's a little bit of calculation that happens there we're reaching out to some third-party vendors
00:14:56.639
we don't want to block that two second transaction so we do that later we kick off a sidekick job and just say we'll deal with that later
00:15:05.279
um and concurrency and this is uh this is where it's going to get a slightly uh perhaps slightly controversial one of
00:15:10.740
our primary constraints is time um but there's as I mentioned a lot of different things that we're doing at the
00:15:16.019
same time uh to make sure that this uh transaction is valid um as I mentioned we at the same time we
00:15:23.100
need to calculate their running balance calculate how much they've uh they have available this week but we're also
00:15:29.220
checking the location of the truck we're checking uh the location of the uh vendor we're checking to see if they've
00:15:35.639
unlocked the card using SMS um and so we need a mechanism that as we're adding more checks it doesn't
00:15:42.000
increase the amount of time it takes to respond uh to Visa and so in our case we decided the best
00:15:47.279
way to do these would be to run these checks in parallel um word of warning concurrency makes
00:15:54.420
things complicated um the to quote the uh the great
00:15:59.940
philosopher Aaron Patterson just because you can doesn't mean that you should
00:16:05.639
um so I I believe we should all strive to keep our systems very simple very easy to understand
00:16:12.480
um very easy to maintain and improve multi-threading is fantastic for Frameworks
00:16:18.000
if you add it to your application code you're adding complexity that's going to be difficult to uh to mitigate and to
00:16:24.779
work around and it adds risks um Clara and I'm going to mispronounce
00:16:31.560
her name I believe is Morgan iron on Tuesday did a great talk about uh using concurrency and Ruby a lot of the
00:16:37.740
different strategies for it if you get a chance to watch it uh go back and watch it on YouTube it was a very good talk
00:16:43.820
she mentioned some of the different strategies you can use for concurrency and some of the the typical bugs you
00:16:48.959
might run into and some of the risks um there's uh some there's uh you know
00:16:55.019
using thread.new which has been around a long time there's newer features in Ruby like uh fibers and our actors uh we're
00:17:01.440
using thread.new but we are actively looking to see if uh one of these other Solutions might be a good fit for the uh
00:17:07.140
for the job um some of the other things you need to be aware of if you're if you're looking at doing um at you know spawning threads
00:17:14.100
in the middle of your application code paying attention to the connection pools you have if if you've made a calculation about how many database connections you
00:17:20.459
need for the number of say puma workers you have then you start spawning extra threads and they start connecting the database those are going to get out of
00:17:26.520
control so you need to be aware of that um be aware about uh shared context you
00:17:32.160
know shared State between those threads um and you have to be a lot more purposeful about observability
00:17:38.760
um for example if you're um we need to know now that we're now that
00:17:44.640
we're doing things in separate threads we need to know not just how long it takes for a uh for this endpoint to
00:17:50.460
return we also want to know how long each of these threads is taking so we can uh you know be proactive about dealing with issues
00:17:57.000
um and also testing is kind of functionality is a little bit difficult um especially if you don't want to rely
00:18:03.059
on sleeps to test things like timeouts um here's an XKCD about regex's but I think
00:18:09.240
the same thing probably applies to uh to threads I've got 99 problems so I use threads
00:18:16.260
and now I have 100 problems so just word of warning just be careful about that
00:18:24.120
so here's kind of a a quick sketch of of what that endpoint is doing
00:18:29.640
um in our situation so we've got this authorizer Runner that gets the request we spawn thread.new for each different
00:18:37.700
kind of sub authorizer that we have so we've got in this example a credit authorizer one that's looking at the
00:18:43.380
location of the truck one that's checking um the the SMS unlock
00:18:48.440
and anybody that's worked with Reds you've seen thread.join where you wait
00:18:53.820
for the thread to finish and you get the results of the thread um what I didn't realize until very recently is you can actually pass a
00:19:01.080
timeout to that that method which is a number of seconds so in this case at the
00:19:06.179
bottom here you can see and I don't think I have a mouse here oh yeah here you can see um we're passing 1.5 so we're saying
00:19:13.620
okay all of you threads have a one and a half seconds to finish if you haven't finished then you're going to give me
00:19:19.260
the results anyways and and you're going to keep going it doesn't not kill the thread and that's important
00:19:25.020
um because timeouts can do really dangerous things don't ever use Ruby timeout if you can if you can help it
00:19:31.500
um so you can see in this example let's say the credit authorizer took about a second
00:19:36.960
it said yes we want to you know based on their available credit we want to approve it we want to approve them for 500
00:19:43.220
you know worth of fuel but the location authorizer looked and it said hey this uh this truck is nowhere near the the
00:19:49.740
gas station it was gonna it says to deny it and something happened with the SMS
00:19:54.960
unlock it took too long it took over two it took over a second and a half we don't get that result we don't care
00:20:00.480
about that result we we use what we've gotten to make the best decision we can so we're going to deny it based on the
00:20:06.840
fact that the truck wasn't in the in the appropriate location um the the code that does all this is a
00:20:14.400
little bit Arcane and a little bit difficult to uh to do in a short talk like this so um we're going to skip it but um we're
00:20:21.539
gonna have some code at the end for for one of the other features that we're we've been working on
00:20:27.960
so in talking about working with constraints how do you how do you have the observability to know um you know if
00:20:34.380
the thing if that business critical part of your system how well is it working
00:20:39.539
um the biggest thing is going to be observability you have to be absolutely
00:20:44.880
certain that this critical part of your functionality is working you need to know how it's performing
00:20:50.760
um and for that monitoring is key uh we use a tool that's not sponsoring this
00:20:57.120
conference so I'm only going to mention the sponsors of the conference we've got some great Monitoring Solutions uh New
00:21:02.220
Relic I know is in the room they're a great solution for monitoring Scout is
00:21:07.860
also sponsoring the conference here and honey badger and you can use all of these to to get a lot better Insight of
00:21:14.580
of those critical portions of your your application and there's going to be some there's some great talks on on exactly
00:21:20.940
how to do that but just keep in mind that that's going to be absolutely key that you need to know not just how long
00:21:28.740
something's taken um but some of the other things that we're looking at is uh what's the ratio
00:21:33.900
of for example denials to approvals if we all of a sudden start approving 100 of our transactions something's probably
00:21:41.220
wrong there we should probably page the on-call and and roll back whatever code was just uh released if it's the other
00:21:46.620
way around if we're suddenly denying uh a large portion we need to you know page somebody and roll it back
00:21:53.400
um even like watching the approval amounts and having some anomaly detection as you start getting into these uh higher end tools like uh like
00:21:59.700
like New Relic um you get things like anomaly detection that they can kind of keep an eye on
00:22:04.860
that and see if it's uh if it starts if there's they start to see anything weird um and you can instrument very specific
00:22:16.380
um absolutely create some easy to use easy to understand dashboards that whoever is
00:22:23.940
going to be supporting your code especially any on-call they need to be able to look at that within a few
00:22:29.220
seconds know if something is wrong and and where where that's happening they need to be able to dive into that pretty
00:22:34.620
quick and so designing dashboards is not something that as developers we tend to spend a lot of time on but I
00:22:40.919
highly encourage you to spend a lot of time invest time in in building very easy very actionable
00:22:46.740
dashboards and if something goes wrong you should have proactive alerts you should be
00:22:52.980
paging somebody to say hey this critical part of our business uh is not working right now we need to get this under
00:22:59.159
control um and another strategy that we're using
00:23:04.799
and this one is this one's kind of fun and we're going to spend uh the rest of our time on it is uh Shadow mode so for
00:23:10.799
us just monitoring isn't enough I don't want to push code and then find out that it wasn't working and and the result of
00:23:17.400
that is that maybe some drivers are stranded on the side of the road that's not acceptable to me that's not acceptable to the company
00:23:24.360
um but we want to be continuously pushing new new functionalities so how can we do
00:23:29.400
that um one common strategy is called Shadow mode there's a few different ways of
00:23:34.500
doing this the way we've chosen to implement this is every time we get a request in
00:23:39.840
production we we handle it using the existing code and we we push it back up
00:23:45.179
we push the result back to Visa and then what we do is we kick off a sidekick job I'm going to keep mentioning sidekick
00:23:51.860
and we will run the uh with the same
00:23:57.419
inputs we'll run it against the existing code and the new code that we want to roll out
00:24:02.820
and then we can compare the the outputs um and and see if they're if if we're
00:24:10.020
making like refactoring changes they should be identical they should never change and we want to see what the difference is if they do change
00:24:16.140
if we're changing our Logic for for when we approve somebody we need to know exactly when uh one has changed and why
00:24:22.620
it's changed and be able to uh to have some observability around that um
00:24:28.740
and uh I mentioned before like single responsibility principle uh we're not perfect at it uh we're still improving
00:24:35.100
and refactoring things to to get our our things nice and tight while we're working on that it's kind of helpful to know what's going on inside of those
00:24:41.400
code and so we've actually added a little bit of tracing functionality where you can actually see what the state is of your logic as you're
00:24:47.940
stepping through the code and and some of the um some of the data and variables that's inside of your code
00:24:55.440
um and yes I contradicted what I said earlier but you know we're all human and uh and life is rough
00:25:02.039
um there's other way other strategies for doing this another popular strategy is to record all of your API traffic and
00:25:07.980
then replay it against a non-production environment and and compare the results make sure that's working fine um there's some other strategies we're
00:25:14.220
not going to get into but this one seemed to work best for us
00:25:20.220
um so before I get into kind of stepping through the fun uh the uh the code that we've written I just want to call out
00:25:26.820
these are two uh tools that we absolutely would not be able to do this without sidekick I believe is who here
00:25:34.440
is using psychic okay so needs no introduction um if you're not using the Enterprise version
00:25:40.260
I would encourage you to talk to your employers and have them uh pay for it and support that open source project uh
00:25:46.020
flipper is a a feature flag framework that also has an Enterprise version that's worth paying for
00:25:52.500
um uh and and is just a great piece of uh software that we're using if at any of the maintainers of any of
00:25:59.100
those projects are here I would like to buy them the beverage of their choice because they've saved me so much
00:26:04.679
um so I've only got a few minutes I'm going to run through this really quick and then I want to look at the code
00:26:10.020
um uh that real-time authorizer when the
00:26:15.299
request comes in um it'll kick off a sidekick job and in that sidekick job as I mentioned we're
00:26:21.419
going to run the same exact code that we ran before but this time we're going to instrument it we're going to add some
00:26:26.700
tracing around we're going to record the the outputs we're also going to do some
00:26:32.460
tracing and capture some values within the code and and that's there's kind of a cool tool for using that that we're
00:26:38.820
we're going to talk about and for each of these we're just saving a record to the to database using active record
00:26:44.960
it's a snapshot Shadow snapshot here we're capturing the
00:26:50.340
inputs the outputs uh those variables that we've captured whether it's the control so if we have the feature flag
00:26:56.940
off we're calling it control if the future flag is on we we name that variant you know which feature is it
00:27:03.240
that we're trying to roll out and then for any of those variants we we're capturing a diff of the outputs so that
00:27:08.880
we can compare them later um and here's how we how we built it this
00:27:15.299
is very simplified code it's missing a lot I just want to kind of give you a gist of how we built it if it's something that there's a lot of interest
00:27:20.880
in maybe we'll we'll think of coming up with the way of of open sourcing it
00:27:26.400
um but the first thing is we do is we Define this uh this list of hashes that
00:27:32.279
represent our interests so what this is saying is as I'm stepping through the code when I get to the credit calculator
00:27:37.440
there's a variable called pending amount I really want to know what that was at that point in time that we were that we
00:27:42.960
were stepping through uh that we're running that logic for both the control and the variance see if that changed
00:27:48.799
same thing in some fictional transaction service if there's a transaction count if there's amount due those are
00:27:54.120
variables that we're going to keep an eye out for and we pass those to this class called Shadow Tracer and all this is running
00:28:01.679
inside that that sidekick worker and I'll get into what that shadow Tracer
00:28:07.320
looks like in a minute um and then in that same sidekick worker um we run the same block of code twice
00:28:14.279
once with the feature flag flipped on once with it flipped off um
00:28:19.320
and this code's a little bit uh there's a little bit going on here um that Tracer that we created earlier we
00:28:25.860
told what we're interested in um it has a method called capture and we pass it a block and inside that block is
00:28:31.620
where we actually run the code that we're interested in in this case we're just running this credit authorizer
00:28:36.960
we're telling it whether the feature's enabled or not and we're passing it the uh the request that we got from Visa
00:28:43.440
um and then based on that output we'll uh we have this method that does a little bit of cleanup and then just saves that uh that information to um to
00:28:51.299
active records so we're saving whether whether it's a variant or not what the inputs were what the outputs were and
00:28:57.059
what those uh captures were and it'll it'll take care of calculating the diff and that's something that we're not really going to go over
00:29:03.900
um but that shadow tracer here's a little bit of code that was
00:29:09.720
extracted from uh from that class we're using this tool called tracepoint has anybody here you had to use
00:29:15.360
tracepoint um it's a it's a pretty cool uh bit of
00:29:20.820
functionality that's built into Ruby um and you can subscribe to events that are happening as your Ruby code is being
00:29:26.940
executed so in this in this case this uh line Trace what I'm doing is I'm um I'm
00:29:32.760
saying I want to um in any any code that uh that runs
00:29:38.520
while this line Tracer is enabled if there's an event called line or return which means uh
00:29:46.220
we've started executing code on a new line or the the method has returned
00:29:52.500
um I want to run the code inside the block so every single line that's run we're going to run this uh this code
00:29:58.740
that's inside this block um this is why we're not running this in uh concurrently when the transaction is
00:30:05.460
running and we're responding to Visa because we could do something really easily and just screw up our performance but we're running this only in a
00:30:11.340
sidekick worker where performance doesn't matter um and then uh you don't really have to
00:30:17.220
get too deep into how this is working but essentially what we're doing is we're looking to see uh okay for each
00:30:22.919
line that we're running are we in that class that we cared about if we are let's look at all of the local variables
00:30:29.460
that are available to us right at that point in time if any of them match those things that we're interested in I want
00:30:34.799
to stash it away so I can look at them later and see what they see what they look like um and all the capture method is doing
00:30:41.279
is it's enabling uh that uh at Trace point and then at the end it's
00:30:48.120
disabling it and I realized I miss uh I must have deleted some memoization there because that's going to work a little
00:30:54.000
bit differently than you would expect um and uh that's that's kind of one of the
00:31:00.299
tools that we've used so we can push out uh changes and have 100 certainty that
00:31:05.520
it's working exactly the way we expected um I've got a Shameless plug I'm out of
00:31:11.640
time but um we are hiring uh come see us down in the Expo Center
00:31:17.640
um we're hiring for a lot of different uh roles um if any of you are here at this
00:31:22.679
conference you are our target audience we are we are the folks that we want to hire so come talk to us
00:31:28.740
um and I'm out of time I can't take questions up here but I will be available down here at the side if anybody has any questions and if they're