Summarized using AI

Showing Progress of Background Jobs with Hotwire Turbo

Michał Łęcicki • June 09, 2024 • Hamburg, Germany • Talk

In this presentation titled "Showing Progress of Background Jobs with Hotwire Turbo," Michał Łęcicki discusses how to effectively display the progress of backend jobs in Ruby applications using Hotwire, a modern tool that integrates Turbo and Stimulus. The talk is aimed at developers who want to enhance user experience by notifying users of the status of long-running background jobs. The key points of the presentation include:

  • Introduction to Hotwire: Michał briefly explains what Hotwire is and its components including Turbo and Stimulus, setting the stage for the practical usage examples that follow.
  • User Interaction and Feedback: He emphasizes the importance of updating users about the status of background jobs, especially when these jobs take a significant amount of time to complete.
  • Historical Context: The presentation offers a historical perspective on handling background job progress, referencing a 1985 technique that required additional gems (like Sidekiq) to manage job statuses, highlighting issues with complexity and confusion for developers.
  • Using Hotwire for Progress Representation: Michał describes various solutions to implement progress displays in Hotwire. He discusses a specific application model where users request jokes from an API, and the process of how to handle and visualize the progress of fetching these jokes.
  • Handling Multiple Background Jobs: The talk also covers the scenario of multiple concurrent jobs, illustrating how to manage updates to ensure smooth user experience without overwhelming the frontend. He suggests using a Stimulus controller as a mediator to handle updates efficiently.
  • Different Implementation Approaches: He suggests multiple approaches for broadcasting updates from service objects rather than directly from ActiveRecord, achieving a more decoupled architecture. He also presents alternative solutions to tasks not requiring database interactions.
  • Conclusion and Takeaways: Michał concludes by emphasizing the significant role of Hotwire in enhancing communication between backend and frontend, encouraging attendees to explore these techniques in their applications. He provides additional resources for further learning.

Overall, the presentation efficiently combines theoretical knowledge with practical application, demonstrating the capabilities of Hotwire in modern Rails development.

Showing Progress of Background Jobs with Hotwire Turbo
Michał Łęcicki • June 09, 2024 • Hamburg, Germany • Talk

Ruby Unconf 2024

00:00:00.199 please welcome m w shitski he's a ruby
00:00:04.560 developer and he will talk about showing
00:00:07.520 progress of background jobs with TBO
00:00:10.120 please give him a round of
00:00:16.439 applause
00:00:19.600 hello before I tell you about the topic
00:00:22.119 of my presentation I will quickly
00:00:24.160 introduce myself uh my name is m
00:00:26.840 winitzki if you like challenges you can
00:00:29.000 try to pronounce my last name but you
00:00:31.119 don't have to uh I'm a ruby developer at
00:00:35.040 visuality
00:00:36.960 uh before we start uh I like to ask you
00:00:41.000 a few questions I know that yesterday I
00:00:43.760 already made a little research what is
00:00:47.120 the state of you and what you think
00:00:50.000 about turbo hot wire uh but I have
00:00:53.680 another question how many people
00:00:56.879 completed the tutorial uh this hot r
00:01:00.320 turbo hot wire tutorial please raise
00:01:02.719 your
00:01:04.479 hand okay few of you and how many of you
00:01:08.880 want to use hot wire on
00:01:12.119 production
00:01:15.799 want lots of you want to do that that's
00:01:18.960 great uh maybe after my talk I will
00:01:21.960 motivate you to try either the Hotwire
00:01:25.840 tutorial or to start using Hotwire on
00:01:29.720 produ production we will
00:01:33.560 see and the topic of my presentation is
00:01:36.799 how to display progress of something
00:01:39.759 that is happening on the back end with
00:01:41.720 hot wire I thought that this topic is
00:01:45.119 nice because it shows a use case that
00:01:48.920 might
00:01:50.079 be that you may need at some point in
00:01:53.119 the development because sooner or later
00:01:55.880 you will have some background job and
00:01:58.640 sooner or later you will have a
00:02:00.200 background job that takes more than few
00:02:02.680 seconds and it's nice to inform users
00:02:05.600 what is the state of this job and when
00:02:08.119 it will be
00:02:09.119 finished in such case we can easily use
00:02:12.760 hot
00:02:14.720 wire but let's back let's get back to
00:02:18.560 history to 1985 and how it was done at
00:02:23.120 that
00:02:25.959 year so before if you want to achieve
00:02:29.400 that you have background job and you
00:02:31.920 want to display a progress bar for
00:02:34.239 example you need to add one more Ruby
00:02:36.959 Jem and there is a great Ruby Jam
00:02:39.239 sidekick status probably there are more
00:02:41.519 that can achieve that but this is uh to
00:02:44.879 me it's more popular and it provides a
00:02:48.239 very convenient API that you could use
00:02:51.239 but as we know from yesterday it's
00:02:53.480 against the laziness we are we are
00:02:56.480 adding one more ruby gem to our game
00:02:58.959 file and also it provides the API that
00:03:02.080 might be confusing a little if you are a
00:03:04.680 new developer or if you are new in the
00:03:06.920 project you need to go through this
00:03:09.200 documentation of this gem to read what
00:03:12.959 those methods are doing total at it
00:03:16.640 sounds like a
00:03:17.879 magic also it's getting more complex if
00:03:21.599 the job becomes more complex and if it
00:03:24.080 invokes more than one service few
00:03:27.239 Services then you need to pass this
00:03:30.560 magic methods inside those Services etc
00:03:34.200 etc so there are many challenges with
00:03:36.680 this approach it's easy at the first
00:03:39.040 glance but it's it has many
00:03:45.120 drawbacks now it works okay but we live
00:03:48.879 in a modern era of hot wire at least
00:03:51.360 those of us who are using it on
00:03:52.920 production so we can achieve the same
00:03:56.799 with hot wire and in my presentation I
00:04:00.239 won't uh tell you about the theory about
00:04:04.760 how hot wire exactly works because you
00:04:07.879 all heard about it and you know that hot
00:04:11.120 wire consists of Turbo stimulus and
00:04:14.439 Strada in my presentation you will see a
00:04:18.040 little of Turbo and a little of
00:04:21.759 stimulus but the piece of theory that we
00:04:24.600 need for the purpose of this
00:04:26.759 presentation is the fact of Broadcasting
00:04:30.080 updates happening on the back end to the
00:04:32.360 front end we could do that before
00:04:35.000 Hotwire with action cable but now we
00:04:38.639 have a really convenient API that is
00:04:41.240 doing that
00:04:43.440 easily and what rails provides to us is
00:04:47.000 a really convenient API that we can hook
00:04:51.759 to changes happening on the database
00:04:53.919 level so when something changes in the
00:04:57.400 database we can easily broadcast a
00:05:00.479 message to the front end that hey
00:05:02.560 something was
00:05:06.680 updated example
00:05:09.440 time for the purpose of this
00:05:11.479 presentation we will use a very simple
00:05:13.639 application that I created and it
00:05:15.840 consists of two models uh one of them is
00:05:19.120 a joke and another one is a joke request
00:05:21.919 so imagine you are a user that wants to
00:05:24.240 have some fun and you create a joke
00:05:27.319 request uh in there you uh you specify
00:05:31.000 how many jokes you want to fetch from
00:05:33.440 the API that provides jokes we will use
00:05:36.600 Chu noris
00:05:37.880 API because it's free and it's it has
00:05:40.720 really nice end point to provide a
00:05:42.360 random joke about Chu
00:05:44.520 Norris so the idea is that we create a
00:05:48.039 joke request and when you hit the button
00:05:51.840 submit and you specify how many jokes
00:05:54.639 you want to get there is you hit the end
00:05:57.720 point and in the end point we we spawn a
00:06:00.479 background job and the background job
00:06:03.000 downloads the jokes and we want to
00:06:05.960 display the progress of this
00:06:09.199 process also I need to mention that uh I
00:06:12.199 put some jokes that I get from this API
00:06:15.800 into my slides some of them might
00:06:19.240 be inappropriate maybe but that's not my
00:06:23.759 fault that's this
00:06:27.560 API uh so this is the plan for my
00:06:30.639 presentation actually I will show you
00:06:32.720 different ways how we can achieve that
00:06:35.599 uh maybe some Alternatives uh a little
00:06:39.280 bit of JavaScript and also I have a
00:06:42.160 bonus at the very end so let's go with
00:06:45.479 with the first
00:06:46.840 solution it's that
00:06:49.639 simple with hot
00:06:51.960 wire we can use callbacks that can
00:06:56.520 invoke methods which will broadcast the
00:06:59.199 update to the front end and for example
00:07:02.800 we have a method broadcast replace to
00:07:05.720 that says hey write to open that to the
00:07:09.479 stream that is open with the specific ID
00:07:13.160 and Target a specified HTML element on
00:07:16.840 the page and replace it with a new
00:07:19.520 element with updated
00:07:23.360 state in the invocation of this method
00:07:26.160 probably the F first argument is the
00:07:28.360 most important because it specifies the
00:07:31.440 exact stream we are writing to and
00:07:34.759 usually the biggest challenge is how to
00:07:37.120 identify those streams because when I
00:07:40.520 create a joke request and you create one
00:07:43.680 we want to broadcast the updates to the
00:07:46.720 correct
00:07:48.120 stream but in our application we are in
00:07:52.120 a good position because when you want to
00:07:54.800 fetch jokes you create a record in the
00:07:56.960 database so this jokes request record in
00:08:00.199 the database can serve the purpose of
00:08:03.120 identification of the
00:08:06.280 stream the target is the ID HTML element
00:08:10.879 on the page and then we can provide a
00:08:13.000 partial that will replace that element
00:08:15.800 and as you can guess for the progress
00:08:17.680 bar it means just render the updated
00:08:20.840 progress bar before we fetched for
00:08:23.240 example just one joke after we fetch the
00:08:26.840 the other one we just have a progress
00:08:28.960 bar that is one step
00:08:33.120 later so we did it on the active record
00:08:37.240 uh uh level let's say to make it fully
00:08:40.640 work we also need to update the front
00:08:42.680 end we modify the HTML file we need to
00:08:46.320 open the stream and you can see that the
00:08:48.720 name of the stream is exactly the same
00:08:51.279 as it was in the previous slide so we
00:08:54.519 identify that by joke request and the
00:08:57.760 name jokes progress bar and and that's
00:08:59.839 exactly the same here and we render of
00:09:02.640 course the partial with Do's progress
00:09:04.720 bar and it works it's really that simple
00:09:08.200 and it works the solution is easy it's
00:09:11.279 fast but it also has some drawbacks and
00:09:15.600 maybe one of them is that we were using
00:09:18.200 a
00:09:19.440 callback we don't like callbacks
00:09:23.200 much and it it can easily escalate into
00:09:27.920 so-called C's h especially in Legacy
00:09:30.839 applications so maybe there is
00:09:33.120 alternative approach that could be
00:09:35.760 better also what we did we bind the
00:09:40.680 logic responsible for updating something
00:09:43.440 on the front end with the active record
00:09:47.079 uh level so even if you create a record
00:09:51.160 in the database or somewhere somewhere
00:09:53.480 else this broadcast is being
00:09:56.920 invoked that's not so nice
00:10:00.600 and the most important you cannot always
00:10:03.279 use this solution if imagine that we we
00:10:07.040 are supposed to fetch jokes but we don't
00:10:09.640 save them into database in such case we
00:10:13.040 we don't have a callback that we can
00:10:14.839 hook on the active record model so let's
00:10:18.519 search for the Alternatives the second
00:10:20.880 solution which is a little
00:10:25.000 better does almost exactly the
00:10:28.320 same but we invoke the broadcast from
00:10:32.079 the level of the service or the job that
00:10:34.959 is doing those actions so we are not
00:10:39.000 doing that in the active record model we
00:10:41.720 are doing that in the service for
00:10:43.480 example and we can use exactly the same
00:10:45.920 method we need to just specify on what
00:10:48.279 class it's invoked it's turbo streams
00:10:51.000 Channel but all other arguments are
00:10:53.720 almost the same since we are independent
00:10:56.639 from the active record we also need to
00:10:59.320 update the partial a little to provide
00:11:02.120 the input it needs now it receives not
00:11:04.800 the active record um model now it
00:11:08.880 receives the number of the pro actual
00:11:11.920 progress and the end stage of the
00:11:16.279 progress and modification for the front
00:11:19.000 end we actually just modify the state of
00:11:22.800 the
00:11:23.880 partial uh the the input of the partial
00:11:26.959 the stream stays the same
00:11:30.920 and it works so we have no call backs
00:11:34.519 this solution is easy to test because we
00:11:36.800 just have one specific uh place that
00:11:40.079 makes a broadcast it's easy to
00:11:43.600 identify
00:11:45.360 and this is a huge Advantage especially
00:11:48.600 on the production when when you not when
00:11:52.079 you just not have one broadcast but 10
00:11:54.959 places or 100 places it's really
00:11:58.240 convenient to easily find the places
00:12:01.000 that that are actually doing those
00:12:03.320 broadcasts it's it will be more
00:12:06.160 convenient to debug any problems if you
00:12:08.279 have
00:12:09.399 them but it looks to be not the default
00:12:12.399 way uh I asked you about this uh hot
00:12:15.560 rails tutorial and as I remember there
00:12:18.639 is the default way of adding broadcasts
00:12:21.959 in this tutorial is uh using the active
00:12:24.600 record approach so maybe it's not the
00:12:27.440 most typical approach there and also we
00:12:31.800 still need to identify those
00:12:35.040 streams we are in a really good position
00:12:37.800 that we are still we still have this
00:12:40.399 jokes request object in the database and
00:12:43.880 we are using it for identification but
00:12:47.760 if the requirements are changed a little
00:12:50.600 and we say that hey don't touch anything
00:12:53.320 in the database please provide me an
00:12:55.480 endpoint that fetches the specific
00:12:58.560 number of jobs
00:13:00.800 then identifying uh this stream
00:13:03.920 correctly will be much more challenging
00:13:06.760 most probably we will need to create
00:13:08.720 some ID or U ID on the Fly and pass it
00:13:12.639 through all the methods to have it in
00:13:15.199 the background worker and also have it
00:13:17.680 on the front end maybe store it in the
00:13:23.800 URL so that was the second
00:13:26.720 solution we have also the third solution
00:13:29.360 solution which is actually not a
00:13:31.320 background
00:13:34.360 worker you can have a different approach
00:13:37.120 that will work for the requirement that
00:13:39.600 I just U mentioned if you don't want to
00:13:42.519 touch database at all you could use and
00:13:46.600 if you are able to get rid of the
00:13:49.360 background jobs you can use a turbo
00:13:52.240 frame tag and you can say that hey when
00:13:55.959 you are on this
00:13:57.320 page go uh add a source attribute to the
00:14:01.920 turbo frame tag and when the user enters
00:14:05.120 this page the turbo automatically hits
00:14:07.800 the end point behind this
00:14:11.160 source and the endpoint can just fetch
00:14:14.880 jokes that would also work it would be
00:14:19.040 even um if you want you can do it even
00:14:24.079 asynchronously and you can you can add
00:14:26.959 additional attribute lazy loading
00:14:30.040 which will make the turbo make this uh
00:14:34.279 heat to the end point only at the moment
00:14:36.720 when user enters this specific place at
00:14:39.759 the page so if it's on the bottom of the
00:14:42.560 P at the page user will need to scroll
00:14:45.839 down to actually see those
00:14:51.519 results in this uh last scenario there
00:14:55.519 is no background processing there is no
00:14:58.560 database
00:15:00.399 and well it's to be honest it's
00:15:03.079 completely different solution so it
00:15:05.279 depends on your requirements if you need
00:15:07.440 this solution with background jobs or
00:15:09.519 this alternative
00:15:16.120 approach okay now probably you are
00:15:21.839 wondering what if we have more than just
00:15:25.800 one job so we were talking about
00:15:29.199 background jobs and it's easy to imagine
00:15:32.959 that they escalate quickly if you want
00:15:36.079 to fetch 100 jokes maybe it will be a
00:15:39.079 nice feature to make it parallel maybe
00:15:42.040 let's say we will download one job will
00:15:45.279 download 25 jokes so for the bigger
00:15:48.279 number we will just spawn more
00:15:50.920 jobs but the problem is that those jobs
00:15:54.480 are running at the same time and they
00:15:56.839 will try to update the progress bar at
00:15:59.480 the same time so there will be
00:16:02.519 stuttering and most probably progress
00:16:05.000 bar won't finish at
00:16:09.040 100% how to solve this issue if we get
00:16:13.120 one step back and we look at the
00:16:16.319 previous
00:16:17.959 Solutions we had a very simple
00:16:21.120 design we had the state of the front end
00:16:24.199 and when something happened on the back
00:16:26.160 end the back end decided that well let's
00:16:29.959 update the front end and the method
00:16:32.639 broadcast replay to was doing this
00:16:35.480 update the back end was responsible for
00:16:38.040 that to make it work with more
00:16:40.880 background jobs we need to shift our
00:16:44.399 thinking and we need to give more power
00:16:47.240 to front end to
00:16:49.560 JavaScript so the solution in this case
00:16:53.360 is to use a mediator stimulus controller
00:16:57.040 that will take care of those updates
00:16:59.519 and the stimulus controller will decide
00:17:02.199 if you need to update the progress bar
00:17:04.079 or not and how to update it so basically
00:17:09.079 background background jobs are telling
00:17:11.760 that hey there is an update update
00:17:13.959 update but the stimulus controller will
00:17:16.520 make sure that this update of progress
00:17:18.520 bar is
00:17:24.559 smooth lots of code
00:17:30.120 I'm not sure if you want me to explain
00:17:32.880 everything what is happening if you want
00:17:34.960 to use stimulus maybe now I will ask you
00:17:37.120 a question uh who has ever used
00:17:41.720 stimulus okay so like more than half of
00:17:45.280 the room is already familiar with that
00:17:47.440 so you know what this magic does you
00:17:49.799 need to provide a data controller you
00:17:52.760 can provide some values as the input for
00:17:55.159 the controller you can have a Target
00:17:58.640 that that uh will be used for targeting
00:18:01.440 HTML
00:18:03.520 elements so the key point is that you
00:18:06.120 need to modify the current HTML
00:18:09.280 structure and use the convention
00:18:11.520 provided by stimulus to provide the
00:18:13.679 needed data by
00:18:15.880 stimulus and now I will show you some
00:18:19.679 JavaScript not sure if you saw this
00:18:22.039 sticker it's it's from the men's
00:18:25.720 bathroom but but that's how I feel when
00:18:28.840 I need to write some
00:18:32.720 JavaScript so adding stimulus means we
00:18:36.440 create a new stimulus controller and
00:18:39.240 stimulus controller is a JavaScript code
00:18:42.360 that will be invoked on a specific Pages
00:18:45.320 we know on which Pages because we
00:18:47.200 provided the data controller
00:18:50.600 attribute and we are using those
00:18:52.919 attributes that we specified in HTML
00:18:56.039 values and targets to for for a
00:18:59.360 definition for the controller but what
00:19:02.000 is the most important is the connect
00:19:05.159 function this is the function that is
00:19:07.200 invoked when the page is loaded and the
00:19:10.360 JavaScript from the controller that is
00:19:12.520 used on the contol on the on this page
00:19:15.280 is
00:19:16.440 invoked so what our connect uh function
00:19:20.760 needs to do we will use quite a smart
00:19:25.120 trick here and we will overwrite the
00:19:28.400 exist existing turbo action there is
00:19:31.760 before stream render
00:19:34.120 event which we can
00:19:36.799 overwrite we are basically telling hey
00:19:39.679 when you want to add a new joke don't do
00:19:42.320 it and there is a if condition that if
00:19:47.320 you want to add this exactly joke
00:19:50.200 element to the specific Target then
00:19:54.080 let's run some custom code and the
00:19:57.320 custom code you will see in moment is
00:19:59.679 responsible for updating the progress
00:20:01.640 bar but then we can fall back to the
00:20:04.120 default which is just rendering a new
00:20:06.400 element on the page the
00:20:09.919 joke and what happens in our custom
00:20:13.679 function we increment the state of the
00:20:17.200 progress bar we have the internal value
00:20:20.039 that represents the actual state of a
00:20:22.200 progress bar we update the HTML element
00:20:25.880 and we can update also other elements we
00:20:28.080 can have also the count of the
00:20:32.159 jokes and it should
00:20:36.440 work here I don't have pros and cons I
00:20:40.039 just have a feeling and this fault that
00:20:43.520 when you need to do something more
00:20:45.120 complex on the front end you usually
00:20:47.840 don't have a choice you need to give
00:20:50.799 some give more power to JavaScript in
00:20:53.840 our case to
00:20:55.919 stimulus but maybe
00:20:58.799 something uh that that is worth to
00:21:01.159 mention here when you see that this
00:21:04.679 moment comes it's better to do it
00:21:07.120 quickly like when you see that back end
00:21:10.279 is not enough to handle this complicated
00:21:13.559 logic it's better to well say the truth
00:21:17.080 and use this JavaScript and
00:21:19.840 stimulus I have also a bonus because I
00:21:22.640 have few more
00:21:24.720 minutes um when I Was preparing the the
00:21:28.840 this test application I thought well is
00:21:32.320 it possible also to make it work with
00:21:36.080 pagination is it possible that we have
00:21:39.279 this updating from the background but we
00:21:41.480 also render the pagination and we switch
00:21:44.039 to new pages when we need to right
00:21:46.960 because we don't want to uh render 100
00:21:50.159 jokes on one page and of course it is
00:21:53.360 possible uh the most challenging part is
00:21:56.240 to figure out where when we need to
00:21:59.600 switch to the new page and it's easy if
00:22:02.720 we have 10 jokes on one page on the 11
00:22:05.480 11th joke we need to clear the existing
00:22:09.120 page we need to add new joke and we need
00:22:12.240 to reender the pagination element if not
00:22:15.840 we just add a new
00:22:18.840 joke and in Turbo or in my solution
00:22:23.679 clearing the page means let's render the
00:22:27.039 GRE that consists of the job but it's
00:22:29.799 empty
00:22:32.080 and the most complex things thing here
00:22:35.320 most probably is R rendering the
00:22:37.679 pagination but it's also not so not so
00:22:42.039 complex because this is what we see
00:22:44.200 usually in the controller in
00:22:47.640 our daily uh end points that display
00:22:50.720 many records right so that's the same
00:22:53.279 pagination it's just updated and we can
00:22:56.279 render that if we have a partial for
00:22:58.240 that
00:23:03.039 oh let's sum up so
00:23:06.520 yesterday yesterday we heard that rails
00:23:09.159 is Magic and to me the hot wire is
00:23:14.080 really significant part of this
00:23:16.679 magic and in my presentation um to be
00:23:21.520 honest I didn't want you to just give
00:23:23.600 you a solution how to solve exactly this
00:23:26.279 problem of the progress bar with wire I
00:23:29.760 wanted to give you a taste of using a
00:23:32.279 hot wire and you can see that if you
00:23:35.360 want to use hot wire you need to have
00:23:38.279 slightly different approach how the back
00:23:41.720 back end needs to communicate to front
00:23:43.720 end or how to divide responsibilities
00:23:46.400 between back end and the front end but
00:23:49.200 it's nice so give it a
00:23:54.080 try if you want there are some links
00:23:57.400 that describe this solution and the uh
00:24:00.960 the solutions that are similar to that
00:24:04.279 also if you want to look at the
00:24:06.480 repository there is a
00:24:11.640 link
00:24:13.919 oh
00:24:17.760 okay
00:24:22.240 okay so now is thank you thank you
00:24:34.600 thank um there are any questions I
00:24:37.480 assume you're available now to have a
00:24:39.679 chat maybe you can uh come here and then
00:24:43.360 talk to M if there any questions and we
00:24:46.440 continue with the next talk at 2:30 and
00:24:50.320 I think we're going to learn about
00:24:51.919 Anonymous
00:24:54.360 databases okay thank you
00:24:59.520 we we do
00:25:00.600 the we do the Q&A just um over here so
00:25:04.880 mik will will stay here and then
00:25:10.720 um and there's a speaker gift of course
Explore all talks recorded at Ruby Unconf 2024
+5