Background jobs

Summarized using AI

Rails Permanent Job - build a Ruby on Rails server using ServerEngine

Paweł Strzałkowski • August 30, 2023 • Warsaw, Poland

Summary of the Video: Rails Permanent Job - Build a Ruby on Rails Server Using ServerEngine

In this presentation by Paweł Strzałkowski at the Ruby Warsaw Community Conference, the main focus is on developing multi-process servers using the Ruby on Rails framework through the use of relevant gems, particularly the Rails Permanent Job gem, which acts as a wrapper for the ServerEngine gem.

Key Points Discussed:

  • Game Development in Ruby on Rails: The speaker engages the audience by discussing personal experiences in game development across various programming languages, emphasizing that creating games in Rails is feasible.
  • Scheduled Tasks with Cron: The presentation highlights the use of Cron for scheduling tasks in Unix-based systems, illustrating how it can handle jobs executed at defined intervals but noting its limitations in handling high-frequency tasks due to potential overlaps.
  • Custom Rake Tasks: Paweł explains creating Rake tasks for executing jobs at sub-minute intervals and discusses the risks involved, especially when using Sidekiq for task execution, which can lead to inaccuracies and overlaps.
  • Continuous Processing: Moving on to the need for continuous data processing, he underscores the challenge of reliable job execution in frameworks like Sidekiq, which are not ideally suited for this without risking overlaps.
  • Introduction to ServerEngine: The speaker introduces ServerEngine as a framework for building robust multi-process servers. He details its features, such as automatic restarting of failed workers and integration with TCP servers, allowing for resilient server solutions.
  • Rails Permanent Job: The presentation introduces the Rails Permanent Job gem, which simplifies the implementation of ServerEngine. It is described as user-friendly and production-tested, allowing developers to manage multiple jobs easily without complex setups.
  • Game Implementation: Paweł showcases a simple Ruby on Rails game demonstrating how game logic can run on the backend while the front end handles updates, showcasing the interplay of server processes and game states.
  • Takeaway Messages: Emphasizing that not all problems should be approached with Sidekiq and encouraging developers to explore alternative solutions like Rails Permanent Job, he leaves the audience with an invitation to experiment with game development and the discussed tools.

Concluding Thoughts:

The presentation concludes with an interactive multiplayer game demonstration, inviting audience engagement and showcasing the practical application of the discussed concepts. The importance of vibrant interaction in game development through backend processing is reinforced. Overall, the talk provides valuable insights for developers looking to enhance their Ruby on Rails applications through robust server management strategies.

Rails Permanent Job - build a Ruby on Rails server using ServerEngine
Paweł Strzałkowski • August 30, 2023 • Warsaw, Poland

Video from the first edition of Ruby Warsaw Community Conference with a presentation by Paweł Strzałkowski. The speaker introduces the concept of building multiple process servers using Ruby on Rails framework. The speech describes a helpful tool - Rails Permanent Job gem, which is a wrapper for ServerEngine gem.

Additional links:
- Rails Permanent Job gem - https://github.com/pstrzalk/rails_permanent_job
- ServerEngine gem - https://github.com/treasure-data/serverengine
- Sneakers gem (using ServerEngine) - https://github.com/jondot/sneakers

Presentation plan:
[0:23] Beginning of the presentation
[1:25] Games in Ruby on Rails
[5:45] Using Cron to perform scheduled actions
[9:10] Custom rake tasks for handling sub-minute intervals
[12:20] Continuous read for a data feed
[12:48] Do not use Sidekiq for scheduling next jobs
[15:45] Concurrent solutions
[18:10] Introduction to ServerEngine
[21:00] ServerEngine code examples
[23:55] Introduction to Rails Permanent Job
[25:17] Rails Permanent Job code examples
[30:16] Ruby on Rails game implementation presentation
[31:56] Summary
[33:34] Multiplayer Ruby on Rails game showcase
[35:08] Q&A
____________________________________________

► Looking for a dedicated software development team? Contact us at:

https://visuality.page.link/page

► SUBSCRIBE to learn more about software development:
http://bit.ly/SubscribeVisuality
http://bit.ly/SubscribeVisuality
http://bit.ly/SubscribeVisuality

► Read what clients say about us on Clutch.co:

https://clutch.co/profile/visuality

► Find us here:

Instagram: https://www.instagram.com/visuality.pl/
Facebook: https://www.facebook.com/visualitypl
Linkedin: https://www.linkedin.com/company/visualitypl/
X: https://twitter.com/visualitypl
Dribble: https://dribbble.com/VISUALITY
GitHub: https://github.com/visualitypl

Ruby Warsaw Community Conference Summer Edition 2023

00:00:20.580 Let's see if it works. Hi, guys!
00:00:26.640 Have you ever wanted to write a game? It's not a rhetorical question — I genuinely want to know if any of you have attempted game development and perhaps failed.
00:00:39.059 I can certainly raise my hand; I've tried my hand at it in every technology I've worked with.
00:00:44.640 My first game was actually a Blackjack bot on IRC, despite being far too young to play Blackjack. I don’t really know why that was my starting point, but eventually, I wrote games in Turbo Pascal, Java, JavaScript, and even React Native.
00:00:58.079 Eventually, I moved on to C# with Unity, and that was a lot of fun. So, have you ever pictured writing a game beyond Rails?
00:01:14.000 Is it even possible? Let’s take a look at how it might look.
00:01:53.640 Now, let's move to the fun part.
00:02:21.599 Whoa, everything is moving! Just for your information, this isn’t built in JavaScript. Everything visible here is generated on the backend and sent to the front end.
00:02:35.580 So, the world is being processed on the backend, and we can maneuver to dodge dragons and collect rubies. Additionally, we can connect another player — now we have two knights on the board, and the second player can still move.
00:02:48.540 Well, if you select correctly. So, we have a game here, and it's written in Ruby on Rails. Now, let's examine how it works.
00:03:36.360 The game logic runs entirely in the backend and is stored persistently in a database. The browser refreshes to pull the updated state periodically, allowing for ongoing interaction and a constant flow of changes within the game world.
00:04:05.280 Sorry! The game movements are basic Ruby on Rails endpoints — nothing fancy here. However, we need to ensure that the world progresses, which is done via a game loop. The game loop is a process that runs continuously, like a 'while true' loop that gathers inputs, calculates the next state, and renders outputs.
00:04:41.880 By combining the mechanics of controllers for movement endpoints and the game loop, we create a vibrant world, enabling us to build a fully functional multiplayer game, and we can achieve so much more!
00:05:15.479 My name is Paweł. I work as a software consultant, team leader, and developer at Visuality. Today, much to some people's surprise or delight, I won’t be talking about domain-driven design. Instead, I will show you how to build a multi-process server using Ruby on Rails.
00:05:29.220 However, before we dive deeper, we need to address a few basic questions. Firstly, how do you check application state every day? For instance, doing so once a day is straightforward; you can use Cron.
00:05:47.940 Cron is a time-based job scheduler found in every Unix-based operating system and serves as a system-level tool. It's often used with popular Ruby on Rails gems to schedule tasks at precise hours and minutes based on a defined time expression.
00:06:31.979 For example, a job that executes a shell script at 2:05 AM daily. This method works well for simple tasks but what if you want to check something, like a currency rate every hour? That's also simple — just modify your time definition.
00:07:10.280 How about checking a transaction state every minute? While there's a Cron definition for it, running jobs every minute can create potential overlaps, as job execution might exceed the one-minute interval. You might need some form of throttling to ensure that only one job runs at a time.
00:08:15.300 In that context, reliability is paramount—if you state that a job should run at a specific time, it must do so. While system-level Cron ensures that it will execute, many of us tend to use Sidekiq, which can be unreliable when attempting to schedule tasks at exact times due to potential overloads.
00:08:42.420 So, what about checking a system’s state every second? To achieve this, we need to consider creating a loop that executes code at short intervals. Although I initially had beautiful color-coded code in my presentation, I had to remove it just before this talk.
00:09:11.040 For the sake of simplicity, I've put a 'sleep 1' in the example, meaning that, ideally, we'd want to execute this code every second. Yes, we can put it all in a rake task and achieve consistent monitoring; this could be a very simple approach enough to solve the issue.
00:10:15.300 For instance, if you’re using Heroku, the minimal Dyno costs around $7. While this approach might work effectively, it's crucial to maintain a measure of what's running and possibly restart your system if needed. Let's seek to solve another task—how to continuously read from a data feed.
00:11:20.220 We have a Rake task that reads messages, processes them one by one, and sleeps for, say, 10 seconds. However, I strongly suggest not to adopt such a method, as seen in this example where it might schedule itself again after processing.
00:12:00.840 The issue is that Sidekiq jobs rely on Redis for a read-write pattern, making it hard to control the scheduling effectively or predictably, leading to potential overlaps if jobs take longer than expected. Thus, we must bear in mind that Sidekiq is designed for straightforward job execution.
00:13:07.280 When it comes to processing job executions, if failures occur, how can we monitor and resume threads effectively? Furthermore, there are questions regarding deployment; gracefully stopping instances is important when new versions are ready without simply terminating processes.
00:14:14.160 This complexity is where the ServerEngine comes into play. It provides a framework for building robust multi-process servers and was first introduced by Masahiro Nakagawa during a RubyKaigi presentation in 2014.
00:14:51.000 The documentation is clear and accessible since it's open-source. To summarize the structure: we can define workers and run as many as needed, whether as processes or threads. Moreover, ServerEngine allows for TCP server integration for incoming communications.
00:15:47.940 Additionally, there's a supervisor mechanism to ensure failed workers are restarted automatically. This resilient system enables robust handling of memory or network issues that may arise in your server setup.
00:17:20.160 While ServerEngine may sound complex, it only requires a few lines of code to implement a server that works efficiently. Given that there are many widely-used gems like Sneakers—which integrates with RabbitMQ and uses ServerEngine—it's wise to consider implementing this framework in your projects.
00:18:38.640 However, many Rails developers may perceive ServerEngine as complicated, preferring simpler solutions and quick implementations. This is where Rails Permanent Job comes in—a wrapper for ServerEngine that simplifies its usage, making it more approachable for developers.
00:19:46.680 Rails Permanent Job is production-tested, having been deployed without any issues. It's designed to run event consumers seamlessly in your applications without needing complex setups or management.
00:20:38.799 The library provides a simple way to create jobs using a class with a 'call' method. For instance, you can run your tasks using Rails Permanent Job's Rake tasks. You can even define the number of workers easily, allowing for effective resource management during execution.
00:21:40.920 If you add multiple jobs, the library will ensure that they run sequentially, while you can define after-jobs to manage delays between tasks or define complex logic in your operations.
00:22:51.000 The library's flexibility allows developers to support various scenarios effectively, enabling practical implementations for services like AWS SQS consumers. This way, starting a job is as simple as calling the process without in-depth configuration.
00:24:00.360 Let's summarize key takeaways: not every issue needs to be solved using Sidekiq. When considering alternatives, remember to explore solutions beyond the conventional Ruby on Rails paradigm. Make sure to watch the video on ServerEngine, though it's lesser-known, it holds considerable potential for Ruby developers.
00:25:32.640 Lastly, I encourage everyone to try developing a game! It's a rewarding experience that can teach you a lot, whether for personal enjoyment or as a side project. Please explore the Rails Permanent Job gem and share feedback on any features you might find beneficial.
00:27:15.540 I invite you to join the multiplayer game demo! You'll get to see how your characters interact on the screen, and as you navigate the game, try to avoid catching a dragon, as that will cost you points.
00:29:45.840 I appreciate your engagement throughout this presentation, and I'm open to any questions you may have.
00:30:32.640 If you have queries regarding the Rails Permanent Job or need assistance related to multithreading or job scheduling, please feel free to ask.
00:31:39.600 Thank you for your participation, and if you wish to explore more, join me at the after party for further discussions. Would there be any inquiries or feedback you'd like to share?
00:32:47.460 If interested in playing further, feel free to join us — we’re excited to keep the fun and engagement going!
Explore all talks recorded at Ruby Warsaw Community Conference Summer Edition 2023