Functional Programming

Lighting Talks Day 2

Lighting Talks Day 2

by Alina Leskova, Nobert Wójtowicz, Andrew, Andrzej Krzywda, Jan Dudulski, and Yaroslav Shmarov

The video "Lighting Talks Day 2" from the wroc_love.rb 2022 event features a presentation by Andrew, who shares insights on utilizing Sidekiq for batch processing and discusses the evolution of mathematical thought alongside the relevance of mentoring in the Ruby programming community.

Key Points Discussed:
- Introduction to Sidekiq Batches:

Andrew begins with a personal experience related to Sidekiq and how it can be used to process large datasets efficiently. He emphasizes the importance of a synchronous approach to avoid business issues when handling large tasks.
- Batch Processing Mechanism:

He explains that he parsed a file with around a thousand rows, adding a worker for each row in a batch to ensure efficient processing. He designed a unique identifier for each row to manage the results.
- Redis Storage:

After executing all jobs, results were stored in Redis under a sequence number key, enabling easy retrieval.
- Result Compilation:

A callback worker was used to gather results from Redis, creating a new results file that corresponds with the original data.

  • Mathematical Evolution:

    Transitioning from technology, Andrew reflects on the historical evolution of mathematics from Aristotle and Socrates through Galileo to Newton and Leibniz. He highlights a significant shift in understanding time and speed, applying this historical context to inspire a modern understanding of technology and innovation.

    • Encouragement to Think Critically:
      He urges the audience to consider the essence behind modern technology and how it can transform future innovations.
  • Mentoring in Software Development:

    Andrew transitions to discussing his role as a mentor for junior developers. He underscores the benefits of mentoring within the Ruby community, particularly during discussions about Ruby's future.

    • Ruby's Community Dynamics:
      He touches on the debate regarding Ruby's declining popularity but reassures that the community remains vibrant, with a steady increase in Ruby developers despite its slower growth compared to languages like Python. He believes in nurturing community strength rather than competing with other programming languages.
  • Configuration Management in Ruby:

    Andrew concludes with insights on configuration states in Ruby, emphasizing the need for approaches that allow developers to manage configurations without cluttering their codebases. He introduces the noa config library and his own configuration method, which aims for a more intuitive syntax while adhering to clean code principles.

Takeaways:

- Understanding and efficiently implementing Sidekiq can lead to significant improvements in data processing.
- Historical perspectives in mathematics can provide valuable insights for modern technological advancements.
- Mentorship plays a crucial role in community growth, especially in the context of programming languages like Ruby.
- Emerging libraries and frameworks can help simplify configuration management in programming, improving code clarity.

00:00:16.800 Thank you, everyone. I think it's the worst decision I've made in my life, but I'm here, so we can start. There was a presentation about Sidekiq, and a question came up about Sidekiq batches. I remembered that I used them once in all my four years of experience, and I can share with you how you can do it.
00:00:27.760 First, we need the source for our work. In my case, it was a file containing around a thousand rows that needed to be executed with some comments, and we needed results. If we don't approach this in a synchronous manner, we'll face issues; our business will suffer. Therefore, we tried to optimize it by using Sidekiq and batches. So, we parsed our file, and for each row, we put a worker inside the batch. For example, if we have 100 rows, we end up with 100 workers. I designed a unique identifier for each row which helped identify the sequence.
00:01:02.480 For this example, the unique ID consisted of the time, name, and a sequence number, which I'll start with one. When I executed the command, the results were stored in Redis under a key representing the sequence number. After finishing all my 100 jobs, I would have 100 results in Redis. Moreover, I set a callback for the batch, which recognized the unique source name. In the callback, I created another worker to gather results knowing we had 100 rows in our file. When all jobs were finished, this last worker would read from Redis the source name and the sequence numbers.
00:03:08.400 Knowing the sequence, I gathered the results from all the jobs and created a new result file. I included the source parameters, my comments on them, and the results. The sequences in this result file matched those in the source file, and the business was happy. That's all for now. I hope nobody has questions. However, one question that has come up is how fast an RTMS rocket needs to move to escape Earth's atmosphere and reach a stable orbit.
00:04:42.160 Honestly, I don't care how to calculate this because I'm assuming that everyone here is a fan of Kerbal Space Program. But it's fascinating to think about how when asked how fast you were going, nobody flinches, even though that perspective is quite modern. Looking back at the time of Aristotle and Socrates, they worked with mathematics that was primarily about geometry and trigonometry, believing the world was perfect. They didn’t concern themselves with the real world and how math applied to it.
00:05:02.560 They couldn't explain momentum or how things move, perceiving it as merely a side effect. However, things changed thanks to thinkers like Galileo and Descartes. Galileo realized that time is a unit of measurement, which was an explicit understanding that had not been established before him. Descartes developed the Cartesian coordinate system, merging geometry and algebra into one field. This laid the foundation for modern mathematics, leading to the invention of calculus by Newton and Leibniz. The crucial aspect here is time; it became understood as a measurable unit. This is essential because until that point, the concept of speed was almost science fiction, while now it’s commonplace, as we often think of it in the context of speeding tickets rather than reinventing mathematics.
00:06:54.960 In considering this, I believe we need to evolve our understanding, much like how mathematical advances spurred future generations. When listening to talks by experts, I urge everyone to think not just about the current technology or trends but about the essence behind them and how these concepts can be applied in new ways. My goal is to inspire people to think critically about their work and foster new ideas that will lead to future breakthroughs in technology, similar to those seen in the past.
00:09:16.480 My name is Andrew, and I work for an outsourced company where I serve as a main developer or sometimes a tech lead, depending on the project. In outsourced companies, projects and teams often vary. Thus, mentoring junior developers can become part of the job, and I enjoy it greatly. I believe mentoring offers several benefits, including keeping my skills sharp and helping develop engineers within our company, ultimately making our company better.
00:10:02.400 Additionally, mentoring aids in community growth. With discussions around whether Ruby is dying, we must focus on the positive: we are creating more engineers. I also find fulfillment in mentoring refugees and immigrants in my free time, hoping to inspire others to do the same. The key for me in all volunteering efforts is identifying what motivates you personally.
00:12:03.840 Now, regarding the ongoing debate about Ruby's future, I wonder why this question of Ruby dying persists. It seems many companies are transitioning away from Ruby due to performance issues, turning to alternatives like JRuby or newer languages such as Rust. While I understand why some companies make this move, I believe we should support Ruby and not abandon it. Many people I know have switched to Elixir or other languages, but I’ve seen no significant outflow of Ruby developers recently, reinforcing the notion that Ruby's community is still vibrant.
00:14:06.080 When you analyze the data published each year about programming languages, you can observe that while Ruby's popularity might be slower compared to languages like Python or JavaScript, the number of Ruby developers has been increasing, not declining. Although the growth rate is slower, it’s essential to recognize that we shouldn’t compete with Python, given its diverse applications beyond web development. We have a strong community and should focus on nurturing that strength rather than lamenting our standing compared to other languages.
00:20:15.840 Let’s move on to talking about configuration states in Ruby. Nowadays, we have various options to manage configurations, yet many of those require your code to be aware of the infrastructure. There's a lot of room for improvement. The noa config library takes a different approach, allowing you to define configuration through classes without the consistency issues related to where your configuration is stored.
00:20:34.680 In this way, the infrastructure doesn't bleed into your codebase. However, while I appreciate the concept, I find the syntax not very user-friendly. Thus, I thought about applying similar ideas while adopting a more intuitive syntax, which led to the development of a new configuration method that aligns with the principles of dry struct, allowing easy definitions and modular loaders that can fetch values from various sources. This library is still in development but shows promise. Thank you for your attention.
00:22:00.000 Thank you! That's all from me!