Dataflow: Declarative Concurrency in Rails

Talks

Larry Diehl

1 talk

#declarative-programming

#concurrent-ruby

#functional-programming

#ruby

#event-driven-programming

#actor-model

Dataflow: Declarative Concurrency in Rails

by Larry Diehl

The video presentation by Larry Diehl at LoneStarRuby Conf 2009 focuses on 'Dataflow: Declarative Concurrency in Rails,' exploring programming within a declarative concurrent model. Diehl discusses the evolution of Ruby and its communities, noting a shift towards other innovative languages like Clojure and Erlang. He emphasizes the importance of motivating Ruby developers to explore alternative programming paradigms to remain innovative.

The presentation covers:
- Purpose: Diehl aims to revitalize interest in Ruby and introduce concepts of data flow to enhance developer engagement with new paradigms.

- Lexical vs. Dynamic Scope: He contrasts lexical scope, which provides predictable returns, with dynamic scope that complicates reasoning about code. The complexities of mutability are highlighted as well.

- Declarative Model: He explains the declarative model as focusing on specifying what the algorithm should do, leading to referential transparency, and emphasizes how immutability is crucial in this context.

- Concurrency: Diehl describes how declarative concurrency allows for clear dependencies in multithreaded executions, minimizing race conditions and streamlining testing protocols for concurrent code.

- Practical Examples: He shares advanced scenarios involving threading and future queueing methods that illustrate how Ruby can adopt successful ideas from other languages.
- Call to Action: The presentation concludes by encouraging Ruby developers to experiment with these concepts, recognizing the benefits of diverging from traditional practices to foster innovation.

Overall, Diehl advocates for the data flow model as a refreshing approach that can lead Ruby into a new era of programming creativity, urging the community to embrace different methods while respecting Ruby's philosophies.

00:00:21.080 I guess I’ll get started now. It’s about three.

00:00:27.119 This presentation is on data flow. It’s basically a way to program in the declarative concurrent model. I don’t mean eventually as in the library can’t do it; I mean I’ll eventually get there while explaining it. It’s not very well known, but I’ll show you some references if you’d like to read up further about it at the end of the presentation.

00:00:34.380 So this is me. I work on Engineered Cloud. I use layers of Liquid for pretty much everything. All right, this is sort of the outline: first, I’m going to talk about the purpose of why I’m giving this presentation and sort of how this library was created. Then I’m going to gradually explain the concepts, starting with basic programming theory and expanding from there. But don’t worry, it gets into really cool stuff, and at the end, I’ll share some tips.

00:00:53.160 The purpose—sometimes there are pictures that don’t really mean anything—but the purpose is that Ruby had its success early on, both pre-Rails, having a pretty good hacker community and people doing innovative stuff, as well as post-Rails when there was a big influx of people. People were pretty happy, saying things like, 'Oh yeah, Ruby is awesome! Let’s give talks about Ruby because it’s so cool!' However, from my perspective, that enthusiasm has seemed to die down a bit recently, especially after Rails became the norm.

00:01:36.420 At this point, we see other languages that have copied Rails or Ruby’s useful libraries, including testing libraries and web frameworks. We also have languages such as Clojure, Haskell, and Erlang that are innovating in pretty cool ways. A part of that is because they still have relatively small user bases, so they don’t encounter the backwards compatibility problems that larger communities, like Ruby’s, face.

00:02:01.320 In Ruby, we have many people depending on old versions of libraries, and I want to encourage more people to try out other programming paradigms that are somewhat similar, like data flow. This allows us to keep hackers in the Ruby community instead of letting them leak out into other communities where cool innovations are happening. Not that innovation isn’t happening in Ruby, but we don’t want to rest on our past successes.

00:02:26.640 Now we’re going to start out talking about lexical scope. In lexical scope, you can see here that we have a variable, and then we define a method. The method closes over the variable. The really useful thing to note here is that looking at this method definition, it doesn’t matter if it’s defined inside of a class or if it’s in a module mixed into any other class; you can always know what it returns.

00:02:41.700 This property is really powerful because you can look at those lines of source code in one place and not really have to worry about them changing from under you. However, in traditional dynamic scope, if you’re familiar with certain communities, you see a block of scope that’s reassigning a variable used inside of it. This is a default scoping model for languages that were being used before people realized it was problematic.

00:03:06.960 You can view object orientation and the way instance variables are used as a sort of dynamic scope. In this sense, the object state represents the current environment. When you look at a method definition, it might seem very simple, and everyone knows what it does. But in fact, at runtime, you have no idea what it returns. Stepping back and reflecting on this highlights the complexity underlying a seemingly simple piece of code.

00:03:35.580 If that method is mixed into a class, the class may already have instance variables set, and if it’s already in a class with other methods called, you won’t know how those could affect the state. As Rubyists, we are used to writing code like this, but we don’t think much about the mental overhead it takes to track different state variations and their implications. Even if you feel like these cycles aren’t worth thinking about anymore, they consume resources that could be spent on other tasks.

00:03:55.800 Yet, using dynamic scope is not inherently bad. What it provides is modularity, but at the cost of reducing reasoning about the program, especially in functional programming. If you want to introduce something as an argument in functional programming, you’ve got to thread that argument through all other methods calling it. This is a trade-off that I think Rubyists and many users of mainstream programming languages should consider.

00:04:19.740 Another factor that complicates reasoning is mutability. In our example, an initialize method is called, and later you might call a foo method. You’re not guaranteed that it will return the symbol :foo because other methods might have changed the state. The same scenario occurs when ensuring that something returns a consistent result.

00:04:31.340 In the synchronous world, you can assume that the return value will be consistent, but in the concurrent world, this is not valid as threads change state in the background. You might expect this code to return the symbol :foo, but if a thread changes the state right after that value is set, it could return something entirely different, like a reference to a fictional character in the video game Tribes.

00:05:05.400 Now we have a savior—the declarative model. In its most ideal abstract sense, the declarative model focuses on writing specifications or blueprints of what you want the algorithm to do rather than explicitly writing out how it works. In a general sense, you might envision an advanced artificial intelligence figuring out how things should happen based on context. But in the declarative model, if your specifications can change meaning at runtime, implementing an algorithm for that becomes complex.

00:05:32.520 This is why, when discussing declarative programming, the context of mutable state comes up, as immutable state is a fundamental crux, allowing for referential transparency. This means each time a function is called with specific arguments, it will always return the same result. Generally, the declarative model is a superset that includes functional programming, which emphasizes referential transparency.

00:06:08.520 Now, stepping back from concurrency, we will focus on declarative synchronous programming. For those of you familiar with purely functional languages, you’ll see that you aren’t allowed to reassign variables once you have bound them. For instance, if you declare my_var and then try to rebind it, you are stepping outside of the declarative model.

00:06:30.840 In languages like Erlang, you would encounter a runtime error if you tried to rebind variables. In the case of dynamic languages, such as Ruby, variable manipulation is more flexible, creating room for potential confusion. The mutability of Ruby has allowed its frameworks like Rails and ActiveRecord to thrive, and while one may not want to position Ruby against other paradigms, it’s advantageous to evaluate problems through different lenses.

00:07:12.360 The declarative model being utilized is based on ideas from an obscure programming language called Oz. When introduced to the library, we also won’t use the usual equals sign for assignments since that would breach semantics. Instead, we employ the 'unify' concept, which might seem alien but can be interpreted as a binding operation. For example, if we unify my_var with 'bound', it is akin to saying my_var = bound.

00:08:04.260 When we attempt to rebind my_var, it results in a data flow unification error as there’s an internal safeguard preventing this action. While it’s possible to reset the variable yourself, doing so would lead us away from this declarative model of equity. Still, this declarative structure allows for both models, so you can choose the best model relevant to the problem you are solving.

00:08:51.179 The alternate syntax for instance variables in this context lets us declare my_var which functions like a reader method, but can also be unified. In discussing declarative concurrency now, this adds an exciting layer to programming models available to Ruby developers. It feels as if a whole new magical layer of functionality is being unveiled. Here’s an illustration of that excitement in a playful way.

00:09:42.360 For example, when you begin setting up concurrent data flow, you want to assert that various states and changes propagate correctly throughout multi-threaded executions. Unlike traditional programming structures, where state modifications garner race conditions, this model follows a clearer lineage of dependencies. One might not realize it yet, but soon you’ll see that this concept unlocks entire frameworks of working with simultaneous data flows.

00:10:43.560 Let’s delve into an advanced scenario where you’re going to instantiate various threads creating multiple outputs. This time you might wrap a need_later block around each action. Testing concurrent threaded code is essential, ensuring each component behaves as expected despite operating on separate threads with their unique states. This testing framework assures consistency and reliability across all processes.

00:11:01.560 We can explore additional features by directly passing a parameter to a threading worker while also allowing the unifying processes upon completion. Asynchronous programming becomes smooth, almost seamless within this model. The library allows us to mitigate delays traditionally present in web development, turning test outcomes into expected and optimized results.

00:12:03.180 Envisioning the application of future queueing methods introduces a lens of specification for operations. By analyzing how you're mapping various data points, you can break up workloads across numerous processors while leveraging memory efficiently. The ease of utilizing these data flows streamlines several repetitive tasks that have, until now, been heavily reliant on synchronous operations.

00:13:08.100 In concurrent processing, significant performance gains can be seen through simply rewriting how tasks are executed, shedding light on underutilized resources in a Ruby context. Elements from other languages can resonate with Rubyists by presenting valuable abstractions for threading.

00:13:35.460 For example, you might extend basic data structures with the future queue that supports enhanced flow control when sending and processing tasks. Each task runs in its own thread, allowing efficiency during high-load processing while maintaining modularity through established or defining methods.

00:14:14.820 Now, returning to our implementation, we use ports to facilitate a smoother integration with various models. Applications spanning from queueing systems to actors can be built on top of the originally outlined structure. This layering creates robust systems that support both timing and controlled messaging. The potential for sequential and remote messages significantly augments the programming experience.

00:14:59.460 As I draw to a close, I encourage you to remember how easy it is to experiment with simple concepts that yield complex systems. These foundational ideas liberally borrowed from prominent programming languages permit refreshing interpretations on Ruby’s operational paradigms. That not only encourages creativity but reinforces the spirit of experimentation that pervades the Ruby community.

00:15:27.540 For those interested in practical applications of these ideas, think about how your projects could adapt. Know that you aren’t confined to one specific paradigm for solving problems. Each approach yields advantages, and one could structure a Ruby app to optimize performance through expanded concurrent models.

00:16:10.440 Amidst the plethora of programming topics, I hope you leave with insights that not only guide Ruby’s future development, but also help preserve its growth. The community thrives when individuals engage with diverse methods while respecting Ruby’s core philosophies. Let's celebrate the beauty of programming languages by continually embracing their charm.

00:16:57.120 If you are curious about integrations or would like to collaborate on these concepts, feel free to reach out. I’m excited about where this could lead us creatively. As I wrap up this presentation, remember the foundational work that has shaped programming as we know it today, traversing beyond the nuances of Ruby into fields where synergy abounds.

00:18:02.880 Ultimately, presenting these parallel ideas exemplifies the essence of software engineering. Continual exploration allows us to redefine conventional wisdom while pushing boundaries. This event reminds us of our collective mission in fostering innovation and collaboration within the programming world, particularly in Ruby's unique landscape.

00:20:04.920 Thank you all for your time and enthusiasm! I hope the concepts discussed today ignite further exploration within your own work. Be sure to explore the community and share any inquiries that might spark creativity moving forward.

LoneStarRuby Conf 2009