Extending Ruby with Ruby

Metaprogramming

Michael Fairley

#programming-languages

#decorator

Extending Ruby with Ruby

by Michael Fairley

In the talk "Extending Ruby with Ruby" by Michael Fairley at Rails Conf 2012, the speaker discusses how Ruby's metaprogramming capabilities allow developers to incorporate features from other programming languages such as Python, Scala, and Haskell into Ruby. The talk emphasizes that while Ruby has its own strengths, it can also be extended to borrow useful features from these languages, thereby enhancing its functionality without needing core language changes.

Key Points Discussed:
- Introduction to Metaprogramming: Fairley explains that metaprogramming often involves techniques like respond_to?, send, and monkey patching to enhance Ruby's capabilities.
- Function Decorators Inspired by Python: The talk showcases Python's function decorators as a mechanism to clean up repetitive code and simplify transaction management in Ruby.
- An example demonstrates how decorators can manage database transactions elegantly, reducing boilerplate code and enhancing readability.
- Memoization in Fibonacci Example: The speaker illustrates how memoization can improve efficiency in Ruby and proposes a decorator-style implementation reminiscent of Python's.
- Partial Application from Scala: Fairley discusses partial application in Scala, highlighting its clarity compared to Ruby's syntax. He presents an example where Ruby can mimic Scala's style through a custom method, although he criticizes its ugliness and inelegance.
- Lazy Evaluation from Haskell: Lastly, lazy evaluation is explained through a Haskell example, demonstrating how results are computed only when needed. Fairley proposes a Ruby implementation for lazy evaluation, showcasing its potential to optimize performance in certain use cases, albeit at the cost of complexity.
- Cautions Against Extending Ruby: Throughout the presentation, Fairley notes the importance of caution when adding features, as it can lead to performance implications. He concludes that while it is feasible to import features from other languages into Ruby, it must be done judiciously to retain clarity and performance in Ruby applications.

In conclusion, this talk encourages Ruby developers to leverage the language's flexibility to integrate powerful features akin to those found in Python, Scala, and Haskell, while being mindful of the potential pitfalls of such extensions.

00:00:25.480 Hi everybody, thanks for sticking around for the last talk. I know you're all exhausted, so I’ve got a nice, relaxing one for you that’s going to be nothing but walls of code in four different programming languages.

00:00:27.840 On the other hand, we’re not going to be searching for the meaning of life, the universe, and Jason serialization. We’re going to be bending Ruby until it breaks and having some fun with it.

00:00:39.960 I’m Michael Fairley, and you can find me on Twitter and GitHub as Michael Fairley. Very clever, I know. I work at 1,000 Memories, where we help families, groups of friends, and organizations collect and digitize old paper photographs, organize them online, and then share them with the people who care.

00:01:01.160 We have an iPhone app called Shoebox that is the easiest way to scan old paper photos. You snap a picture of a photo, and we automatically detect the edges, crop them, and give you some color correction controls. You end up with a great-looking picture, and you can get it on the App Store.

00:01:11.840 We have an Android version coming out very soon, so if you’re an Android user, stay tuned. I’ve learned a lot about my family history while my parents have been scanning some of our old family photos on Shoebox.

00:01:36.520 This is my great-grandmother, who I grew up calling Nina, but her real name is actually Ruby. My great-grandfather, who I grew up calling Papa, his real name is actually Singleton. So, it’s in my blood that I should be a programmer.

00:01:49.700 So, extending Ruby with Ruby—what does this mean, and why is this the title of the talk? Well, when we talk about metaprogramming in Ruby, we often think about things like using `respond_to?` and `send` to save ourselves some typing and to DRY up our code a bit or monkey patching a library that doesn’t quite do what it should.

00:02:05.000 But metaprogramming in Ruby is actually really powerful, and when combined with some other things like executable class bodies, we can actually bring in features from other programming languages into Ruby, thus extending Ruby with Ruby. I’m going to take a feature from each of these three languages and show you how we can bring it into Ruby.

00:02:31.760 There are some good ideas, some bad ideas, and some absolutely crazy ideas in this talk, so just be warned. Let’s get started with Python! Up until about two years ago, when I started doing a lot of Ruby and Rails development, Python was my primary language.

00:02:48.520 One of the things I miss most from my Python days is function decorators. Function decorators are some syntactic sugar Python has for helping you mix in common behavior into your methods and functions, in a very similar vein to how we mix modules into our classes to use common functionality.

00:03:08.560 I’m going to show you what these are, why they’re useful, and then how to add them to Ruby. So, starting with Ruby, if we were hypothetically running a bank in Ruby, we would want a method like this called `send_money`, where we pass it two accounts and the amount to transfer.

00:03:27.480 We subtract the amount from one’s balance, add it to the other’s, and then save it. There are a couple of things wrong with this code, but the most notable is the lack of a transaction. If `from.save` fails or if `from.save` succeeds and `to.save` fails, we’ve lost the money into thin air and our customers are quite unhappy.

00:03:51.040 Active Record makes this a simple problem to solve; we throw a transaction block around it, and we’re good. Let’s look at this in Python. Now, the non-transactional version looks nearly identical. I think there are five characters that have changed, and they’re all symbols.

00:04:07.280 But if we don’t use decorators, once we add in transactions, things get very, very ugly. This is exactly what Rich Hickey was talking about yesterday in his talk on Monday when he was discussing that four of them are the actual domain logic we’re trying to express and then six of them are boilerplate for doing a transaction.

00:04:32.360 This is messy because you are required to remember all six of these lines and exactly what to type in every time you want a transaction. If you screw it up, money is disappearing into thin air, which is not a good failure scenario for your software.

00:04:48.240 What Python does give us, though, is a way to easily pass around and manipulate functions as first-class objects, and we can use that to solve the problem of the ugly code for the transaction.

00:05:03.440 We take our `send_money` method after it’s defined, pass it into a method called `transactional` that will wrap the original `send_money` method into a new method that exhibits the transactional behavior, and then we assign that method to `send_money` again.

00:05:29.320 So, all future calls to `send_money` will actually go to this new transactional version. So what does `transactional` look like? It might look like this: it takes in the function `send_money` in our case, defines a new function, and that function has all of the transactional boilerplate we saw before inside of it.

00:05:43.440 It calls the old function—the original `send_money`—and then it returns this new function. So we pass `send_money` into `transactional`, and we get out a new version of `send_money` that exhibits the transaction. Function decorators are a little bit of syntax for doing this elegantly. It looks like this: you put `@transactional` before the method and it automatically does this.

00:06:04.280 This is translated into that, so you don’t have to specify the name send to money three times; you only specify it once when you’re defining the method. Also, it’s at the top of your code instead of out of sight, out of mind, at the bottom.

00:06:25.520 What I just showed you is that function decorators and blocks can do the same thing, but I think there are a few cases where our usage of blocks in Ruby falls down. We have this very naive implementation of Fibonacci.

00:06:39.320 When you want to calculate the Fibonacci of 10, you're going to recalculate the Fibonacci of five, like, sixty-something times, which is incredibly slow and inelegant. An easy way to solve this is by memorizing our results—by caching the result of the computation and then using it whenever we need the value again.

00:06:52.680 We do this in Ruby by something like this: we initialize this memoization hash if there’s not one, and then we fill in the hash with the values of the computation keyed off of the arguments to the method. There are some gotcha cases here: if this method were to return false or nil, those actually pass through or equal, so you would need code like this to check to see if the key exists in the hash rather than relying on `or equal` to solve it for you.

00:07:10.880 Again, we have two pieces of functionality in this method; there’s the code that evaluates the Fibonacci algorithm and then there’s the code that does memoization, and these two things have been complexed together.

00:07:26.680 We could solve it with blocks. We could make a memo block like this. This is somewhat inelegant, though, because blocks don’t know the name of the method that called them or the arguments that it took, so we have to pass those along manually.

00:07:43.000 If we wanted to time this method and synchronize the access to it, we’d end up specifying this name `fib` all over the place, plus we’d end up specifying the argument all over the place. Whenever we want to rename this method or change the arguments it takes, we now have to remember to change it in three or four different places; otherwise, our code won’t exhibit the behavior we expect.

00:08:02.280 There was a solution to this called Active Support Memorizable. This is deprecated as of Active Support 3.2 and is gone in master. It was this little thing you put after your method and you call Memo fib, and it undef fib, wraps it in a new method and redefines `fib`. That sounds a lot like a function decorator, and you're right; it’s very similar.

00:08:16.080 The Python community felt very strongly this was a bad idea for a couple of reasons: one of them being that if you were to move the method elsewhere, you might forget to bring the code that was after it that actually changes how it works.

00:08:33.960 Now your code is not next to the code that runs with it, and it’s kind of awkward. I also think that you should be able to trace the execution of your code from top to bottom, and in this case, the very first piece of code that runs is at the very last line on the slide. It’s inelegant and hard to debug.

00:08:45.880 So let’s look at this Fibonacci example in Python. We have Fibonacci and we want to memorize it; we put `@memorize`, we want to time it; we put `@time`, we want to synchronize it; we put `@synchronize`, and that’s the end of the story.

00:09:05.600 So what we’re going to take the next few minutes to do is see how we can do the same thing in Ruby and have it look like this. Notice that the `@`’s have become `+`’s and the name of the decorator is capitalized, indicating that it’s a class, but otherwise, it’s the same syntax.

00:09:30.000 To dive into the code, back to our `send_money` example, we want to make this transactional via a decorator instead of the Active Record block we had. So what’s `transactional`? Transactional is a subclass of decorator.

00:09:41.000 It has one method, `call`, and `call` gets passed to the original method as well as the argument it’s currently getting called with. This is a little different than how Python does it, and we’re sort of forced into this due to the way Ruby binds methods to objects.

00:10:03.480 For the most part, it’s functionally equivalent. Inside of `call`, we run the transaction, and inside of the transaction, we make the original method call.

00:10:12.220 So what is this decorator class? What’s in there? The answer is not much; a couple of class methods. This first one is the unary plus. In the same way that you can do `-3` to make a negative number, you can also do `+3` just to indicate that it’s positive.

00:10:19.860 But those plus and minus are actually class methods on `Fixnum`. We can override this with this plus-ad notation, so when we call `+transactional`, this code runs, instantiating a new method.

00:10:34.240 It stores it away in a class variable, and then we have methods to get the decorator out and clear it. There’s a little more to the story of the context of where you put the decorator.

00:10:49.240 I’ve made it so that we have to extend method decorators in a class where we want to use them. You could have directly extended the class class itself, but I find this to be inelegant because it forces every class in Ruby to use decorators.

00:11:07.480 I think it’s a little rude to users of your library or users of a library that uses your library to force your classes to have behavior you don’t expect. So what’s in this method decorators class? There’s one method: `method_added`.

00:11:21.000 `method_added` is a hook that the Ruby interpreter calls when a new method is defined. It’s an instance method on class and it’s very useful for modifying the behaviors of methods like this.

00:11:39.440 The first thing we want to do is call super so that other libraries who want to override `method_added` get their chance to run. We get the current decorator if there is one, and we don’t do anything and return if there’s not one.

00:12:03.720 If there is one, we make sure to clear it out, so that when we redefine the method in a few lines, we don’t re-invoke `method_added`, which could end up in an infinite loop.

00:12:22.760 We have to get the original method that was just defined, so we grab it with this instance method and store it away. We start redefining the method inside of the method.

00:12:37.220 We have to rebind the so instance method returns an unbound method, which is a method that doesn’t know what object it belongs to. When you call `bind` on it, it tells it what object it should belong to, and then we call the decorator.

00:12:53.600 This is the same call we just defined in `transactional` a few minutes ago. I forgot to mention that please feel free to interrupt with questions at any point; I’ve left some time for those at the end, but I’d be glad to use those in the middle if anything doesn’t make sense.

00:13:02.500 So what else do we need to do to have a production-ready library that is identical to Python’s version of decorators? Well, we only supported one decorator, but it’d be easy enough to add a list that the decorators are added onto and then loop over it when we redefine the method.

00:13:20.220 Our decorators do not respect method visibility; when we call `define_method`, it defines a public method. So in this case, the `send_money` method would actually end up being public, which is quite bad. `method_added` is not called when singleton and class methods are defined.

00:13:49.360 Something I didn’t show is that you can pass arguments to the decorator itself, allowing you to specify some options or parameters that it needs. We could make a retry decorator that retries the method up to ten times if the method throws an error.

00:14:01.840 This would be useful when you're talking to something like the Facebook API, which throws an error on a large percentage of requests. I’ve actually done all this work for you and rolled it into a gem called `method_decorators`. I think you should use it.

00:14:22.600 I think this makes our code cleaner and easier to read, and easier to reason about. If you want to check out the source code or the tests, they’re on GitHub as well. The main reason I’ve done this is so that we can end the Ruby versus Python debate once and for all.

00:14:39.600 We can bring the features from Python into Ruby, but Python cannot bring features from Ruby into Python. So next time you’re having an argument, you have the trump card, but then you have to make sure to plug your ears and go 'la la la la' when they bring up things like NumPy and first-class functions.

00:14:56.200 Let’s move on to talking about Scala for a little bit. If you haven’t played around with Scala, I would highly recommend it. It’s a language that is very similar to Ruby in a lot of ways. It has blocks.

00:15:03.120 The result of every line is an expression, so you can return implicitly from the end of functions and things like that. It has modules and mixins—often taken for granted in Ruby but pretty rare in other languages—and it’s also statically typed.

00:15:15.680 However, it makes heavy use of type inference, so you don’t actually go around typing `button button = new button` like you do in Java. Scala has this thing called partial application, which is a general functional programming technique.

00:15:30.320 Scala’s method of doing it is really elegant and something that I like a lot. In Ruby, if we wanted to take the numbers one, two, and three and turn them into strings, we would use some code like this.

00:15:48.480 Of course, you all know that we can use the symbol-to-proc version to make that a little shorter. I’ve always found this to be quite awkward; no newbie to Ruby or Rails has ever looked at this and said, 'I know exactly what that's doing. I know what ampersand colon means.'

00:16:06.000 It’s just a piece of magic that we’ve told people, 'Don’t worry about it. It works.' If we look at the same thing in Scala, though, we go over the numbers one, two, and three and map them to a string.

00:16:24.320 This first `x` is the argument to the anonymous function that’s being defined, and the second `x` is the usage of it. But we can partially apply this function like this, and I think this is incredibly intuitive syntax.

00:16:41.600 You can look at it and say Scala is filling in this blank with the argument of the function, and it makes sense. Scala can also use this technique inside of more complex functions that have some math or other function calls, and we replace the argument with an underscore.

00:17:01.200 If we take these same complex blocks in Ruby, there’s not a good way to shrink them down to something a little more readable. It is possible, though, to shrink them down to this, which is almost identical to the Scala version, except there’s an added ampersand at the beginning.

00:17:17.160 There are a lot of bugs with this, and it also requires that the underscore be the first argument and the first thing in the block, which I don’t really like. But with a little bit of added ugliness, we can actually make something that is equivalent to Scala’s partial application.

00:17:37.640 So, you’ll notice this is incredibly ugly. I think it’s a horrible idea, but it’s interesting that we can do this. I’m going to break the syntax down a little bit just because it’s kind of awkward. We have a block that is what we want to partially apply, and we pass it to a function called underscore.

00:17:53.840 The result of that function, underscore, has `to_proc` called on it by the ampersand, and then that results in the block getting evaluated in an interesting way that we’ll show in a minute.

00:18:14.000 So, what’s the first underscore method that I talked about? It’s simply a method defined on object that returns an imp partial application. Like I said before, I think sticking methods into base classes is probably a worst practice, and this is bad code.

00:18:30.000 But it’s interesting. So what is this partial application that we’re building here? We pass it a block, and it stores that block for future use. When the ampersand results in `to_proc` getting called on it, we have to return something that’s callable.

00:18:45.000 So, we define a new lambda that is called each time we want to evaluate this block. Inside the lambda, we define a new execution, which I’ll tell you what that is in a minute, and we pass it the arguments that the block was called with, as well as the block’s original binding.

00:19:08.000 A binding is the context in which the block was defined. We often talk about closures as being a function with some context; this binding is that context, and we can use it in `eval` or `instance_eval` to run some code as if we were inside of the block.

00:19:30.000 This will become useful in a minute. We then `instance_eval` the block on this new execution we’ve defined. So what’s an execution? Execution is an instance of BasicObject. If you haven’t seen BasicObject before, as of Ruby 1.9, it is the superclass of Object.

00:19:50.600 It has almost no methods defined on it, so if you call `BasicObject.new.class`, you actually get an error because it doesn’t have a class method and doesn’t respond to all these things we take for granted. It's a blank slate to work with.

00:20:11.000 So it gets passed the arguments that the original block was called with and also that block’s binding.

00:20:31.360 Because we `instance_eval` on the previous slide, this `method_missing` will be called for all of the names inside the block we don’t know about. For most of the names, it’ll get called, and we use `instance_eval` on the binding to pull the original method out of the context in which the block was defined.

00:20:50.600 Then we call that method with the arguments that were passed. If the method that was called is underscore, we instead shift off an argument; so `args` here is going to be the number one, perhaps.

00:21:07.520 We can fill in the underscore with the arguments that were passed in, which is exactly what partial application is. Let’s look at this again: we have this underscore method that builds a new partial application with the block it’s given.

00:21:25.440 The partial application has `to_proc` called on it and returns a lambda that builds a new execution. This will evaluate the block when this block is getting evaluated. `method_missing` is called with underscore, and this underscore method returns the number one, two, or three.

00:21:41.680 When `rand` is called, it’s reevaluated in this context. We run this in IRB, and we get one, two, and three plus random numbers. Again, I think this is ugly and bad, but it’s kind of cool that we can do this, and it’s a testament to how flexible and powerful Ruby is.

00:21:58.960 Let’s move on to talking about Haskell for a little bit. Haskell is a very different language than Ruby, and I’m not going to go into too much of that, but it has what's known as lazy evaluation, meaning that the result of some code you write is not evaluated until its result is needed, even if you call that code.

00:22:13.160 Let’s see this in an example. This is the only Haskell I’m going to show you, I promise; it’s not fun to look at. We have this method `rails_comp`, and we declare that it’s going to take an argument of type `IIO, Network Stream Result Response String`, which you should just think of as a response from an HTTP request.

00:22:37.080 It returns something of type string. So, we define the implementation of this method; it takes the argument `response` and is going to return the string "I'm lazy." We put this in a script, define the main method to print out the result of calling this `rails_comp` method with the result of this HTTP request.

00:22:55.360 I ran this code on my laptop, and it printed out "I'm lazy." I turned off my Wi-Fi and ran it again, and it still printed out "I'm lazy." So either Haskell doesn’t need a network connection to make HTTP requests—which would be really awesome—or there’s something else going on, which is lazy evaluation; we never actually used the result of that HTTP request, so the request was never made.

00:23:22.760 Let’s contrast this with Ruby: we have the method `rails_comp` which takes in an HTTP response and returns "I'm not so lazy." We have a method to make an HTTP request given a URL, and then we print the result of making that request and passing it to `rails_comp`.

00:23:40.680 If you run this code, it prints out "I'm not so lazy." You turn off your Wi-Fi, you run it, and it errors. But we can define this thing called lazy, which will let us be lazy in the same sense that Haskell is.

00:24:02.040 The block passed to `lazy` will only be evaluated if the result from it is actually needed. Let’s see how we can build this in seven lines of code. We store the block; and again, this is a `BasicObject`, so whenever any method is called on it, `method_missing` gets triggered.

00:24:19.920 `method_missing` will evaluate the block once and then send on the method that was called on it. In future calls of `method_missing`, it’ll just send along the method. This is very similar to what Active Record does with associations.

00:24:42.960 When you call `blog_post.comments`, it doesn’t actually make a SQL query right then and there. When you call `blog_post.comments`, all it does is execute a query and return an array. If you call `blog_post.comments.class`, it executes the SQL query, and the result you get is an array.

00:25:01.440 Even though that `comments.class` clearly can’t be an array, it’s doing something almost identical to this. Let’s look at another example; we have this method `three` that lazily prints out the number two and returns three.

00:25:15.440 We run the method three and assign its result to `x`. We print one, and then we print `x`, so this will actually print out one, two, three because of the lazy evaluation.

00:25:31.080 So when we assign `x` to three, `x` is actually going to be this lazy block, and then at the bottom, when we try to print `x`, the block runs, printing two and then returning three, which then gets printed by the last line.

00:25:48.840 If you’re paying close attention, you realize lazy could just be a decorator. I promised I would talk about the good, the bad, and the crazy, but we haven’t really seen anything crazy yet.

00:26:07.600 So, remember I said that Haskell applies this laziness everywhere? We can loop over every class in Ruby and every method inside of those classes and redefine them to be lazily evaluated.

00:26:28.560 You have to make a couple of exceptions; for example, `puts` obviously can’t work lazily, and the methods we’re using here can’t be lazy. But after you do this, you can actually write working programs where you do `puts 1 + 1`, and that plus is evaluated lazily.

00:26:49.840 But you still get the result, too, and it’s incredibly slow. However, there are some places where this laziness technique could be useful, not the full replacing everything, but just our blocks.

00:27:06.760 So, pretend we have a Rails app. This is funny; this is the first Rails code I’ve showed at RailsConf. Our action gets some tweets from the Twitter API, and then our view loops over those tweets and prints out the author and the tweet.

00:27:25.680 In Rails 3.1, they added something called streaming responses, meaning that it can actually start sending back the result of rendering your page before the page has fully been rendered.

00:27:45.000 This is incredibly useful because the things at the top of your page are often your CSS and JavaScript files that require extra requests to get. If you send back the first two kilobytes of your HTML, the browser can start fetching those JavaScript and CSS files before the rest of the pages even come down.

00:28:02.680 The problem here is that even if we were using that technique, we have to wait for the controller action to finish, and hitting the Twitter API can take a long time. But we could wrap this request to Twitter in a lazy block.

00:28:20.560 You’ll notice we didn’t have to make any changes to the view, but now we can use the streaming response to render the top of the page. Once it tries to render the tweets, it’ll make the request.

00:28:37.760 The moral of the story is next time you hear your coworker say, 'I really wish we could use Java-style annotations right here,' or 'tail call recursion would make this code so much cleaner,' realize that we can build these things into Ruby ourselves.

00:28:56.560 Ruby is incredibly flexible, and I've just shown that we can bring in features from other programming languages into Ruby. It’s often a really bad idea, and there are often performance implications.

00:29:12.880 But there are several places where it absolutely makes sense, and that’s why Ruby is cool. I’m Michael Fairley; I work at Thousand Memories, and you can find me on GitHub and Twitter as Michael Fairley.

RailsConf 2012