Compiling Ruby

Metaprogramming

Compiling Ruby

Kevin Newton

@kddnewton

#compiling

#ruby-vm

#instruction-sequence

#yarv-yet-another-ruby-vm

#performance-optimization

#abstract-syntax-tree-ast

#metaprogramming

Compiling Ruby

by Kevin Newton

In this talk titled 'Compiling Ruby' presented by Kevin Newtown at RubyConf 2017, the focus is on understanding the Ruby execution process and how recent enhancements to Ruby can improve performance and readability. The discussion begins with the execution flow from source code to output, detailing how Ruby interprets a program through various stages:

Code Requirement: Ruby starts by requiring code to be executed, which is subsequently read and tokenized.
Tokenization: This stage involves breaking down the code into recognizable tokens, although semantic meaning is not applied yet.
Abstract Syntax Tree (AST): A tree structure is formed from the tokens, adding context and meaning to the code.
Instruction Sequences: With Ruby 1.9, the process evolved to create instruction sequences from the AST, allowing more efficient execution.

Newtown highlights the evolution introduced in Ruby 2.3, which allows compiled instruction sequences to be saved as binary files for later execution using the newly added RubyVM::InstructionSequence API. This enhancement enables:
- The possibility to load pre-compiled instruction sequences, circumventing the initial compilation steps.
- Improved performance through intelligent execution of instruction sequences, facilitating optimizations within Ruby’s virtual machine.

The presentation features examples of the compile_file method and how to utilize the to_binary function, followed by considerations around machine-specific binaries. The importance of checking whether files are up to date before recompiling is also discussed.

Kevin also introduces practical applications, such as the 'BootSnap' gem, which showcases efficient loading of instruction sequences and presents an innovative gem called 'Vernacular'. This gem allows developers to implement their custom Gsub methods that hook into the compilation pipeline. Newtown emphasizes:
- The potential for Ruby to extend capabilities through flexible operator overriding and introduce efficiency back into the coding process.
- The ability to investigate and manipulate the Abstract Syntax Tree for better performance and code clarity.
- The importance of creating a robust framework where programming constraints can be defined for better functionality.

In conclusion, Newtown encourages the Ruby community to explore new practices to enhance Ruby’s efficiency and maintain its usability in an evolving programming landscape, urging for continued discussions on innovative methods to improve code quality and performance.

00:00:10.429 So I think we're going to get started. Everybody smile! Since I'm a millennial, I feed off of avocado toast and Facebook likes.

00:00:20.250 Feel free to like that on Twitter! It'll be a hot topic! Hi, my name is Kevin Newtown, which I use on the internet.

00:00:26.060 If you can't remember my last name, just remember it's in alphabetical order! I only recently discovered that, 27 years later.

00:00:39.270 I work at a company called Culture HQ. We build better workplace communities. If your workplace could use a better community, come and talk to me; we have a lot of fun.

00:00:46.739 Today, we're going to talk about Ruby, specifically what happens when Ruby executes a program. If my math is correct, that will output 10. Let's go to our command line and see 10 printed!

00:00:59.550 We will discuss everything that happens between when you type 'ruby example.rb' and when that 10 gets printed. What we are focusing on is the Ruby execution process. There are many parts to this process, starting with code getting required. After the code gets required, the source is read, tokenized, and an abstract syntax tree (AST) is built.

00:01:18.930 We then interpret the abstract syntax tree. This was the case starting from Ruby 1.8, but that approach is a bit old now. Nowadays, we build instruction sequences, which are executed. This is the flow that your code goes through when Ruby is executed.

00:01:47.130 Let's delve into tokenization. Tokenization is a well-discussed topic in computer science. It is part of lexical analysis, which is responsible for running through your program and identifying various tokens. As it processes, it builds a comprehensive list of tokens. Notably, while in Ruby, 'puts' and 'a' represent two distinct semantic types—'puts' being a method call and 'a' being a variable—lexical analysis does not yet have this semantic meaning.

00:02:22.849 Now, with our list of tokens, we can build an abstract syntax tree. Starting at the beginning, we create a root node that matches the longest pattern available. For example, it recognizes 'ident op equals integer new line' as a local variable assignment. From there, we can see the next pattern is a method call, which is also an expression. Thus, we build our abstract syntax tree, injecting semantic meaning into what was a mere list of tokens.

00:02:46.909 Next, we prepare our instruction sequences. In Ruby versions prior to 1.9, the process was one of immediate interpretation following the building of the abstract syntax tree. You needed two main components during this: a state, representing a global state for your program while interpreting, and a stack for the stack-based virtual machine being used. So, when we type 'ruby example.rb', we traverse down our tree.

00:03:11.780 Starting at the leftmost node, we encounter an integer and push that onto the stack. Next, we see our local variable assignment—'a'—which we also push onto the stack. Given that it's a local variable assignment, we pop that off and it gets placed into our local table. Moving further down the tree, we encounter another integer, let's say 5, and similarly, we push this onto the stack along with the local variable 'a', pulling it from our local table and replacing it with 5 before sending the '+' operator to it.

00:03:47.590 Thus, we obtain 10, which is yet another output pushed onto the stack before reaching the 'puts' statement. The implicit receiver, 'self,' is then evaluated with the resulting integer, and successfully, we interpret our abstract syntax tree. This execution process changes with Ruby 1.9, which introduced YARV, or Yet Another Ruby Virtual Machine.

00:04:18.470 YARV does not interpret the abstract syntax tree directly. Instead, it builds instruction sequences from the abstract syntax tree, much like creating assembly code from C. This changes the execution process by allowing us to optimize a lot more than previous versions. However, we trace through the process to see how it works. Previously, 'put object' was an operation that pushed objects onto the stack. Now we have an instruction that directs 'set local' where the compiled instruction set has been developed.

00:04:56.930 From here, we can execute these instruction sequences in a more intelligent manner, with room for all sorts of optimizations. For instance, a threaded VM approach is implemented where previous instructions can point to the next ones, enabling quicker execution through the entire process. The flow remains similar, but you will notice that efficiency is greatly enhanced, and numerous performance enhancements result from this change. I'm not an expert on virtual machines, but it's clear when you observe a significant speed increase that something has fundamentally improved.

00:06:00.950 With Ruby 2.3, we introduced an exciting new feature: the ability to take the compiled instruction sequences and write them out to a binary file for later execution. This was introduced by [unknown speaker], a year and a half ago. If you browse through the Ruby bug report, you can find the details—it’s quite impressive and serves as the foundation for the remainder of this talk.

00:06:20.470 Let’s consider an example: in Ruby 2.3, we have a new API called 'compile_file' on the 'RubyVM::InstructionSequence' class. This function produces an instance of the instruction sequence, and you can call 'to_binary' on it to output a binary string. If you inspect that file, you'll notice it is considerably larger than the original. However, it contains all the necessary semantic information inferred from the source code that allows it to be executed elsewhere.

00:07:04.330 A couple of caveats must be considered: this binary file is machine-specific; it holds architecture-specific data. Additionally, certain identifiers will differ across machines. Therefore, while you might lift that file and run it elsewhere, you may not achieve the expected results. However, we can load this file from a binary state, and upon evaluating it, we'll obtain the same output of 10. As mentioned, a significant enhancement offered in Ruby 2.3 is that we can programmatically load those instruction sequences.

00:07:36.970 This means we can select from two paths. The first path is the standard one, where we typically require a file, read it, tokenize it, and construct instruction sequences. Conversely, instead of executing them directly, we might write those instruction sequences to a binary file and only execute them when we need to. Then, when requiring that file again, we retrieve and execute the pre-written instruction sequences.

00:08:05.270 Internally, Ruby does not require you to understand C in order to make use of this functionality. Essentially, if 'RBI SEC Lodi SEC' returns a value for a specified filename, it will skip all other code construction and immediately evaluate the existing instruction sequence. This mechanism checks whether or not the function exists on the RVVm::InstructionSequence singleton class and rewards efficiency by allowing it to return that immediately.

00:08:43.100 For demonstration, I’d like to walk you through an example of integrating this 'load ISEC' function. This will identify every single file that is required. Each time a file is loaded, it runs through this class. The 'source_path' variable denotes the file path you wish to compile. This mechanism helps determine if the file is updated lending itself to the efficiency of not recompiling code we already have, thus speeding up the execution. It’s vital to check whether the ISEC file has been revised recently compared to the source code. If so, we can simply return the instruction sequence loaded from binary.

00:09:07.750 Alternatively, if it has not been updated, we will compile it anew, thus Python will go through the regular process. In order to ensure total execution safety, we can also rescue both syntax errors and runtime errors, returning nil for any failures. This provides a robust fallback.

00:09:43.980 Several successful examples already exist in the field. One is BootSnap, a gem coming out of Shopify that compiles your ISEC files if instructed to do so, efficiently loading them appropriately. You can find classes that integrate with Ruby's VM instruction sequences. Another gem worth mentioning is YomiComo, also included with the Ruby installation—check inside the resources for a handy example loader.

00:10:04.150 Now, let’s focus on an exciting aspect. I was observing all this functionality and, for reasons that will become clear once we start diving deeper, I felt compelled to experiment with it and explore the boundaries of this compilation process. By going back to the 'Load ISEC,' I realized there were other elements at play, so we started looking deeply into compiling and inspecting the instruction sequences, splitting them apart for better comprehension.

00:10:54.500 At this stage, we examine the instruction sequence compiled part—a great opportunity! What we can do here is we can handle content as a variable that holds the actual Ruby source in a string. The next line compiles it into an instruction sequence while allowing us to use methods like Gsub for arbitrary substitutions, demonstrating the extensibility of Ruby language. This allows greater flexibility.

00:11:20.150 The question arises: why would someone go to such lengths? Ruby's inherent flexibility allows for complete operator overriding—something we don’t recommend. However, this flexibility means that Ruby's virtual machine cannot optimize constants, like 24 times 60 times 60. It simply doesn't know whether the multiplication operation has been overwritten at that instance.

00:11:56.300 Instruction sequences would contain a lengthy list of operations: first an integer, executing put object instructions and then multiplication. This procedure balloons memory usage and execution time. Many programming languages incorporate an instruction elimination process, allowing optimization by substituting sequence outputs into their values. In Ruby, the language has to sacrifice some of that due to the inherent flexibility that it provides.

00:12:28.450 In closing, I recognize that while language design should allow clarity and visibility of code but, at the same time, it is essential not to sacrifice efficiency unnecessarily. Thus, I've created the Gsub method to replace some of these constructs to reintroduce semantic meaning without incurring extra overhead. We can look at literals like dates, which are notoriously slow in Ruby and can be improved dramatically without compromising visibility. Instead of using 'Date.parse', you could take advantage of 'STRFTIME' formats which expect a particular format and parsing it.

00:13:09.736 I want to define a method that performs that during compilation time—this, of course, can be achieved in numerous ways. You could abstract more complex date handling into one variable, but I appreciate the readability. Moreover, if you recognize an underlying pattern from Elixir, you can apply it here! This yields precisely the outputs needed without unnecessary complication.

00:13:53.000 The takeaway is this: I've built a gem called 'Vernacular' that extends this process further, allowing developers to write their arbitrary GSub implementations that can hook into the compilation pipeline. When executed, it allows for innovation in code usage. We should think beyond lexical analysis and enable more profound semantic enrichments within our Ruby code.

00:14:39.420 This has gotten pretty outrageous in terms of complexity, but now we have a way to inject our semantic motivations into our coding processes and fend off the impacts of inefficiency. It becomes essential to extend the Abstract Syntax Tree (AST) while rewriting it efficiently to fit our purposes and outputs. Delving into the bowels of Ruby’s parser opens a door to profound optimizations possible.

00:15:43.790 We can reconfigure Ruby's AST, altering how particular constructs interact and the behavior they exhibit. Let's construct a more dynamic system in terms of validation, demonstrating how we can impose constraints directly within the language for better overall functionality. Consider the common request for type checking within the language—though not inherently supported—you could redirect the behavior in Ruby to uphold those conditions.

00:16:54.180 You could expand this to ensure outputs of functions hold to specified types or return errors when conditions aren’t met. Wrapping each execution of such functions in a protective measure would lead to superior code integrity. My goal is not simply to bolster performance but create approachable elegance in defining constraints necessary in programming logic.

00:17:32.000 In summary, through the capabilities added via the Vernacular gem and beyond, we can motivate better programming practices. Using alternative paradigms can optimize our workflows—Make what you need concise and clear. For non-native extensions, the whole Ruby structure can be observed to preside over methods rather than only reflective constructs, ensuring legibility.

00:18:10.320 The fundamental change initiated encourages ongoing discussions within the Ruby community. If you possess the requisite knowledge on interacting with the parser, you're participating in creating an efficient pathway toward desirable function architecture. Despite being unique, treatment through avenues of thought highlight technical depth.

00:19:08.160 What I ultimately hope to achieve with this compilation process is to create a well-established hub of expressive functionality accessible to all. Coming from different programming cultures and diversifying the Ruby landscape promotes growth and sustainability. To further inspire the development of Ruby, we desire improved job prospects and increased utilization.

00:19:44.200 So, to conclude, I ask everyone to consider how we can inject new programming practices into our Ruby work. A bright future ahead for Ruby awaits and the conversation grows richer when many force open the doors of innovation. Thank you!

00:21:02.960 We have some time for a Q&A.

00:21:09.560 I’ll anticipate two questions: one, yes, I’m using this in production, get over it; and two, yes, if MG8 comes about, it will die, but that’s sad.

00:21:20.160 Any other questions? I don’t have too much time left, but I’m here for it.

00:21:29.970 What’s the most compelling use of Vernacular in production?

00:21:35.080 The answer is testing macros! I have an all-caps debug macro that requires IRB binding, as it can cause a little nuance, allowing for smooth testing capabilities.

00:22:08.220 I’ve made the adjustment to emphasize that this functionality can resonate deeply within practical testing scenarios; it streamlines lengthy descriptions into compact environmental setups.

00:22:30.450 It also integrates well within functional testing practices that contrast standard object-oriented approaches, leading to more refined results and elegant code structures.

00:22:59.800 Moreover, the necessary raising of syntax errors and runtime errors during compilation helps reinforce important practices for returning nil should errors arise. This provides a fallback during development to overcome errors gracefully.

00:23:43.940 Hence, the compatibility with the parser library extends beyond the norm. I've formulated a scheme so the interactions on loading the parser gem and its execution won't collapse due to inevitable complexities.

00:24:04.510 In that way, we nurture not only collaboration but extend a rich language of functionalities. It hands programmers the agency to produce more effective results.

00:24:24.560 Great! Sounds like I’ve covered everything I wanted to discuss without losing track. If anyone still has lingering questions, feel free to approach me, as I have plenty of time.

00:24:46.150 Thank you!

RubyConf 2017