Compiling Ruby to Binary

Talks

Torsten Rüger

#ruby-implementations

#ruby-vm

#ruby-on-rails

Compiling Ruby to Binary

by Torsten Rüger

In the video titled 'Compiling Ruby to Binary,' presented by Torsten Rüger at Ruby Unconf 2019, the speaker explores the intriguing concept of compiling the Ruby programming language into binary format. Rüger, who transitioned from building wooden frames to programming, passionately discusses the challenges and possibilities inherent in compiling dynamic languages like Ruby. He aims to demonstrate that Ruby does not strictly have to be interpreted, countering common misconceptions about dynamic languages.

The key points discussed in the video include:

Introduction to Compiling Ruby: Rüger begins by explaining the essence of compiling Ruby into binary, emphasizing the potential for speed improvements over traditional interpretation.
Comparison with Other Languages: He notes the 'two-language problem,' where developers must choose between usability and performance, highlighting the inefficiencies of interpreted languages, particularly his experience with Ruby on a Raspberry Pi.
Basic Compiling Process: Rüger outlines that in a compiled language, the compiler converts source code into an executable binary, using a 'hello world' example to illustrate the compiling process.
Foundational Structures and Challenges: He describes early success in implementing basic structures while acknowledging hurdles related to Ruby’s dynamic nature, especially with runtime and dynamic method invocation.
Compiler Structure Layers: The compiler is discussed in layers, where a Ruby parser converts code into an untyped data structure and subsequently into typed versions leading to machine code.
Error Handling and Object Management: Rüger speaks on handling exceptions and the unique object-oriented model he has developed, which treats all data as objects, simplifying lower-level manipulation.
Debugging and Visualization: The project employs an online visualization tool and supports debugging by allowing users to monitor compilation and diagnose outputs interactively.
Performance Benchmarks and Community Involvement: He shares preliminary benchmarking results, noting that while the current performance is slower than C, future developments will allow for optimizations through runtime compilation.

In conclusion, Rüger invites the community to contribute to the ongoing project, underscoring the significance of open collaboration and the potential that compiling Ruby holds for both improving performance and expanding the capabilities of the Ruby language. His presentation serves as a call to action for Ruby developers to explore these new possibilities together.

00:00:02.540 Welcome everyone! As we learned from the previous session, we start punctually instead of six minutes late. It's now 1:00 sharp. Thank you for being here.

00:00:09.380 Great! You all made it back. Thanks for joining us. Up next, we have Torsten. Torsten actually builds wooden frames for a living before he started programming. So, please give a very warm welcome to him.

00:00:29.590 Hi there! Sorry, that was actually the wrong way around. I now build houses after coding for a relatively long time. Today, I’m going to show you how it's possible to compile Ruby to binary. This ties in nicely with what was mentioned earlier about not letting anyone tell us what's possible or not. People often claim that you need to interpret a dynamic language, and that’s just not true.

00:01:09.909 A little about me: You can find me on GitHub, although I’m not very social. I've been coding on and off—not professionally—mostly as a hobby. This project is truly a passion for me. I'm originally from Finland, and here's a typical Finnish way to enjoy your time, which may be seen as odd in other parts of the world.

00:01:37.719 I run a bed-and-breakfast halfway between Helsinki and Lapland with my wife, which has allowed us to spend some enjoyable time together. In the next half hour, I will show you that compiling Ruby is not difficult, although it can get relatively deep. I’m trying to find the balance between giving you an understanding of how it can be done without cheating and sticking to 100% Ruby, which would amount to about 10,000 lines of code.

00:02:38.150 I love Ruby and have for quite some time. I hope this could become one of the many contributions that the Ruby community brings to the larger programming community. Compiling dynamic languages like Ruby remains a challenge, primarily due to performance issues. Does anyone here know about Smalltalk? I found Smalltalk's approach quite impressive when I read about it in the 90s.

00:03:20.970 The major problem Ruby and most other dynamic languages face is speed, particularly when compared to big companies that devote lots of resources to enhance performance. I experienced this firsthand when I first bought a Raspberry Pi. Essentially, Ruby turned my laptop into a Raspberry Pi and a Raspberry Pi into an Arduino, which shouldn't happen—it all results from being interpreted. This exemplifies the classic two-language problem: having to choose between a nice language and a fast language. With this project, I hope we won't have to make that choice anymore.

00:04:34.710 By compiling Ruby, we can address these performance issues effectively. A faster implementation means we wouldn't have to choose between beauty and speed. I often refer to this phenomenon as the 'Rails effect', where developers empowered to modify their tools achieve a significant boost in productivity.

00:05:03.620 Before diving into the compiling part, let’s quickly review some basics. Any program takes input and produces output. In MRI Ruby, your source code acts as input that produces output as well. However, in compiling languages, we have two stages: first, the compiler processes a source file and produces an output file, which is binary.

00:05:36.530 So if we were to do this in Ruby, we would create a hello world program and run the compiler I’ve written, resulting in an executable binary. This approach is feasible with the code I created. Subsequently, we can execute the hello program, which functions seamlessly, similar to how one would expect a more complex program to run.

00:06:58.370 To achieve this with a Ruby compiler, we need to create a program called Ruby. When we call Ruby, it’s the output of another compilation process. In some architectures, we need to compile the compiler itself first. How many people are familiar with this concept? This is how many decent compilers are structured—the C compiler follows a similar process in the initial step.

00:07:43.820 I will delve deeper into this project today and discuss the different components involved. As of now, basic structures and classes are working. We have a calling convention that is easy to understand compared to the C calling conventions, which I will illustrate later. There’s also a memory layout and a process for creating binaries, which I won't cover in deep detail since there's a standard (the ELF standard) that specifies how binaries are laid out.

00:08:25.010 We have functional control structures including if-else statements, and while loops. However, no breaks or continues yet but they are part of the process. Dynamic method sending is key; it involves identifying the method to invoke and managing arguments, all while ensuring efficiency through caching.

00:09:03.660 I have even been able to implement blocks. In Ruby, blocks are implicit, allowing us to capture and pass them as arguments, performing ‘where’ and ‘return’ operations seamlessly. Ruby is a complex language and although there are features that haven't been implemented yet, the foundational aspects were crucial to tackle first.

00:10:15.360 Regarding runtime compilation for procs, it’s a bit trickier, but exceptions can be handled similarly, where you’d need a properly defined calling convention. The beauty of structuring a new Ruby compiler from scratch is that you can keep it simple and designed for today's needs.

00:10:51.390 Every object in this environment is well-defined. Has anyone worked on extensions or explored the MRI code? Unlike MRI where everything pretends to be an object, in RubyX, everything is genuinely treated as an object, even data types like integers. This allows for a standardized approach to managing lower-level functions and data.

00:12:09.200 Consequently, we face challenges due to Ruby’s dynamic nature. We can change classes at any moment, equating classes with types presents issues, as types must remain immutable. Each object possesses a type, which does not change, while classes constantly evolve; therefore, managing type references is crucial.

00:13:11.520 Let's summarize: the compiler is structured in layers, similar to any program. Starting with Ruby code parsed by a Ruby parser, which produces an untyped data structure before creating a typed version. Ultimately, we can generate machine code for the compiled output.

00:14:11.530 The compiler comprises various layers where each processes essentially folds down into greater details. Importantly, I have created a virtual object-oriented language to handle the complexity of Ruby, eliminating redundancies and simplifying operations. We limit to straightforward operations like 'if', avoiding excessive syntaxes that Ruby typically offers.

00:15:43.390 Moving on, machine layers utilize instruction lists instead of syntax trees, focusing on linear processing without complex structures such as control or memory variables. The architecture consists of a minimalistic model that can process tasks logically.

00:16:09.990 The ARM and RISC architectures are implemented for broader compatibility, and I aim at creating an interpreter for easier developing of the compiler itself. As a result, debugging binaries becomes more manageable, although the process might seem daunting initially.

00:17:52.091 By providing a visual interface for the interpreter, debugging becomes intuitive. The project is structured to enable tests, which is significantly helpful, and there's an online visualization tool that helps demonstrate this functionality.

00:19:31.630 The project represents about 500 lines of code and employs Opal to visualize activities within the RubyX project. Through this interaction, programmers can single step through the compilation, monitoring how objects load into registers and ensuring accurate evaluations.

00:20:21.850 In this debugging environment, users spot instructions and visualize outcomes such as 'hello world' and relevant return values produced by executed code. This method of testing results in comprehensive visual feedback.

00:21:31.530 There’s one last question opportunity. Yes, let’s take one. So you’re inquiring about benchmarks? While this isn’t a full implementation yet, I carried out preliminary micro benchmarks. The benchmarks indicate it runs at about 50% slower compared to C, but runtime compilation can substantially increase speed by inlining.

Ruby Unconf 2019