00:00:01.760
Hey everyone, welcome to my RubyKaigi 2020 Takeout edition talk. My name is Vladimir, and today I'm going to answer some questions regarding transpiling Ruby.
00:00:09.679
Ruby is our favorite language, and we're going to talk about its present, past, and future. We're also going to discuss a specific tool called a transpiler. So, what is a transpiler? A transpiler is a program that translates source code from one language to another at the same level of structure.
00:00:27.119
It is usually called a source-to-source compiler because it's a specific case of compilers. The input and output source code is usually of the same language. One of the most popular transpilers is Babel from the JavaScript world, and it's due to Babel that the word 'transpiler' became popular. Another transpiler from the front-end world, like PostCSS, is a tool for transforming CSS, which allows you to use modern syntax in older browsers.
00:00:57.441
But how does all of this relate to Ruby? Why are we talking about such hackery as transpiling for a primarily server-side language? Why do we need a transpiler for Ruby? Let's take a look at the current Ruby feature set.
00:01:14.880
Currently, Ruby 3.x is in development, and its working version number is 2.8. I will use this version as the foundation for my talk, as it is going to be released in a few months. Ruby is evolving rapidly these days. The most recent release brought some new features, including new syntax, and the upcoming release will add even more new syntax features.
00:01:34.679
Let’s take a look at an example of a Ruby 2.8 program. It includes pattern matching. Pattern matching is fairly recent; it's almost a year old. You may know about it and perhaps even used it. However, many people aren’t using some of the newer features, such as the endless method definition, in their everyday lives.
00:01:50.800
We also see the rightward assignment, which is essentially just an assignment but in a different direction. This is also coming in the next version. Unfortunately, it's hard to use these features today because you need either an edge Ruby running on your system or you have to build it from source.
00:02:05.440
So, you definitely don't use these features in production, right? And maybe you would like to use them in the current version that you are using. How can we do that? Well, with the help of transpilers. We have a transpiler for Ruby, and that was one of the reasons I started working on this project: yes, I am working on Ruby Next.
00:02:23.520
I've been working on Ruby Next for about a year. The main reason for that is to make it possible to utilize modern syntax features with older Ruby versions. Today, I'll discuss why I decided to build it and how it works.
00:02:36.400
So, what is Ruby Next? In simplest terms, it allows us to demonstrate how it works before diving into the whys and hows. Assume you have an older Ruby version, say 2.5, and you want to run code with pattern matching in it. Without the magic of Ruby Next, it would raise an exception. But with Ruby Next, it simply works!
00:02:54.919
That’s what transpiling does for you: it enables you to write modern Ruby today, even if your environment is based on an older version. My name is Vladimir, and let me first introduce myself briefly.
00:03:06.560
You can find me on GitHub under the nickname Falcon, where you'll discover my open-source projects and information about my social networks and writings. I'm working for a company called Evil Martians, which consists of a team of senior developers, front-end and back-end, as well as designers and DevOps engineers, spread across the globe with bases in Japan, the USA, and Russia.
00:03:24.240
We help both large companies and small startups enhance their projects, optimize performance bottlenecks, and implement innovative ideas. Part of our work involves contributing to open-source tools, which we use in our daily lives. We strive to improve these tools and create new ones that benefit the community we love, including Ruby.
00:03:39.840
We write about everything we do in our blog, where most of our articles, including the popular ones, have been translated into Japanese, Chinese, and sometimes other languages. That's everything for the introduction of myself and my company. Now, let's switch to the main topic and discuss why we want to transpile Ruby.
00:03:57.679
I already mentioned the issue of backporting — the desire to use new features with older versions for various reasons. Let’s take a look at some data from RubyGems. As you can see, not a lot of people or machines are using edge Ruby or the latest version, even the previous ones. The majority of Ruby users, whether human or robot, are still running older versions and cannot use new features.
00:04:32.640
What does this mean for them? They are probably okay with using older syntax. However, personally, as a gem author, maintaining several dozen other gems means that I cannot just drop support for older Rubies. I have to support at least those versions that are officially supported. As of today, that’s Ruby 2.5 and above.
00:04:55.679
Unfortunately, I cannot use pattern matching with Ruby 2.5, and most gem authors cannot either. Thus, they should stick to the older version. Even if someone decides to create a new gem and supports only Ruby 2.7 and above, there is a high chance people will come asking not to drop support for older versions.
00:05:14.960
A case in point is with Hanami's API gem. This request reveals another side of the problem with integrating new features and syntax: compatibility issues with alternative Rubies like JRuby, TruffleRuby, and others.
00:05:33.920
These implementations are not always compatible with the latest Ruby features, and we have to wait for maintainers to add support for MRI features. Currently, JRuby supports Ruby 2.6, as does TruffleRuby. As for MRuby, it has limited support for 2.7 features but lacks pattern matching, for example.
00:06:01.440
Meanwhile, RubyMotion and Artichoke also do not support pattern matching. This means these alternative Ruby implementations will not be able to use new features without backporting. Therefore, transpiling can assist them in overcoming these limitations.
00:06:34.720
This isn’t the only reason I’m discussing today, but let’s save the last one for the end of the talk and switch to the next question: how to transpile Ruby?
00:06:50.840
Remember, I mentioned earlier that a transpiler is a source-to-source compiler. Generally, compilers work like this: first, we parse the source code into some intermediate representation, often an abstract syntax tree (AST). Then we perform some analysis and optimizations, before generating the final target code.
00:07:08.320
In the case of transpilers, we parse the source code, analyze and optimize it, and then generate new source code. This typically involves modifying the AST and generating a new source code from it.
00:07:32.640
Let’s talk about the first step in the transpiling process, which is generating an AST from the source code. In Ruby, we have three popular ways of doing this.
00:08:01.919
First, there are two AST APIs built into Ruby. They are a bit different, but they allow you to get some intermediate representation of the source code—not just a string—which can be modified, and you can regenerate the source code from that.
00:08:29.198
The last approach is using the standalone gem Parser. It differs in many ways from the first two approaches, and while we could discuss their disadvantages, let’s focus on the advantages of using Parser for generating the AST from Ruby source code.
00:08:48.080
Parser is written in pure Ruby, has limited dependencies, and can be used with any Ruby version. You can parse Ruby 2.7 source code while running on Ruby 2.5, for instance. No recent Ruby version is needed to parse the latest source code. Moreover, Parser is used in many syntax and static analysis libraries, such as RuboCop, which gives us confidence in its performance.
00:09:10.320
Lastly, and importantly, Parser includes rewriting support, letting you change the source code in place, without impacting everything else. One of the main issues with generically generating source code from the AST is that you can lose style information, such as spacing and layout. Parser's ability to rewrite allows us to retain these elements.
00:09:37.680
Now, let's take a look at the simplified source code for the transpile method. The whole idea of transpiling uses a concept called rewriters. A rewriter is a model or class that is responsible for a particular feature.
00:09:55.200
For instance, we have a rewriter for pattern matching, another for numbered parameters, etc. Each rewriter analyzes the AST and applies the necessary transformations to the original source code to generate new source code.
00:10:10.920
The more rewriters we have, the more transformations we can apply to the source code before it's finally transpiled. Here's an example of what a rewriter could look like. It defines callback functions for various node types within the AST.
00:10:35.840
For example, a forward arcs node is part of a method definition, while a send node represents a method call. The rewriter inspects the arguments of the method's code, and if any forwarded arguments are present, it performs the necessary transformations.
00:10:53.759
In this case, we replace three dots with named arguments, rest, and block arguments, relying on the built-in mechanism for Ruby 2.7 and later, thereby avoiding the keyword argument split issue.
00:11:08.000
The transpiling process uses the AST to rewrite the source code in place. Sometimes we may unparse a bit of the AST, as with pattern matching. Here's an example of transpiling.
00:11:38.080
Assuming you have this FizzBuzz implementation with pattern matching, it is concise, elegant, and readable. It highlights the advantages of using pattern matching.
00:11:53.680
However, if we decide to transpile it manually, the equivalent source code for Ruby 2.5 might look simpler, but the transpiled output generated by the transpiler can be more cumbersome.
00:12:08.080
The transpiled code tends to include more local variables and additional checks, primarily to ensure compatibility with the original code, as pattern matching is more complex than it may seem.
00:12:27.329
Nevertheless, transpiled code serves its purpose for machines and can be executed by the Ruby VM, while the original pattern matching implementation remains visible to those reading your code.
00:12:48.720
The power of transpiling lies in utilizing new syntax features more expressively, enabling older machines to handle code that appears more convoluted. Let's examine the intermediate steps: analyzing and optimizing.
00:13:06.080
You might notice that the pattern matching code output by the machine is extensive and includes numerous checks, as pattern matching is a substantial feature. The implementation of the pattern matching rewriter contains around a thousand lines of code.
00:13:27.600
Most of those lines, however, are not dedicated to logic but to optimizations. I was developing this rewriter by reverse-engineering the Ruby implementation using RubySpec and RubyTest.
00:13:47.299
My optimizations led to some surprising results: the transpiled code ended up being faster than the native implementation, which was unexpected for a transpiler.
00:14:05.680
I realized that transpiling is not just a tool for developers to simplify their processes; it's also a research tool that helps improve the Ruby VM. By experimenting with transpiler optimizations, we can significantly enhance future Ruby versions.
00:14:26.640
This is why I am still pursuing this work, even if the community isn’t fully ready to adopt the tool just yet. Now, let’s move on to the other how.
00:14:38.120
How can transpilers be integrated into interpreted languages? It's an interesting question, because, unlike front-end development, we don't have build tools—no packagers or build steps.
00:14:51.440
We simply run our code. How do we transpile? Depending on the use case, it could be done at release time or runtime, which is how Ruby Next operates. Both use cases leverage Ruby's two constants: load path and loaded features.
00:15:31.840
We manipulate these constants to make Ruby function the way we desire it to. This may include loading transpiled files instead of the originals for specific versions.
00:15:53.680
For example, when using RubyNext with gems, you can configure transpiled files through releases and package them into gem archives by executing a command called nextify.
00:16:12.080
In your code, you can add a simple snippet like setup gem load path, which adjusts the load path to resolve the absolute path of the feature you want to load with a require statement.
00:16:30.000
Ruby scans the load path sequentially and as soon as it finds a matching feature, it loads that code and marks it as loaded. This means you have control over when to load transpiled code.
00:16:56.320
Only users with older Ruby versions will have to load transpiled files. This functionality extends to Ruby implementations with compile phases, such as MRuby.
00:17:15.919
In that case, we transpile source files before the final compilation; this is our initial step. We also manipulate target files similarly to how we handle load paths.
00:17:35.440
Finally, let’s return to the core focus and the discussion about evolution. How can transpiling help improve Ruby?
00:17:50.399
Recent Ruby versions have added many experimental features, and we have begun introducing more as well. We need a way for users to engage and evaluate these features before declaring them stable.
00:18:08.560
This approach helps us gauge whether to retain or discard experimental features more efficiently than releasing them in major updates and waiting a year to address feedback.
00:18:35.680
For instance, consider the shorthand feature. This controversial feature has been proposed multiple times but has constantly faced rejection despite significant community interest.
00:19:05.400
How do we share our opinions if we cannot experiment with the features ourselves? Here's where the transpiler assists us: we can implement the feature in Ruby Next and invite community feedback.
00:19:33.440
This avenue enables Ruby's core team to make informed decisions based on real-life usage rather than speculation. By gathering comprehensive feedback, we help drive Ruby's evolution.
00:19:46.480
This collaborative approach empowers us to shape Ruby together. I invite you to try Ruby Next and share your thoughts on potential features we can implement.
00:20:07.520
Thank you for your attention today! I look forward to seeing how we can leverage transpiling to advance Ruby.
00:20:21.360
Thank you very much!