The whys and hows of transpiling Ruby

Ruby

Vladimir Dementyev

@palkan

#transpilation

#ruby

#abstract-syntax-tree-ast

#ruby-implementations

#code-optimization

#ruby-features

The whys and hows of transpiling Ruby

by Vladimir Dementyev

In the RubyKaigi 2020 Takeout edition talk, Vladimir Dementyev discusses the concept of transpiling in Ruby and introduces a transpiler project called Ruby Next. Transpiling allows developers to utilize modern Ruby syntax features on older Ruby versions, which addresses compatibility issues with both gem authors and alternative Ruby implementations.

Key Points Discussed:
- Definition of Transpiler: A transpiler is a source-to-source compiler that converts code from one version of a language to another, enabling the use of newer features in older environments.
- Current State of Ruby: The talk highlights the evolution of Ruby, particularly focusing on features introduced in Ruby 2.8 and upcoming changes in Ruby 3.x, such as pattern matching and rightward assignment.
- Need for Transpiling in Ruby: Many developers still use older Ruby versions and are unable to access newer syntax features due to compatibility constraints. A notable example is the lack of support for pattern matching in Ruby 2.5.
- Introducing Ruby Next: Ruby Next is a transpiler that allows developers to write modern Ruby code while maintaining support for older versions, enhancing usability and experimentation with new features.
- Transpilation Process: The process involves parsing the source code, generating an Abstract Syntax Tree (AST), and applying rewriters for preferred features like pattern matching. The talk illustrates how transpiling can help reframe existing code into a more complex yet functional format for older Ruby versions while retaining original programming paradigms.
- Performance and Optimization: The transpiler can yield faster execution times in some cases, demonstrating its potential as a tool not just for improved development but for influencing Ruby's future enhancements.
- Community Engagement through Experimentation: By allowing the community to experiment with experimental features using Ruby Next, real-world feedback can help decide which features should be retained or discarded before they reach stability.

In conclusion, transpiling in Ruby via Ruby Next is presented as a solution for developers facing compatibility issues while fostering community involvement and evolving Ruby effectively through feedback and experimentation.

00:00:01.760 Hey everyone, welcome to my RubyKaigi 2020 Takeout edition talk. My name is Vladimir, and today I'm going to answer some questions regarding transpiling Ruby.

00:00:09.679 Ruby is our favorite language, and we're going to talk about its present, past, and future. We're also going to discuss a specific tool called a transpiler. So, what is a transpiler? A transpiler is a program that translates source code from one language to another at the same level of structure.

00:00:27.119 It is usually called a source-to-source compiler because it's a specific case of compilers. The input and output source code is usually of the same language. One of the most popular transpilers is Babel from the JavaScript world, and it's due to Babel that the word 'transpiler' became popular. Another transpiler from the front-end world, like PostCSS, is a tool for transforming CSS, which allows you to use modern syntax in older browsers.

00:00:57.441 But how does all of this relate to Ruby? Why are we talking about such hackery as transpiling for a primarily server-side language? Why do we need a transpiler for Ruby? Let's take a look at the current Ruby feature set.

00:01:14.880 Currently, Ruby 3.x is in development, and its working version number is 2.8. I will use this version as the foundation for my talk, as it is going to be released in a few months. Ruby is evolving rapidly these days. The most recent release brought some new features, including new syntax, and the upcoming release will add even more new syntax features.

00:01:34.679 Let’s take a look at an example of a Ruby 2.8 program. It includes pattern matching. Pattern matching is fairly recent; it's almost a year old. You may know about it and perhaps even used it. However, many people aren’t using some of the newer features, such as the endless method definition, in their everyday lives.

00:01:50.800 We also see the rightward assignment, which is essentially just an assignment but in a different direction. This is also coming in the next version. Unfortunately, it's hard to use these features today because you need either an edge Ruby running on your system or you have to build it from source.

00:02:05.440 So, you definitely don't use these features in production, right? And maybe you would like to use them in the current version that you are using. How can we do that? Well, with the help of transpilers. We have a transpiler for Ruby, and that was one of the reasons I started working on this project: yes, I am working on Ruby Next.

00:02:23.520 I've been working on Ruby Next for about a year. The main reason for that is to make it possible to utilize modern syntax features with older Ruby versions. Today, I'll discuss why I decided to build it and how it works.

00:02:36.400 So, what is Ruby Next? In simplest terms, it allows us to demonstrate how it works before diving into the whys and hows. Assume you have an older Ruby version, say 2.5, and you want to run code with pattern matching in it. Without the magic of Ruby Next, it would raise an exception. But with Ruby Next, it simply works!

00:02:54.919 That’s what transpiling does for you: it enables you to write modern Ruby today, even if your environment is based on an older version. My name is Vladimir, and let me first introduce myself briefly.

00:03:06.560 You can find me on GitHub under the nickname Falcon, where you'll discover my open-source projects and information about my social networks and writings. I'm working for a company called Evil Martians, which consists of a team of senior developers, front-end and back-end, as well as designers and DevOps engineers, spread across the globe with bases in Japan, the USA, and Russia.

00:03:24.240 We help both large companies and small startups enhance their projects, optimize performance bottlenecks, and implement innovative ideas. Part of our work involves contributing to open-source tools, which we use in our daily lives. We strive to improve these tools and create new ones that benefit the community we love, including Ruby.

00:03:39.840 We write about everything we do in our blog, where most of our articles, including the popular ones, have been translated into Japanese, Chinese, and sometimes other languages. That's everything for the introduction of myself and my company. Now, let's switch to the main topic and discuss why we want to transpile Ruby.

00:03:57.679 I already mentioned the issue of backporting — the desire to use new features with older versions for various reasons. Let’s take a look at some data from RubyGems. As you can see, not a lot of people or machines are using edge Ruby or the latest version, even the previous ones. The majority of Ruby users, whether human or robot, are still running older versions and cannot use new features.

00:04:32.640 What does this mean for them? They are probably okay with using older syntax. However, personally, as a gem author, maintaining several dozen other gems means that I cannot just drop support for older Rubies. I have to support at least those versions that are officially supported. As of today, that’s Ruby 2.5 and above.

00:04:55.679 Unfortunately, I cannot use pattern matching with Ruby 2.5, and most gem authors cannot either. Thus, they should stick to the older version. Even if someone decides to create a new gem and supports only Ruby 2.7 and above, there is a high chance people will come asking not to drop support for older versions.

00:05:14.960 A case in point is with Hanami's API gem. This request reveals another side of the problem with integrating new features and syntax: compatibility issues with alternative Rubies like JRuby, TruffleRuby, and others.

00:05:33.920 These implementations are not always compatible with the latest Ruby features, and we have to wait for maintainers to add support for MRI features. Currently, JRuby supports Ruby 2.6, as does TruffleRuby. As for MRuby, it has limited support for 2.7 features but lacks pattern matching, for example.

00:06:01.440 Meanwhile, RubyMotion and Artichoke also do not support pattern matching. This means these alternative Ruby implementations will not be able to use new features without backporting. Therefore, transpiling can assist them in overcoming these limitations.

00:06:34.720 This isn’t the only reason I’m discussing today, but let’s save the last one for the end of the talk and switch to the next question: how to transpile Ruby?

00:06:50.840 Remember, I mentioned earlier that a transpiler is a source-to-source compiler. Generally, compilers work like this: first, we parse the source code into some intermediate representation, often an abstract syntax tree (AST). Then we perform some analysis and optimizations, before generating the final target code.

00:07:08.320 In the case of transpilers, we parse the source code, analyze and optimize it, and then generate new source code. This typically involves modifying the AST and generating a new source code from it.

00:07:32.640 Let’s talk about the first step in the transpiling process, which is generating an AST from the source code. In Ruby, we have three popular ways of doing this.

00:08:01.919 First, there are two AST APIs built into Ruby. They are a bit different, but they allow you to get some intermediate representation of the source code—not just a string—which can be modified, and you can regenerate the source code from that.

00:08:29.198 The last approach is using the standalone gem Parser. It differs in many ways from the first two approaches, and while we could discuss their disadvantages, let’s focus on the advantages of using Parser for generating the AST from Ruby source code.

00:08:48.080 Parser is written in pure Ruby, has limited dependencies, and can be used with any Ruby version. You can parse Ruby 2.7 source code while running on Ruby 2.5, for instance. No recent Ruby version is needed to parse the latest source code. Moreover, Parser is used in many syntax and static analysis libraries, such as RuboCop, which gives us confidence in its performance.

00:09:10.320 Lastly, and importantly, Parser includes rewriting support, letting you change the source code in place, without impacting everything else. One of the main issues with generically generating source code from the AST is that you can lose style information, such as spacing and layout. Parser's ability to rewrite allows us to retain these elements.

00:09:37.680 Now, let's take a look at the simplified source code for the transpile method. The whole idea of transpiling uses a concept called rewriters. A rewriter is a model or class that is responsible for a particular feature.

00:09:55.200 For instance, we have a rewriter for pattern matching, another for numbered parameters, etc. Each rewriter analyzes the AST and applies the necessary transformations to the original source code to generate new source code.

00:10:10.920 The more rewriters we have, the more transformations we can apply to the source code before it's finally transpiled. Here's an example of what a rewriter could look like. It defines callback functions for various node types within the AST.

00:10:35.840 For example, a forward arcs node is part of a method definition, while a send node represents a method call. The rewriter inspects the arguments of the method's code, and if any forwarded arguments are present, it performs the necessary transformations.

00:10:53.759 In this case, we replace three dots with named arguments, rest, and block arguments, relying on the built-in mechanism for Ruby 2.7 and later, thereby avoiding the keyword argument split issue.

00:11:08.000 The transpiling process uses the AST to rewrite the source code in place. Sometimes we may unparse a bit of the AST, as with pattern matching. Here's an example of transpiling.

00:11:38.080 Assuming you have this FizzBuzz implementation with pattern matching, it is concise, elegant, and readable. It highlights the advantages of using pattern matching.

00:11:53.680 However, if we decide to transpile it manually, the equivalent source code for Ruby 2.5 might look simpler, but the transpiled output generated by the transpiler can be more cumbersome.

00:12:08.080 The transpiled code tends to include more local variables and additional checks, primarily to ensure compatibility with the original code, as pattern matching is more complex than it may seem.

00:12:27.329 Nevertheless, transpiled code serves its purpose for machines and can be executed by the Ruby VM, while the original pattern matching implementation remains visible to those reading your code.

00:12:48.720 The power of transpiling lies in utilizing new syntax features more expressively, enabling older machines to handle code that appears more convoluted. Let's examine the intermediate steps: analyzing and optimizing.

00:13:06.080 You might notice that the pattern matching code output by the machine is extensive and includes numerous checks, as pattern matching is a substantial feature. The implementation of the pattern matching rewriter contains around a thousand lines of code.

00:13:27.600 Most of those lines, however, are not dedicated to logic but to optimizations. I was developing this rewriter by reverse-engineering the Ruby implementation using RubySpec and RubyTest.

00:13:47.299 My optimizations led to some surprising results: the transpiled code ended up being faster than the native implementation, which was unexpected for a transpiler.

00:14:05.680 I realized that transpiling is not just a tool for developers to simplify their processes; it's also a research tool that helps improve the Ruby VM. By experimenting with transpiler optimizations, we can significantly enhance future Ruby versions.

00:14:26.640 This is why I am still pursuing this work, even if the community isn’t fully ready to adopt the tool just yet. Now, let’s move on to the other how.

00:14:38.120 How can transpilers be integrated into interpreted languages? It's an interesting question, because, unlike front-end development, we don't have build tools—no packagers or build steps.

00:14:51.440 We simply run our code. How do we transpile? Depending on the use case, it could be done at release time or runtime, which is how Ruby Next operates. Both use cases leverage Ruby's two constants: load path and loaded features.

00:15:31.840 We manipulate these constants to make Ruby function the way we desire it to. This may include loading transpiled files instead of the originals for specific versions.

00:15:53.680 For example, when using RubyNext with gems, you can configure transpiled files through releases and package them into gem archives by executing a command called nextify.

00:16:12.080 In your code, you can add a simple snippet like setup gem load path, which adjusts the load path to resolve the absolute path of the feature you want to load with a require statement.

00:16:30.000 Ruby scans the load path sequentially and as soon as it finds a matching feature, it loads that code and marks it as loaded. This means you have control over when to load transpiled code.

00:16:56.320 Only users with older Ruby versions will have to load transpiled files. This functionality extends to Ruby implementations with compile phases, such as MRuby.

00:17:15.919 In that case, we transpile source files before the final compilation; this is our initial step. We also manipulate target files similarly to how we handle load paths.

00:17:35.440 Finally, let’s return to the core focus and the discussion about evolution. How can transpiling help improve Ruby?

00:17:50.399 Recent Ruby versions have added many experimental features, and we have begun introducing more as well. We need a way for users to engage and evaluate these features before declaring them stable.

00:18:08.560 This approach helps us gauge whether to retain or discard experimental features more efficiently than releasing them in major updates and waiting a year to address feedback.

00:18:35.680 For instance, consider the shorthand feature. This controversial feature has been proposed multiple times but has constantly faced rejection despite significant community interest.

00:19:05.400 How do we share our opinions if we cannot experiment with the features ourselves? Here's where the transpiler assists us: we can implement the feature in Ruby Next and invite community feedback.

00:19:33.440 This avenue enables Ruby's core team to make informed decisions based on real-life usage rather than speculation. By gathering comprehensive feedback, we help drive Ruby's evolution.

00:19:46.480 This collaborative approach empowers us to shape Ruby together. I invite you to try Ruby Next and share your thoughts on potential features we can implement.

00:20:07.520 Thank you for your attention today! I look forward to seeing how we can leverage transpiling to advance Ruby.

00:20:21.360 Thank you very much!

See Slides on speakerdeck.com

Vladimir Dementyev

@palkan

RubyKaigi 2020 Takeout