00:00:18.080
It's my pleasure to introduce the next speaker, Kevin Newton.
00:00:23.119
He'll be talking about the future of understanding Ruby code. Kevin is a Ruby committer and a staff engineer on the Ruby infrastructure team at Chawbay.
00:00:28.199
He’s based out of Boston, Massachusetts, and is passionate about music, accessibility, and open source software, of which he creates a lot. Notably, he received the Ruby Prize 2023 last week in Japan for his work on the new Prism Ruby parser.
00:00:57.280
So please, put your hands together for Kevin!
00:01:10.920
Before we start, I'm just on the phone with my son. I live in Massachusetts and there are wild turkeys in our yard, and he's sobbing. He's three years old, so can everyone say hi, Henry, on three?
00:01:16.200
He's going to lose his mind. I'm not FaceTiming him; this is a video. Ready? Three, two, one—you're the best, thank you!
00:01:35.320
Now, let's see if my clicker works. Okay, this talk is entitled 'The Future of Understanding Ruby Code.' Come on in!
00:01:44.759
The concepts we’re going to present today are somewhat technical. If you are a more experienced programmer, I hope this inspires you or gives you something to think about. If you are a less experienced programmer, I want you to let it wash over you.
00:01:59.119
I want you to treat this talk like a Wikipedia article. There will be links and tons of things for you to dive into later. Don't take notes; I’ll post the slides. Just try to enjoy it.
00:02:11.800
Now, we're going to do a little exercise. I want you all to put your hands up in a second if the code on the screen makes sense to you. Raise your hands up; great, put your hands down when it stops making sense. Alright, it's a method call.
00:02:35.440
So, what have we demonstrated? We have learned that there is syntax that is very weird in Ruby. We’ve also discovered that looking to different people about what this code will do is not sufficient because only two people in the room had their hands up.
00:02:54.319
Let's think about it: What does this code do? This isn’t an urgent question; rather, it’s important to ask: How would you find out what this code does? I want you to consider that for this talk; we will come back to it at the end.
00:03:30.120
Now, to discuss dependencies: My son Henry, he’s three years old, and every morning he says the same thing, 'Dad, let's go build towers.' So let’s build some towers. We start with your application. You build an application, which depends on external libraries.
00:03:49.360
Think of something like Rails. External libraries depend on standard libraries from Ruby. Standard libraries from Ruby execute YARV bytecode internally, which depends on the compiler that builds the bytecode, which depends on the parser, which, in turn, relies on the Ruby source code.
00:04:12.239
Okay, we've built our Tower. Now let’s do a very quick whirlwind tour of what this means. You start with your Ruby source. This is Ruby source that comes from Sidekiq. You can call this method in Sidekiq. The parser will parse this into a set of tokens; that’s the lexer’s job.
00:04:30.160
It then parses it into an Abstract Syntax Tree (AST). The compiler then takes this and walks through the tree to generate our instructions. Right now, it is in linked list form. This is what a compiler does—it builds the linked list. From here on, we’re going to build something called instruction sequences, which is the bytecode, also known as an Intermediate Representation (IR).
00:05:00.000
This can call standard library methods like puts. Then you can go and this instruction sequence is what you have built. This is an external library, and it all resides in your application—this is our dependency tower.
00:05:37.920
Now, let's fill this in for CRuby. CRuby is the current parser. It passes through compile.c, which builds YARV instruction sequences that fuel the standard library for CRuby, which powers the gems hosted on RubyGems and ultimately your applications.
00:06:12.239
Let’s talk about this for other Ruby implementations. JRuby uses another grammar file called RubyParser, which builds an IR using IRBuilder. Java, in turn, builds the JRuby IR, which builds JRuby's standard library, powering most gems on rubygems.org. Sometimes you will need to patch in Java for native extensions, which works on almost all applications. Now, let’s discuss Truffle Ruby.
00:06:46.240
Truffle Ruby came from the same origin but works slightly differently, using the Truffle interpreter. It builds an IR as well, though it does not function as bytecode. Truffle Ruby builds the Truffle Ruby standard library, which also works on most gems on rubygems.org and most applications.
00:07:05.520
Now let's talk about Natalie. Natalie has its own parser called the Natalie parser, which processes files, compiles them into Ruby instructions that are declared within the instructions directory, and builds the Natalie standard library that works for most gems on rubygems.org.
00:07:37.160
So, we have all of these different implementations; what happens when Ruby source changes? By 'changes,' I mean new syntax being introduced or some kind of breaking change. And let me be clear, there are indeed breaking changes. The pink will represent these different elements.
00:08:14.880
CRuby is the canonical place where these updates happen. Parts of Y get updated, CRuby gets updated, and everyone is happy; yet everyone else starts working on their parsers to accommodate these changes. Things get a bit more complicated as time moves on.
00:08:30.240
As Ruby source changes again, CRuby tends to quickly update; however, it’s not that simple. Natalie may get further into the instructions, and Truffle Ruby may make progress in its translator. JRuby will also attempt to continue building out the grammar file.
00:09:06.080
The challenge is when source code changes, it leads to a fractured ecosystem because everyone is working off different rules and assumptions. Every representation of Ruby can lead to different implicit assumptions. No one intends to cause this, but it’s tough to represent Ruby uniformly without executing the exact same code.
00:09:30.640
Because each tool has its underlying assumptions, contributors must relearn how to engage with these tools. This knowledge doesn’t translate well to contribute to CRuby—where the canonical runtime exists. If Ruby will continue to exist for 30 or even 100 more years, we need more people contributing to CRuby. Working solely on these tools will not directly contribute to that.
00:10:09.320
Maintaining a Ruby parser comes with a massive maintenance burden. Every time new syntax is introduced, every parser needs an update. A remarkable person named Tom Áno has maintained the JRuby parser for over 10 years, which is nothing short of heroic. He constantly looks for commit changes in parts of Y and translates that code from C to Java.
00:10:38.640
This maintenance process takes a lot of effort and time. Even minor changes to the grammar can have massive implications across the ecosystem. The situation has become problematic. Introducing new syntax to improve Ruby can inadvertently harm the ecosystem, leading to what I’m calling a fractured ecosystem.
00:11:06.200
Let’s discuss the existence of parsers. The canonical parser is parts of Y in CRuby, alongside Ripper, LI Ruby parser, and the M Ruby parser, with Sorbet using the Types Ruby parser historically. They are all maintained, though nearly everywhere.
00:11:37.320
Let’s talk about syntax trees. Parsers can generate different syntax trees. For example, the C parser generates two different syntax trees, one for the C API and one for the Ruby API. Ripper can also build two different syntax trees depending on the options you pass to it. The fact that so many exist creates further complications: it’s a fractured ecosystem.
00:12:00.600
So, what did I do? I created a new parser. I'm proud of this comic—it aligns with my discussion. If all of my pull requests are merged, this is what we would expect to see.
00:12:25.400
Let me clarify: the purpose of these pull requests to remove certain parsers is not to imply they are bad. They are indeed very good parsers and have served their purpose well. The real issue is not their writing, but the maintenance involved.
00:12:55.040
None of these amazing contributors should have to manage these parsers. Their focus should be on developing applications that improve Ruby’s runtime. For instance, JRuby is rolling out support for fibers, which is an incredible advancement. We aspire for this level of progress across the ecosystem.
00:13:22.240
The goal with Prism is to establish a single parser that everyone can use. Prism was built with five design goals in mind: compatibility with C Ruby must be one to one; maintainability for easy contributions and usage; maximum error tolerance to provide great error messages; portability across all Ruby runtimes, and finally, performance in memory and speed.
00:13:47.880
Now let's talk about the timeline for Prism's development. A little over two years ago, I started working on a prototype originally called RubyParser. Later it was renamed to Yarp and subsequently to Prism. We had our first meeting with the CRuby team to propose this concept, and they agreed.
00:14:14.280
The first commit to Yarp happened later that year, and in April last year, I welcomed my daughter into the world! After returning from parental leave, work continued on the parser. We see integration from different projects like Truffle Ruby and JRuby commit to using Prism as their main parser.
00:14:48.000
When I resumed work, I also focused on porting the syntax tree formatter to utilize this new parser. We renamed to Prism. Below are some additional details on how we built Prism.
00:15:14.000
Designing a new syntax tree and parser comes with numerous challenges. The first focus for us was to evaluate every parser previously mentioned and extract the best features from each. Comparing semantic nodes against syntactic nodes was crucial. Some nodes represent the syntax you've written. We aimed to represent the code you intend to run.
00:15:44.000
For example, when considering an assignment node, the left node contains the target while the right node is the value assigned. This distinction is critical, as writing to a constant versus a global variable carries different semantic meanings. We thus concentrated on attaining nodes that represent the semantics of the tree instead of merely the syntax.
00:16:25.760
Designing for Ruby implementations entails making compilation processes as straightforward as possible. We sought to eliminate the need to check child nodes. Our intent was to cover common functionalities for all runtimes simultaneously.
00:16:55.760
Designing for tooling requires using named fields instead of standard array returns, which can often be cumbersome. Having named fields allows you to directly access data without dealing with complex structures, making the corresponding changes within underlying implementations smoother.
00:17:23.840
We also focused on ensuring that we have enough documentation, templates as much code as possible, and built a robust test suite to establish a foundation for Ruby syntax. Currently, we are actively working on existing test suites to drive improvements.
00:17:54.080
Now let’s address the challenges we faced with encoding. There’s a magic encoding comment appearing at the top of Ruby files. There are 90 possible source encodings for CRuby. We currently support about 30. Our goal is to accommodate all of them to ensure total compatibility.
00:18:21.680
Next up are escape sequences—basic escape characters utilized within strings. This can also lead to confusion about how we escape and interpret various characters. We solved this by brute-force testing each combination of contexts, confirming successful handling of all necessary cases.
00:18:47.800
We also had to contend with regular expressions, which require hard checks against local variables in given scopes. This is significant, as local variables can often clash with method calls. Acknowledging local variables becomes increasingly complex through named capture groups.
00:19:00.920
We encountered difficulties with declarations treated as expressions. In this situation, whenever a new line is hit, we had to determine whether we had previously encountered a Heredoc declaration and handle it appropriately. This led to complexities in parsing that we needed to resolve.
00:19:37.680
Now let's discuss the current status of our progress. In the CRuby implementation, we've developed three APIs: the C API, which we are finalizing; the Ruby API used across various Ruby tools; and a serialization API enabling parsing and quick serialization—allowing FFI implementations with fewer native calls.
00:20:12.480
For instance, in JRuby, you call PM serialized parse, one of the functions embedded in our C library, with a buffer to receive the stream. In Truffle Ruby, the process is similar. JavaScript can also access the EST from Prism after it compiles to WebAssembly.
00:20:41.760
Looking ahead, we will focus primarily on improving error tolerance and enhancing messaging across all tools. There’s significant room for performance enhancements, especially given that the CRuby parser is historically less optimized.
00:21:06.760
Currently, CRuby utilizes a parser generated by tools known as Bison. However, recent events have revealed Ruby's context-sensitive nature. This means using a parser generator is a misstep since context-free languages aren't sufficient to address Ruby's complexities.
00:21:34.520
Because of this, we opted for a handwritten recursive descent parser tailored to Ruby’s intricacies. This allows us greater control and enables us to implement more elaborate optimizations than generated parsers can offer.
00:22:05.560
One major enhancement planned for early next year is reallocating all nodes to be adjacent to one another. We also intend to explore automatic vectorization for our lexer and integrate state management more seamlessly within the parser.
00:22:32.760
As Prism is poised to roll out with Ruby 3.3, we will need to focus on ensuring compatibility across different Ruby versions while incorporating enhancements.
00:23:04.400
If you know of any gems using Ripper, please inform me or advise them to stop. Currently, Ripper is labeled as an early alpha version, and we should encourage moving away from it.
00:23:34.240
We now have our Ruby API supporting named types for each node in the syntax tree. We aim to expand this API to provide deeper functionality. For example, we can develop methods that help navigate or compile subtrees effectively.
00:24:00.080
It's imperative that we also foster a contributor community around this single tool. Many believe that only those with advanced degrees or extensive experience can contribute to complex topics like parsers or compilers, which isn’t true.
00:24:27.320
The truth is, we are experiencing a decline in the community. Instead of adding barriers to contribution, we must lower them. We should create an inviting atmosphere for those who want to contribute.
00:24:54.920
I’ve spent the past six years working on parsing Ruby, and to be frank, it's not something I enjoy. My true desire is to engage with what comes next—building Ruby tools can be a lot of fun and is highly rewarding.
00:25:24.040
With tools like Elm, TypeScript, or Rust, we can aspire to accomplish similar capabilities for Ruby, such as automatic formatting or completion based on types. Imagine tools that can highlight relevant variables based on the current selection.
00:25:54.600
Returning to our initial inquiry: what does this code do? Our focus should shift from individual lines of code to questions about how to effectively determine what any given piece of code does.
00:26:22.960
It's crucial to build an ecosystem of runtimes that can represent Ruby in consistent ways and maximize collaboration across the board. Tools should share agreement on Ruby representations, allowing us to explore code in greater detail and ask pertinent questions.
00:26:50.520
Lowering the barrier to entry for contributors around this ecosystem is essential. We need a community that collaborates to empower tool development and deepen everyone's understanding of Ruby.
00:27:22.800
This marks the future of understanding Ruby—not just for the code but for the community as a whole. Let us build this future together.
00:27:43.760
I rehearsed this when I hadn’t had two cups of coffee and went way too fast, so we have plenty of time for questions. Who's got one?
00:27:48.960
That's a great question. The question was how we extensively tested escape sequences, especially with Unicode. The answer is we utilized eval. I have a brute-force test where we parse and evaluate code in Ruby across various encodings.
00:28:03.280
Incompatibilities are primarily tied to parsing more syntax than Ruby permits to enable better error handling. We anticipate applying fixes in Ruby 3.4 if required but believe that valid Ruby code will remain functionally consistent.
00:28:46.800
If you’re interested in contributing but unsure how to begin, we provide detailed documentation. There are smaller issues open and I would encourage everyone to engage. Additionally, if bugs arise due to a lack of clarity, this indicates we need more documentation.
00:29:09.040
You can dive into background reading available within the documentation, particularly sections discussing parsing techniques that structure recursive descent parsers.
00:29:29.600
When it comes to handling multiple versions of Ruby, the practical solution currently is to fork and manage separately since combining two versions within the same process isn’t ideal.
00:29:58.800
Reflecting on my journey in developing parsers, I began working on a project while employed at a startup and later made attempts to incorporate different parsers for Ruby formatting. Through that journey, I realized the intricacies of maintaining a parser and dove deeper into the community.
00:30:28.960
Amid disappointments during pitches to higher-ups, I faced challenges whereby the parsers could cause immense delays in project developments. My knowledge of parsing gained traction upon joining the team at Shopify after actively engaging with several related projects, leading to the successful creation of Prism.
00:31:00.400
Why was the name changed? Initially referred to as Ruby Das Parser, it transitioned to Yarp until feedback prompted a final change to Prism, succinctly capturing its essence.
00:31:30.720
Now regarding round-tripping back to the source code, I have been intermittently revamping SyntaxTree to facilitate a return to source after evaluation. After Ruby 3.3 is released, focus will shift toward finalizing this functionality.