00:00:07.279
A couple of months ago, I was looking at a file in the Ruby codebase called Ruby compile Doc. As the name implies, this file contains much of the compiler that's built into Ruby. Faraz talked a little about the compiler yesterday, and today we'll go into it a bit more, discussing its specifics and why I was looking at it.
00:00:18.920
I wanted to point out something that stood out to me. I was examining a method which we'll dive into a little more later. There was a parameter in here that I wasn't quite sure about at first, called 'popped'. I found it sprinkled throughout the method, which is quite extensive.
00:00:32.439
It was often referenced in places like 'if not popped' or 'if popped', and at first, it wasn't obvious to me what was happening. However, once I read through the code and understood it better, I found it fascinating. So, I decided to call this talk 'Popping into Ruby.'
00:00:39.559
Today, we're going to pop in and learn a bit more about what's happening here, particularly focusing on what this 'popped' integer was and why it was so interesting to me.
00:00:51.800
As Matt mentioned, I am also the co-founder of WB.RB, which is a global Ruby community for women and non-binary individuals, with over 1,000 members. If you identify as a woman or a non-binary person, please come join us. And if you have colleagues who fit this description, please send them our way. We have a thriving Slack space where many engaging conversations are happening.
00:01:05.158
Now, we can think of our Ruby program as cars running smoothly. I’ve driven a car plenty of times, but I've never really looked under the hood. If you asked me what was there, I couldn't tell you much. Today, my hope is to pop the hood of Ruby a bit. Not because there's something wrong, but because there's a lot of interesting activity happening under the hood.
00:01:24.280
Some of what we'll talk about today is how your Ruby code or your Ruby files undergo transformation under the hood; they eventually become bytecode. There’s a parse tree involved in this process. So, we're going to start with the first step, which is getting from Ruby code to a tree structure.
00:01:31.879
This process occurs through a parser, which is what understands your Ruby code. We'll delve into the details momentarily about how the parser works and what it does to produce this tree representation of any Ruby file.
00:01:38.760
Every Ruby file eventually becomes some version of this tree when you run it. To begin, let's look at a small sample program and see, at a high level, what the tree structure looks like.
00:01:50.440
Here, we have a very simple Ruby program. It contains a class called 'Conference' with one method that simply returns the string 'Ruby comp.' Now, to view the parse tree, you can dump the parse tree for any Ruby code similarly to how Faraz described dumping instruction sequences. If you pass the 'parse tree' flag to a Ruby file, you’ll get back the tree structure.
00:02:02.480
As you can see, the parse tree starts with a scope node. Scopes can also be used for blocks or other specific scopes, but code always begins with a scope node. Importantly, all nodes have locations, which is quite useful for tools that need to use the tree.
00:02:12.400
Later, we'll cover what other tools might utilize a parse tree that aren’t directly executing your code. For now, let’s focus more on the body of this scope node and dissect what it contains.
00:02:33.600
Immediately, we see it contains a class node, which matches our expectations given our program. To delve deeper into our tree structure, we observe that the entire program is encapsulated within one class node, directly under the scope node.
00:02:50.440
Within the class node, there are a few children. The first one is the class path, identified by 'C path', which is called a colon node. This isn’t necessarily the most descriptive node, but it provides an ID for the class.
00:03:03.360
The class path itself has a method ID of 'Conference', which is the name of our node. Additionally, the class node has a super node, which in this case is null as there is no superclass for our class.
00:03:17.520
It also contains a body node, which is wrapped in a block node. The crux within that body, in our specific example, is a definition node representing our method definition.
00:03:32.240
The name of our method is represented by the method ID 'name', which aligns with our expectation. As we explore further, we might expect some strings to appear in the method's body.
00:03:43.240
However, we discover the body node remains empty, indicating there are no arguments, represented by an empty node. Importantly, the abstract syntax tree reflects all our code because there are multiple consumers of this tree.
00:03:56.880
It's not just for execution; other tools like Rubocop or linters rely on this parse tree too, which is why the tree can't skip any optimizations.
00:04:12.960
If the tree had optimizations, tools that require the full picture, like linters, would malfunction. Recently, you might have heard about a new parser in Ruby news, previously called YARP but now referred to as Prism.
00:04:29.040
There have been a lot of talks about it, and I've personally been involved in the project. Kevin Newton conceived the idea and has been leading its development. You might have seen discussions about it in Ruby Weekly or Ruby's bug tracker.
00:04:40.680
So, what is this new parser, and why do we need it? The important difference is that Prism does not generate the same parse tree as its predecessor; instead, it creates a slightly different tree. The structures or objects for the nodes in Prism differ from those in the old parser.
00:04:55.760
For example, the 'args' node, which isn't essential here, just takes up space and consumes memory. In Prism, if there are no arguments, there isn't even an args node. We aim to make Prism more understandable and maintainable.
00:05:15.080
One further difference you’d see if you ran it through Prism is that instead of a colon node, you would get a constant read node. The node names remain similar, and their contents exhibit similarity in examples.
00:05:27.360
Kevin has given a fantastic talk on this, which I would highly recommend watching. During his talk at Ruby Kaji, he outlined three motivations for developing Prism.
00:05:37.840
I won't delve into all the specifics, but I want to highlight the importance of error tolerance, which is relevant to us in our daily work. Error tolerance means that rather than receiving feedback only for the first syntax error, we can receive feedback on multiple errors.
00:05:50.920
Usually, when writing Ruby code, if there are syntax errors, we only get notified of the first one, and there’s no recovery. For instance, if we have this class with several syntax issues displayed, when we run the syntax check in Ruby, we're shown a limited number of errors.
00:06:12.720
The feedback can be alienating for novices. The first error might indicate an unexpected local variable or method, which isn't clear unless you're familiar with Ruby. The second message about a missing 'end' statement may cause confusion.
00:06:31.440
With Prism, however, you’d see different error messages. Running this same code would yield a pointable surprise, as Prism specifies that there’s an 'unexpected parameter order' and highlights the exact problem area.
00:06:49.360
This clearer feedback will help newcomers better understand what's going awry in their code. Importantly, once Prism types through the errors, it expects an end to close the class statement.
00:07:02.040
We realize the old parser limits our ability to learn through feedback, but Prism addresses this by providing actionable insights.
00:07:15.040
We are actively working on Prism's development, hoping it will come as a flag in Ruby 3.3. This Ruby version is expected around Christmas, and we believe it will yield significant improvements.
00:07:29.280
We have established that our code goes through a parser, creates a tree, and thereafter becomes bytecode representation. Faraz previewed this content beautifully yesterday.
00:07:41.760
The compiler is responsible for turning the code into runnable bytecode. Because Prism generates a different tree, it requires a new compiler tailored to its structure.
00:07:51.760
My current focus is understanding the existing compiler so we can replicate much of its functionality for our new abstract syntax tree.
00:08:06.480
We’re not creating entirely new bytecode; rather, we are ensuring that the final results align, providing optimization as deemed necessary.
00:08:19.440
In fact, we’ve already begun merging this work and are continuing to implement the compiler functionality for Prism into Ruby.
00:08:37.920
Now, let's explore what this compiler does. A key point to understand, especially regarding the concept of 'popped', is its purpose: simplifying optimization.
00:08:51.840
To help illustrate this method, which I found fascinating, we utilize the 'IC compile each' process. It traverses all nodes within a tree using a switch statement.
00:09:08.080
For each node type it recognizes, it provides instructions on how to compile that segment of the code. For instance, a class node will be handled differently than others.
00:09:22.960
If we dump the instructions for conference.rb, we’ll see a series of commands which may initially appear nonsensical, but I hope they will make more sense shortly.
00:09:35.760
These commands are functional, like a 'put nail', indicating the assignment of a value from the string given in the method.
00:09:47.760
If we deliberate over the structured instructions, we can correlate them to various components of our original tree.
00:10:03.080
This identification allows us to match specific instructions to evident parts of our initial code. For instance, the node indicating 'define class' corresponds with our original class.
00:10:17.760
As we proceed down the instruction list, we can see how these align with our method `name`, which expectantly returns a string.
00:10:29.120
If we want to focus solely on the string, we can set up a basic Ruby file that just prints 'Ruby comp'. When parsed, it yields a simple structure containing a scope node and a string.
00:10:41.600
When compiled, it generates bytecode reflecting only what’s necessary without adding more instructions than needed.
00:10:53.800
In this case, the parser does not have any optimizations; however, the compiler allows for such opportunities, demonstrating why Prism can utilize different bytecode.
00:11:08.080
With the compiler, various optimizations and instructions can be adjusted, which will be super crucial for efficiency.
00:11:26.880
Let’s explore the 'popped' optimization in our project. I propose to expand our Ruby code resolution slightly by appending the integer 2023.
00:11:43.520
Upon running this code through the parser, we see we receive the full representation of nodes that recognize both our string and the literal integer.
00:11:57.520
When it reaches the compiler, though, you'll notice 'put string' is missing, as it doesn't need to perform anything.
00:12:11.720
This unavailable 'put' string doesn't serve any purpose and can therefore be omitted.
00:12:22.960
In essence, this popped optimization means the compiler is intelligently determining what can be excluded from the overall execution process.
00:12:36.080
When it compiles code, if something isn't critical to the methods or processes, the compiler trims those parts, rendering the process more effective.
00:12:53.680
Continuing our exploration, we find various options where the popped integer is integral, such as in scenarios where it's necessary, such as when returning values or in method calls.
00:13:07.920
Each case elaborates on scenarios where popped values maintain their relevance, while discarding others helps streamline performance.
00:13:22.760
This includes scenarios where we're ignoring unassigned values or unnecessary letters, reducing workload for the program.
00:13:38.960
We can further capture computational trends embracing literal expressions, as long as they’re not being retained elsewhere.
00:13:54.960
As we explore further, we envision an instance where we would potentially add complexity rather than brevity. Underlying all this is a discussion surrounding optimization versus necessity.
00:14:06.640
When simply executing may miss critical information, such as when a method call needs each and every parameter addressed.
00:14:23.760
In swift examples, regardless of circumstances, anything equating to an assigned value or method call demands attention, remaining untouchable.
00:14:39.760
On the contrary, values residing in a string, symbol, or array can essentially arise as dispensable references, surfacing potential for eliminated instructions.
00:14:55.680
Upon thorough examination, we derive understanding of when collectors become viable; iterating through value parameters ensures we need to observe their presence.
00:15:09.600
More actually, we encourage intuitive inquiries: Will unintended consequences negate advantageous outcomes? This invokes several striking examples.
00:15:21.440
Referrals to higher-order programming models pinpoint possible redundancies that yield remarkable insights for canny developers.
00:15:36.920
Such reflections circle our understanding of job roles and whether operational rhythm allowing for potential corrections may arise.
00:15:52.480
This examination nurtures active discussions on what pertains to optimizing mechanics while yearning for deeper comprehension.
00:16:06.800
In this cycle, we call upon young developers to not overlook that the dynamics surrounding business logic continually shift and evolve.
00:16:25.720
However, let’s not underestimate the importance of open-source gems or the Ruby structure itself; they lend direct influence on functional business models.
00:16:39.840
Research indicates that innovative modifications push the metrics beyond what was previously prescribed; optimizing tools comes to aid us in navigating further.
00:16:56.640
With recent builds, exploring these sources can yield tangible benefits and unlock substantial improvements, pushing projects to heights we previously restrained ourselves from.
00:17:13.680
Realistically, at both team and individual levels, we need to perpetuate growth in investigating what functions as optimal and effective.
00:17:30.960
In closing, I invite you all to embark on this journey. Though challenging, the insights derived from the depths of our understanding can yield benefits for all.
00:17:47.600
If you wish to contribute to Prism, the open-source initiative needs support leading to key benefits within our framework. Getting involved qualifies as a perfect initiation against extra complexities.
00:18:00.840
Last but not least, as you look upon these discussions you’ve just experienced, reconsider how many more illuminating details can arise through engaging with radical, thorough dialogue.
00:18:15.920
Thank you again so very much for your attention today. I deeply appreciate this platform and your time.
00:18:30.960
He.