00:00:09.880
Hello, hello! So nice to see everyone here today.
00:00:19.800
Now, I will grant that this is the only illustration I will get away with in this talk because we definitely want to be a little bit more technical for this one.
00:00:26.679
So, to start out, who am I? Well, my name is Brandon, I love Ruby enough to come to Japan and speak to all of you lovely people about it and share some of that love here with you today.
00:00:37.239
That said, should we go ahead and get started? On the subject of Ruby, briefly, what are they, and where are we today at a high level?
00:00:41.440
When you see this text, what do you see? Do you see words that make sentences, characters, or strange symbols? For anyone outside of Ruby, it’s very likely they just see a bunch of text; they don’t see anything of value. But to us, we see something different. We see that someone is monkey patching a method and trying to sneak in a method that might have been rejected. We see a method that takes a block and uses map and tally to count objects based on the result of that block, and we can see the output below it with a comment. That’s because, as poor as we might be at it, we are all Ruby interpreters. We see Ruby as code, symbols, tokens, expressions, and more. It’s not just text to us; it has distinct meaning. To me, this symbolizes meaning in a way that we can programmatically interact with it and do wondrous things with it.
00:01:39.880
Now, certainly, I would not trust a Brandon Ruby interpreter, but smarter people than I have created ways to translate these blobs of text that we call Ruby further into ASTs, which we can then work with. These are called parsers. So, how does one go from Ruby code to an AST? There are a few parts that get us there, but before I get too involved in that, it’s worth mentioning that there are some very intelligent people who have spoken at length about this.
00:02:25.439
At the bottom of these pages, you can find QR codes to further links and resources. I have also linked this information in the Discord channel. Hats off to those who you might know as the person behind Rubocop, who has done an incredible job summarizing many of these topics. My presentation is going to be far more high level and will synthesize many of these posts. I’ll cover the three major ones he discussed.
00:02:42.760
We start with Ripper, which is built into Ruby core itself. Back in 2019 and even somewhat recently, it was considered the de facto Ruby parser and is actively maintained by the core team. However, the output of Ripper can be hard to interact with. It consists of many arrays and nested structures, and I’m still not sure what all those nils are for. Programmatically, it is quite difficult to work with, but it did provide our first pathway to interact with Ruby code. Tools like Rubocop and much of the original tooling were built on top of it.
00:03:12.480
Being integrated into the Ruby core, it is fast and accurate, resulting in no drift when patch commits are made. The downside is that it can be challenging to approach, with minimal documentation, making contributions to it difficult. Many pieces of essential metadata are inline in that code, which makes it not very user-friendly for everyday Ruby users. This is why when the Whitequark parser was introduced, it provided a very different approach and a novel implementation, making many user tools more accessible.
00:03:46.000
It drew inspiration from Lisp-like languages, formatting it in terms of s-expressions with rich metadata in a syntax that is far more approachable for the everyday Rubyist. In fact, Rubocop gained popularity during this period because it provided a much more accessible syntax that allowed not just the core maintainers of Rubocop, who work day in and day out with it, but also people like you and me, who don’t spend that much time on it, to do something with it.
00:04:14.759
The one core feature that truly changed the foundation of Ruby tooling and LSPs was Tree Rewriter. Tree Rewriter allows us to rewrite segments of an AST by searching for them and changing them, which opens the door to many manipulations. It is also fascinating because if you happen to make multiple changes and they change the same code, it can intelligently handle changes in different orders.
00:04:46.160
Tree Rewriter provided syntax and abilities to deal with various scenarios where conflicts may arise between rules. This allows tools like Rubocop, which rely on dozens—if not hundreds—of rules in larger codebases, to operate more accurately. If there are conflicts in Rubocop rules, you will receive warnings, which is incredibly useful and forms a lot of the basis of this talk.
00:05:06.919
Making a tool that is easier to understand and work with has opened doors for everyday Ruby users. It was well-documented, with all nodes, syntax, and features clearly laid out in either the code or additional documentation wikis. Tree Rewriter’s compatibility across all Ruby implementations is also a significant advantage, a feature that wasn’t available with Ripper, as it can work with JRuby and other implementations.
00:05:36.200
The concept of Tree Rewriter was revolutionary at its introduction because of its rich metadata capabilities. Additionally, it was one of the first to support pattern matching for structural decomposition, allowing us to represent code in a way that we can identify matching shapes. This is a crucial aspect that we will touch upon later. However, there are some downsides. Ripper is extremely fast because it is written in C; however, Whitequark is not quite as fast. Fortunately, for tooling, speed is not the only concern—what's critical is the ability to represent and interact with the code.
00:06:10.840
There are some issues with maintainership and documentation that were addressed recently, but there remains slight patch drift with MRI, an issue mentioned in various Prism talks. This brings us to the next parser I’ll mention: Prism, the one parser to unify them all.
00:06:30.360
Prism takes a different approach: Ripper was literal and verbose, while Whitequark was minimal. Prism aims to combine the two while providing interfaces on top of its various parser formats. This unification is why Rubocop has received speed enhancements.
00:06:57.560
However, this can be a bit verbose. While it’s immensely useful for Ruby parsers, it may not be as beneficial for tooling. The advantages of Prism include the unification of various parsers into a standard format based on the well-known principle that if you have ten standards, and you try to unify them all, you end up with eleven standards. Yet, I think Prism has succeeded in this unification and has accomplished incredible work. It has comprehensive contextual information and relies heavily on keyword arguments and the types of nodes we are utilizing.
00:07:27.440
Prism is approximately five times faster than Whitequark. When you run Rubocop on a large Ruby monolith comprising several million lines of code, it may take three to five minutes to run that and perform auto-corrections. Imagine if those times were significantly reduced, potentially to just a few seconds, creating a remarkable improvement in CI build times.
00:08:03.640
While Prism provides these efficiencies, it also has compatibility layers with Whitequark, allowing developers to switch over with little effort, aside from some rare niche cases that are currently being amended. One of the downsides is that due to its comprehensive contextual information, developers must factor this in when writing tools, which can be more than what’s often necessary.
00:08:28.920
The complex nesting and the repetitive occurrence of keywords can make it challenging to navigate. Nevertheless, I believe that Kevin and his team have done an incredible job and will continue to do so. I look forward to seeing its developments. The key takeaway is that when we’re dealing with tooling, the ease of use is critical. It doesn’t matter if you build the most powerful tool in the world if no one knows how to use it.
00:09:02.960
What we care about is making it more accessible to everyday developers who are unlikely to conduct PhD-level research into this stuff. The transition to Prism was undertaken to provide full context in light of Ruby’s complexity, which makes sense. However, more often than not, we prioritize practicality over precision. We seek to get close enough—to make decisions, to indicate that particular pieces of code need changing, and to automate refactoring. When conducting code searches, we may not care about the precise format of the code but are primarily concerned about finding sufficiently similar matches.
00:09:45.720
I hope to see tools like Prism evolve further to strike a balance in the future, but for now, the compatibility layers provide a lot of flexibility. With that in mind, where does pattern matching come into the equation? We’ve spent much of this talk addressing parsers, but not much on pattern matching. For those unfamiliar with pattern matching, I’ll provide a brief introduction.
00:10:20.120
Essentially, when working with pattern matching, we look at a code example that illustrates array-like pattern matching such as pattern matching on lines. I often refer to them as single branches because I frequently make use of line breaks without hesitation. Some might feel apprehensive about this, but I will not be swayed. The approach captures many of the core syntax structures, and I'll provide a quick overview of each.
00:10:47.360
At the bottom right, you will find a QR code linking to the main repository for this code and all the relevant tools associated with it. Let’s examine this step-by-step. First, we have a simple array comprised of a couple of numbers and a string. For those proficient in Ruby, this won’t be too foreign. But the difference arises with the introductory keyword in the pattern match.
00:11:18.920
Everything following the keyword in is the pattern being assessed. The values being compared employ a triple equals operator, akin to that of a case statement, but there are additional syntactical elements we will explore. The initial comparison involves matching the value 1 to an underscore, signifying that we don’t care about that value. Alternatively, we could name it something like underscore first number, but here, it’s merely an unused variable we choose to ignore while maintaining the array structure.
00:11:50.240
Next, we match the value 2.0 against the numeric hash rocket n, which consists of two components. First, we compare numeric against 2.0. If this holds true, it assigns the value to n using rightward assignment, alongside that hash rocket syntax, which is the other part of pattern matching. We can delve into more details regarding the differences between in and right-hand assignment, but that would require a more extensive discussion.
00:12:38.640
Below, you can see that n equals 2.0, capturing our extracted value. Additionally, if we only want to capture that value without any checks against it, we could simply reference it as in. Moving forward, we match a string against a regular expression. When I mentioned that anything utilizing triple equals would work, I meant it, regardless if this pertains to regular expressions, class names, or IP addresses, which, by the way, even accommodates subnetting. This capability demonstrates the flexibility offered by this syntax.
00:13:35.440
Next, we explore pin values. If n in the previous area was assigned the value 2.0 and is stated again as n, it would be overridden. Instead, when denoted as a pin, we instruct that n should retain its previous value. This could reference a variable assigned during the pattern match or could relate to something assigned externally, allowing us to ensure that we expect this value to be identical.
00:14:10.720
Lastly, we have the star operator that captures everything following that point. If we position it at the start, it captures all preceding values. If we place it on both sides, it will search for that expression anywhere within the array, generating our find pattern. This indicates that we do not concern ourselves with the remaining values, focusing solely on a specific segment of it, whether it’s at the start, at the end, or within.
00:15:09.520
I appreciate that this was a very rapid introduction, and I have provided more detailed explanations along with articles I’ve authored on this topic, which may be beneficial. If you cannot access these QR codes, be sure to check the Discord channel for links. When working with pattern matching, I utilize additional tools to facilitate the process.
00:15:49.480
The first set of helper methods I employ enables me to reference the Ruby code at the current version I wish to match against, while the second is a deep deconstruct method, providing the array representation I will be employing. You might wonder why the array representation matters when the original structures are structurally identical; however, this utility proves useful for the sake of this discussion and I frequently employ it when debugging or formulizing Rubocop rules.
00:16:30.560
For instance, we could consider the expression 1 + 2 and its array representation, which is relatively straightforward. But what if we encounter something more complex? For instance, I may perceive that the block for this code could be shortened, but my laziness inhibits me from doing that manually; I want Ruby to handle it. Along these lines, there’s a possibility we could use Ruby to make this adjustment. Notice how the right-hand side of the array representation resembles what you might potentially be pattern matching. Could we simply implement a copy-paste action to the right-hand side of a pattern match? Let's give that a try!
00:17:32.840
And surprisingly, it works seamlessly! The deep deconstruct method simplifies this, alleviating the need for explicit planning. All I need to do is execute deep deconstruct, then copy and paste, and voilà, we have pattern matching! However, it is important to recognize that this specificity doesn’t match all potential arrays; it only matches that one particular shape. Nonetheless, we want to represent the general idea of the shape. Hence, how do we articulate that general idea? We're about to delve through that.
00:18:34.000
First, we no longer focus on the array or even whether it’s an array at all. We prioritize identifying the receiver of the block, which we can capture. The captured value at the bottom displays source code associating with it, allowing us to retrieve both the receiver’s value and its original source. We then project this from what we have where we simply care about what is calling—what specifics are being invoked.
00:19:22.640
We can now label the supplied arguments for the block with the argument name, establishing that the provided argument aligns with what’s called. The invocation of the variable v is reaffirmed there, ensuring that the variable represents the same context.
00:19:39.680
Next, we ponder whether the method invoked on it matters. Not necessarily; we simply need to ensure that some method is invoked on it. Therefore, we extract that as a called method, resulting in a fairly generalized match that effectively represents the structure.
00:20:30.840
We can even wrap this structure inside a function, which will enable a single method to determine whether or not something requires shorthand conversion. Returning to the captured variables, we can confirm this with several examples. In our first case, it can be expressed in shorthand since we have identical naming conventions.
00:21:25.520
In the second instance, it also aligns with shorthand, utilizing reject but featuring variable names differing from the previous one. That still holds true. In our next case, we run into an instance where we are dealing with y + 1, which translates as a method with an argument—this does not align with our shape. Likewise, having multiple parameters, such as x and y, also doesn’t conform to our requirements, as we are delineating a shape that indicates a single argument and one caller.
00:22:09.040
Now, remember that earlier discussion around Tree Rewriter? It appears we might have an opportunity to leverage that to enable Ruby to perform these actions for us. We could establish a new rule to examine block nodes, and if they match our pattern, we could substitute that node with our new shorthand version derived from the values we’ve extracted. For those who’ve worked with RuboCop, you will likely recognize this syntax.
00:22:48.600
Refactor, one of my creations, simplifies these processes due to the sheer amount of data that cannot fit onto slides. The primary focus here is the refactoring and upgrades. I aim to perform this adjustment precisely once; I don't wish to continually raise alerts about it.
00:23:39.480
The changes would occur when we engage the Replace function, which alters the node with our newly derived shorthand version and leverages the extracted elements from that pattern mapping, as illustrated in the provided examples.
00:24:23.640
Now that you've observed the capabilities of ASTs and pattern matching in manipulating and refactoring Ruby code, let’s glance at what the future may hold. I’ve hinted at this before, with pattern matching being precise. It raises the question—do we truly care about accuracy or merely about the shape itself? Picture a code repository containing millions of lines of code—five million, for instance—possibly utilized by organizations like GitHub, Shopify, or Gusto.
00:25:13.680
We don’t really care whether a certain argument being queried correlates to Ruby or Python. All that matters is the shape of the query. We might be able to depict that shape adequately. Imagine being able to delineate a structure in which we focus on the shape instead of the specifics, and when we define that shape, we could rewrite rules proficiently—transforming all positional arguments into a literal underscore to capture those elements.
00:26:10.040
This leads to insights such as knowing how many instances of calls exist. It may seem trivial at first, but I’ve found immense value in counting instances that help inform whether I need to refactor my API or amend methods in response to consumer needs.
00:27:09.560
I foresee potential for fuzzier searches that identify code closely related across different axes, although specifics can be complex and intricate. If we establish an index for our public APIs, we can automate suggestions for developers, ensuring they use the right functions effectively.
00:27:37.080
For example, if a public API function is noted as person.group.favorite_language.count and developers write similarly structured code, an intelligent indexing system could notify them if that API already exists. Language servers today present us fascinating opportunities to realize such functionalities.
00:28:24.920
Historically, tools such as Rubular have bridged gaps in regular expressions, showcasing vital captures and demonstrating matching behaviors. Creatively combining those elements with local captures prompts exciting prospects for the development community.
00:29:05.520
Mark Andre, a core committer of Rubocop, developed Rubocop's node pattern, an early incarnation of pattern matching that provided developers with a string-like representation of how to identify matches. They transitioned many of these functions to gains in flexibility, as string-like manipulations frequently produce unpredictable results.
00:30:13.600
The critical realization here is that closeness can be more important than exactness. While writing functions, knowing the general structure suffices. If we can bring forward tools to improve accessibility surrounding these topics, we can attain immense advantages, particularly across enhancements seen through AI and language models.
00:31:12.480
For instance, employing ChatGPT, one might ask it to identify Ruby pattern matches corresponding to classes incorporating specific methods. It could generate relevant code and suggest tools to facilitate the process, bringing newfound potential for automated progress in accessibility and development.
00:31:48.120
As we draw these thoughts together, I maintain strong faith in ASTs and pattern matching to pioneer more accessible tooling. We recognize that future advancements are already integrated into the present as brilliant creators, including some attending this conference, invest time developing these specific tools.
00:32:30.720
As I crafted these concepts, I discovered others were making substantial strides in similar areas, which astounds me. It proves that I can take advantage of existing solutions, a welcome sentiment indeed. With that, where do we go from here? Who knows? I am enthusiastic to see where this journey leads.
00:32:45.440
If you're interested in my work, feel free to connect with me on various social networks. While I am no longer active on Twitter (X), you can find me elsewhere. I’m grateful for the Ruby core team, the Rubocop team, Mark Andre, Whitequark, Kevin Newton, and many others I could not possibly name here. Understanding that this venture is a collaborative effort underscores the significance of community in the development of Ruby tooling and the direction it takes.
00:33:18.440
That’s all I have for you today, so thank you for your time!
