00:00:00.000
Ready for takeoff.
00:00:18.600
Hello everybody, welcome to "Improving the Development Experience with Language Servers."
00:00:24.960
I'm Vinnie, and I'm part of the Ruby Developer Experience team at Shopify. We do a lot of work with developer tools.
00:00:31.679
All of our projects are connected to the editor, so we're involved with gradual typing using Sorbet and Tapioca.
00:00:40.800
We recently started contributing to Ruby Debug, which is a debugger that can connect to your editor for interactive debugging through the UI.
00:00:46.739
Today, I'm going to tell you a little bit more about the Ruby LSP, which is another editor tool that we are working on. If you're interested in following my work, you can find me at ViniStock.
00:01:01.620
To understand language servers, we first need to look back at how language features were added to editors before they existed, so we can appreciate the problems they were trying to solve.
00:01:08.880
By language features, I mean things like formatting, go-to definition, autocomplete, and similar functionalities.
00:01:15.180
Imagine you were a Vim user wanting to get those language features for Ruby inside of Vim. To do so, you would need a Ruby plugin implementing those features for Vim.
00:01:25.799
If you also used JavaScript in a Rails application and wanted specialized features for JavaScript, you would need a separate JavaScript plugin for that. The same situation applies to any programming language you want to support.
00:01:41.520
This separation is beneficial because it allows you to pick and choose only the support for the languages you work with. However, if someone else uses a different editor like Sublime and wants specialized Ruby features, they unfortunately cannot reuse all the features already implemented for Vim.
00:02:04.079
The reasons are twofold: first, plugins made for different editors are often written in different programming languages (for example, VS Code uses JavaScript, while Vim uses Vimscript). Second, the internal designs of the editors can be significantly different; the way they handle buffers, files, and windows—and consequently the APIs they expose—can vary widely. Therefore, there is little hope of simply sharing a plugin made for one editor with a different one.
00:02:29.640
As a result, you would need to recreate that Ruby plugin and reimplement all of its features for every other programming language. Adding as many programming languages or editors into the mix makes this problem even more complex and fragmented, leading to a situation where the developer experience is tightly coupled with the specific editor used.
00:02:54.780
Moreover, there was no standard set of features to expect from a language plugin, creating a lack of guidance on what it meant to implement language plugins. Comparing Ruby in Vim and Ruby in VS Code could yield completely different sets of features, and the discrepancy could be even larger if you compared across programming languages.
00:03:30.420
Consequently, rather than having a unified Ruby plugin for each editor, you had one most popular Ruby plugin, and then smaller plugins would emerge to implement slightly different or specialized functionalities, such as better syntax highlighting, go-to definition, or linting. This fragmentation led to a subpar developer experience where you often needed to install a combination of plugins and properly configure them to avoid conflicts.
00:04:03.840
To tackle this problem, the Language Server Protocol (LSP) was proposed. The idea behind LSP is that it is a specification for creating a background server that runs continuously on the developer's machine and communicates with any editor via JSON.
00:04:34.259
In addition to outlining how to write the background server, LSP provides a list of features that one can expect from any language server, regardless of the programming language. The editor acts as the client in a manner similar to how browsers function in web applications, while the language server runs continuously in the background. They communicate through standard input, standard error, and standard output using JSON.
00:05:39.960
For instance, if you try to go to the definition of a method in the editor, the editor translates that action into a JSON request sent to the server. The server then figures out where the definition of the method is and returns the response as JSON to the editor, which in turn translates this response into the necessary behavior, such as jumping to the specific file and line number.
00:06:03.660
After implementing language servers, the landscape of development began to change. Editors can now implement a client layer to handle these JSON translations from actions to requests and responses. Various editors can connect to a single language server, which implements all the language-specific features, effectively decoupling the editor-specific parts from the programming language-specific parts.
00:06:35.819
All editor-specific components reside in the client layer, while all features are implemented in the language server itself. This allows us to collaborate in improving the overall developer experience. Because clients communicate solely through JSON, they can connect to any language server, irrespective of the programming language, making it a versatile and powerful solution.
00:07:42.040
To illustrate, here's what a request in the LSP looks like: it includes an ID, which is an incremental number used to identify the request and allows the server to inform the editor which requests it is responding to, as responses from a multi-threaded server aren't guaranteed to be in sequential order.
00:08:12.720
The method refers to the feature you are trying to compute, similar to making a request to a specific route in a web application. The request parameters typically include the URI of the text document.
00:08:54.900
Another way to communicate with LSPs is through notifications, which differ from requests in that they do not have an ID, as they don't expect a response back. For instance, a notification when a text document opens is sent from the editor to the server and includes parameters like the URI of that document and its current content.
00:09:20.520
When starts interacting with the language server, it looks something like this: Once you open your editor in a Ruby workspace, the first step is to activate the language server by sending an initialize request to the server. The server then responds with the set of features it currently implements.
00:09:53.460
This means you don't need to implement the whole specification right away; you can do it incrementally, allowing the server to inform editors of the features it supports so that only relevant requests are received.
00:10:09.899
After activation, regular coding operations begin, like opening a file. The editor notifies the server when a file is opened, and it becomes the server's responsibility to maintain an internal representation of that document. It needs to know that a file exists at that specific URI and keep track of its contents at all times.
00:10:34.860
This process is known as text synchronization, ensuring that the text is synchronized from the editor into the server state. Once that notification is received, the editor requests features for the newly opened file, resulting in a series of language feature requests like folding range.
00:11:06.360
To illustrate, if we edit a file named food.rb and add a method definition, the editor will notify the server about the content changes, giving it a chance to update its internal representation. Since contents have changed, the resulting language features may have also changed, prompting another round of feature requests to be computed.
00:11:34.740
The Ruby LSP is a new language server we are developing for Ruby. There are two repositories mentioned here: one is the VS Code Ruby LSP, which is the VS Code extension connecting to the server, and the other is the Ruby LSP gem, which is the actual server.
00:12:20.340
It's important to note that the Ruby LSP gem can connect to any editor that supports the protocol, not just VS Code. Additionally, the server is entirely written in Ruby, so you can follow along with the examples I'm about to show.
00:12:45.240
So, what features does the Ruby LSP currently support? A few examples include folding ranges, which are the little icons found in the gutter of the editor allowing you to collapse or expand code.
00:13:16.560
Also, document highlights—when you click on a Ruby entity, it highlights other occurrences of that entity in the editor. It supports Robocop diagnostics, surfacing violations directly in the editor, and provides quick fixes through code action to address issues directly in the UI.
00:13:56.999
We have hover functionality that displays documentation links for Rails DSL methods. For example, if you hover over the "has_one" method, a link to the documentation will pop up, and clicking it takes you directly to the website.
00:14:34.020
Inlay hints are also available; for instance, if you rescue without specifying a specific error class, the Ruby LSP provides an inlay hint to inform you that the default behavior is to rescue from StandardError.
00:15:11.220
We also have auto-formatting capabilities, so if you save a file with formatting enabled, it will fix any violations. Finally, there's a feature called format on type, which auto-closes end tokens when you break lines, like when adding an "if" statement.
00:15:44.340
Now, let’s see how we implement these types of features in Ruby. In terms of the overall architecture of an LSP, it is essentially an infinite loop that keeps reading JSON requests from standard input.
00:16:22.740
Based on the method attribute—the feature we want to compute—we must define what it means to execute that request. After running the request and obtaining a result, we write the result as JSON into the standard output, returning it to the editor so it can provide the necessary features.
00:17:03.960
Like any language server, the Ruby LSP has various responsibilities, one of which is text synchronization, as previously mentioned. It also needs to parse Ruby files and handle positional requests, where the cursor position matters.
00:17:40.680
Moreover, it handles non-positional requests, which are features computed for the entire file at once. We will primarily focus on implementing non-positional requests, particularly the folding range feature.
00:18:22.220
Folding ranges allow users to expand or collapse blocks of code, and this feature needs to be computed once for the entire file. To do this, we obtain the folding range request, which only includes the document's URI as a parameter.
00:19:10.140
By the time we receive this request, the server must already have an internal representation of the document, as it does not provide the text content. Hence, it must know the document's contents prior to this request.
00:19:57.060
The expected response from the server is a list of ranges, and the minimum necessary to compose a range includes the start line, end line, and category (like 'region' for generic code parts, 'comment', or 'imports').
00:20:20.880
Now, taking a 'post' class as an example, we'd want to identify where we need folding, such as in method and class definitions. However, while synchronizing text from the editor to the server, we only deal with raw text, making it challenging to analyze.
00:20:47.340
To work around this, we parse the file into an abstract syntax tree (AST), which forms an object that describes every component present in the file. The final step of analysis involves traversing the AST and performing analyses needed to collect useful information on where folding can occur.
00:21:22.320
While I don't have time to delve too deeply into parsing today, it's the process of converting the raw Ruby code string into an object representing the structures present in that file—often depicted as a tree or linked list.
00:22:23.700
In our 'post' class, we identify the class node as the entry point of our AST, which contains two child nodes: the 'post' constant and the body. Inside that body, we find a method definition consisting of two child nodes: the 'title' identifier and the method's body. Finally, in the body, we only find the string 'rubyconf'.
00:23:10.680
We want to be able to search through this AST to identify where we want to fold our code. There are numerous ways to traverse an AST, but we'll utilize the syntax tree method that the Ruby LSP employs, which uses the visitor pattern.
00:23:55.440
In the visitor pattern, we start with a parent class called 'Visitor,' from which all non-positional requests inherit. The entry point for our analysis is the visit method, which accepts any node type present in our Ruby file.
00:24:34.740
By invoking 'accept' on each node and passing it the current instance of the visitor, every node type must implement the accept method. This method is analogous to invoking specific visit methods in the Visitor class.
00:25:08.940
For example, if we encounter a class node, we invoke the visit class method from the visitor. The idea is to avoid using a large case statement by letting the node invoke the appropriate process method for its type back on the visitor.
00:25:45.360
To enable traversal of the AST, we set a default behavior for node-specific visits, which defaults to visiting each child node of the current node being processed. This design allows us to traverse the AST and reach every node in the framework efficiently.
00:26:36.420
Once this pattern is established, subclassing the visitor allows us to implement specific requests such as folding ranges. We initialize this subclass with the AST for the current file and an array to accumulate results.
00:27:08.640
Visiting the AST means traversing every node defined in the Ruby file and returning a list of ranges found along the way. When we identify nodes we wish to fold, we can override the default visits in our implementation.
00:27:44.760
For instance, if we find a class node, we take its location information and push a new range for that definition into our result array. Continuing this process, we can pause the analysis as needed; if we discover a specific class node, we can push its range.
00:28:22.140
Completing this process, we also implement folding for method definitions. Essentially, the same method applies as with the class nodes. We capture the location of the method node and add a new range for it as well.
00:29:05.880
By following these methods, we've developed functionality to easily collapse the entire 'post' class that we discussed earlier. This functionality for non-positional requests works similarly; as you're typing, we synchronize your text edits from the editor into the server.
00:29:48.540
When you pause typing, once the debounce period finishes, we parse the file, pass the resulting AST to every non-positional request, and all of these requests are also implemented using the visitor pattern.
00:30:25.920
The visitor subclasses allow each request implementation while utilizing the basic traversal logic, thus minimizing complexity.
00:30:58.920
I invite everyone to try out the Ruby LSP and provide feedback. If you are interested in contributing, remember that it's pure Ruby, and I'm more than happy to help anyone willing to contribute.
00:31:44.640
The features we currently support include format on save, format on type, document highlights, hover functionality, Robocop diagnostics, quick fixes, selection ranges, semantic highlighting, inlay hints, document links, and folding ranges.
00:32:10.200
Looking to the future, I am very excited about what lies ahead for Ruby tooling. We can implement many more features that I mentioned earlier, as we continue to improve.
00:32:55.200
Interactive debugging directly in the editor UI is on our radar, utilizing the Debug Adapter Protocol (DAP), which serves to connect debuggers to editors. Additionally, we can explore gradual typing for more precise features such as go-to definitions and autocomplete.
00:33:48.180
As a result, any editor can become a fully featured Ruby IDE, and we can collaborate towards achieving that vision. Thank you very much!
00:34:28.200
You can connect with any editor that supports the protocol.
00:34:39.740
Oh, sorry, yes, the question was if there's only a client for VS Code.
00:34:51.400
You can connect with any editor that supports the protocol.
00:35:06.080
Some community members have managed to configure Neovim to connect to it.
00:35:17.060
The reason for a VS Code extension is required is that there is no built-in LSP support.
00:35:36.260
However, integrating it with Neovim or other editors supporting the LSP protocol is possible.
00:35:43.600
The question was whether Ruby LSP can work alongside Solargraph.
00:36:05.020
Yes, both can be used together as the language servers can merge responses, even from conflicting ones.
00:36:21.480
At Shopify, we actually use Ruby LSP alongside Sorbet, which also provides LSP features.
00:36:50.640
The question involved whether the LSP could be connected to a REPL environment for executing code.
00:37:06.280
While the specification does not explicitly address that, it is indeed possible to create such a capability.
00:37:35.460
However, challenges may arise when managing larger projects due to performance issues.
00:38:00.660
The following question was if format on save is powered by Robocop.
00:38:12.740
Yes, it hooks into Robocop when available, while providing a fallback to SyntaxTree.
00:38:24.800
We initially supported Shadow environment as our in-house Ruby version manager, but support for RB Environment has been added due to community contributions.
00:38:41.180
Additionally, we have support for CH Ruby and RVM as well.
00:39:00.360
The question was whether the client controls the process running the server and how many server processes are spawned.
00:39:25.640
Yes, the client does control it. The editor spawns the process and holds the standard I/O pipes to communicate.
00:39:45.860
Only one server instance is spawned for each coding session, which remains active as long as the workspace is open.
00:40:05.920
Lastly, the question was raised about potentially integrating another tool aside from the server and editor.
00:40:25.480
Yes, integration of additional tools for providing error highlights or other features is feasible, just like with Robocop.
00:40:39.920
If there are no further questions, I'm here until the conference ends.
00:40:54.000
Feel free to approach me for any discussions about developer experience.
00:41:08.500
I also ordered stickers for the Ruby LSP that haven't arrived yet, so come see me after they do.
00:41:29.500
Thank you again!
