Improving the development experience with language servers

by Vinicius Stock

The video titled "Improving the Development Experience with Language Servers," presented by Vinicius Stock at RubyConf 2022, explores the benefits of using the Language Server Protocol (LSP) to enhance the development experience for Ruby programmers. Stock, a member of the Ruby Developer Experience team at Shopify, discusses the importance of a well-supported development environment and how LSP plays a crucial role in unifying language features across different code editors.

Key points discussed in the presentation include:

- Historical Context: Before LSP, developers faced fragmentation with language plugins tailored for specific editors, which resulted in a disjointed developer experience. Each editor had its own implementations for language features like autocomplete, syntax highlighting, and debugging, which could not be shared across different programming environments.
- Introduction to LSP: The LSP addresses these issues by allowing a single language server to provide language features over a protocol, enabling any editor to communicate with the server using JSON. This decouples the language support from the specific editors, simplifying development tools.
- Operational Mechanics: Stock describes how the LSP functions, detailing the communication between the editor (client) and the server, the request-response cycle, and examples of the types of requests made, such as initialization and text synchronization.
- Ruby LSP Features: He emphasizes the capabilities of the Ruby LSP, highlighting features like document highlighting, diagnostics via Robocop, hover functionality for documentation, inlay hints, and formatting capabilities. The feature implementation process was also discussed, including how to handle both positional and non-positional requests using abstract syntax trees (AST).
- Future Prospects: In terms of future development, Stock expresses excitement about what could be possible with the Ruby LSP, suggesting further improvements such as integrating interactive debugging and gradual typing into editors.

In conclusion, the Ruby LSP represents a significant step towards creating a unified and enjoyable developer experience across various programming environments. The integration of LSP into Ruby tooling promises to make any code editor a fully functional IDE, ultimately enhancing productivity and satisfaction for Ruby developers. Vinicius Stock encourages contributions to the Ruby LSP project and expresses eagerness for collaborative development in the Ruby community.

00:00:00.000 Ready for takeoff.

00:00:18.600 Hello everybody, welcome to "Improving the Development Experience with Language Servers."

00:00:24.960 I'm Vinnie, and I'm part of the Ruby Developer Experience team at Shopify. We do a lot of work with developer tools.

00:00:31.679 All of our projects are connected to the editor, so we're involved with gradual typing using Sorbet and Tapioca.

00:00:40.800 We recently started contributing to Ruby Debug, which is a debugger that can connect to your editor for interactive debugging through the UI.

00:00:46.739 Today, I'm going to tell you a little bit more about the Ruby LSP, which is another editor tool that we are working on. If you're interested in following my work, you can find me at ViniStock.

00:01:01.620 To understand language servers, we first need to look back at how language features were added to editors before they existed, so we can appreciate the problems they were trying to solve.

00:01:08.880 By language features, I mean things like formatting, go-to definition, autocomplete, and similar functionalities.

00:01:15.180 Imagine you were a Vim user wanting to get those language features for Ruby inside of Vim. To do so, you would need a Ruby plugin implementing those features for Vim.

00:01:25.799 If you also used JavaScript in a Rails application and wanted specialized features for JavaScript, you would need a separate JavaScript plugin for that. The same situation applies to any programming language you want to support.

00:01:41.520 This separation is beneficial because it allows you to pick and choose only the support for the languages you work with. However, if someone else uses a different editor like Sublime and wants specialized Ruby features, they unfortunately cannot reuse all the features already implemented for Vim.

00:02:04.079 The reasons are twofold: first, plugins made for different editors are often written in different programming languages (for example, VS Code uses JavaScript, while Vim uses Vimscript). Second, the internal designs of the editors can be significantly different; the way they handle buffers, files, and windows—and consequently the APIs they expose—can vary widely. Therefore, there is little hope of simply sharing a plugin made for one editor with a different one.

00:02:29.640 As a result, you would need to recreate that Ruby plugin and reimplement all of its features for every other programming language. Adding as many programming languages or editors into the mix makes this problem even more complex and fragmented, leading to a situation where the developer experience is tightly coupled with the specific editor used.

00:02:54.780 Moreover, there was no standard set of features to expect from a language plugin, creating a lack of guidance on what it meant to implement language plugins. Comparing Ruby in Vim and Ruby in VS Code could yield completely different sets of features, and the discrepancy could be even larger if you compared across programming languages.

00:03:30.420 Consequently, rather than having a unified Ruby plugin for each editor, you had one most popular Ruby plugin, and then smaller plugins would emerge to implement slightly different or specialized functionalities, such as better syntax highlighting, go-to definition, or linting. This fragmentation led to a subpar developer experience where you often needed to install a combination of plugins and properly configure them to avoid conflicts.

00:04:03.840 To tackle this problem, the Language Server Protocol (LSP) was proposed. The idea behind LSP is that it is a specification for creating a background server that runs continuously on the developer's machine and communicates with any editor via JSON.

00:04:34.259 In addition to outlining how to write the background server, LSP provides a list of features that one can expect from any language server, regardless of the programming language. The editor acts as the client in a manner similar to how browsers function in web applications, while the language server runs continuously in the background. They communicate through standard input, standard error, and standard output using JSON.

00:05:39.960 For instance, if you try to go to the definition of a method in the editor, the editor translates that action into a JSON request sent to the server. The server then figures out where the definition of the method is and returns the response as JSON to the editor, which in turn translates this response into the necessary behavior, such as jumping to the specific file and line number.

00:06:03.660 After implementing language servers, the landscape of development began to change. Editors can now implement a client layer to handle these JSON translations from actions to requests and responses. Various editors can connect to a single language server, which implements all the language-specific features, effectively decoupling the editor-specific parts from the programming language-specific parts.

00:06:35.819 All editor-specific components reside in the client layer, while all features are implemented in the language server itself. This allows us to collaborate in improving the overall developer experience. Because clients communicate solely through JSON, they can connect to any language server, irrespective of the programming language, making it a versatile and powerful solution.

00:07:42.040 To illustrate, here's what a request in the LSP looks like: it includes an ID, which is an incremental number used to identify the request and allows the server to inform the editor which requests it is responding to, as responses from a multi-threaded server aren't guaranteed to be in sequential order.

00:08:12.720 The method refers to the feature you are trying to compute, similar to making a request to a specific route in a web application. The request parameters typically include the URI of the text document.

00:08:54.900 Another way to communicate with LSPs is through notifications, which differ from requests in that they do not have an ID, as they don't expect a response back. For instance, a notification when a text document opens is sent from the editor to the server and includes parameters like the URI of that document and its current content.

00:09:20.520 When starts interacting with the language server, it looks something like this: Once you open your editor in a Ruby workspace, the first step is to activate the language server by sending an initialize request to the server. The server then responds with the set of features it currently implements.

00:09:53.460 This means you don't need to implement the whole specification right away; you can do it incrementally, allowing the server to inform editors of the features it supports so that only relevant requests are received.

00:10:09.899 After activation, regular coding operations begin, like opening a file. The editor notifies the server when a file is opened, and it becomes the server's responsibility to maintain an internal representation of that document. It needs to know that a file exists at that specific URI and keep track of its contents at all times.

00:10:34.860 This process is known as text synchronization, ensuring that the text is synchronized from the editor into the server state. Once that notification is received, the editor requests features for the newly opened file, resulting in a series of language feature requests like folding range.

00:11:06.360 To illustrate, if we edit a file named food.rb and add a method definition, the editor will notify the server about the content changes, giving it a chance to update its internal representation. Since contents have changed, the resulting language features may have also changed, prompting another round of feature requests to be computed.

00:11:34.740 The Ruby LSP is a new language server we are developing for Ruby. There are two repositories mentioned here: one is the VS Code Ruby LSP, which is the VS Code extension connecting to the server, and the other is the Ruby LSP gem, which is the actual server.

00:12:20.340 It's important to note that the Ruby LSP gem can connect to any editor that supports the protocol, not just VS Code. Additionally, the server is entirely written in Ruby, so you can follow along with the examples I'm about to show.

00:12:45.240 So, what features does the Ruby LSP currently support? A few examples include folding ranges, which are the little icons found in the gutter of the editor allowing you to collapse or expand code.

00:13:16.560 Also, document highlights—when you click on a Ruby entity, it highlights other occurrences of that entity in the editor. It supports Robocop diagnostics, surfacing violations directly in the editor, and provides quick fixes through code action to address issues directly in the UI.

00:13:56.999 We have hover functionality that displays documentation links for Rails DSL methods. For example, if you hover over the "has_one" method, a link to the documentation will pop up, and clicking it takes you directly to the website.

00:14:34.020 Inlay hints are also available; for instance, if you rescue without specifying a specific error class, the Ruby LSP provides an inlay hint to inform you that the default behavior is to rescue from StandardError.

00:15:11.220 We also have auto-formatting capabilities, so if you save a file with formatting enabled, it will fix any violations. Finally, there's a feature called format on type, which auto-closes end tokens when you break lines, like when adding an "if" statement.

00:15:44.340 Now, let’s see how we implement these types of features in Ruby. In terms of the overall architecture of an LSP, it is essentially an infinite loop that keeps reading JSON requests from standard input.

00:16:22.740 Based on the method attribute—the feature we want to compute—we must define what it means to execute that request. After running the request and obtaining a result, we write the result as JSON into the standard output, returning it to the editor so it can provide the necessary features.

00:17:03.960 Like any language server, the Ruby LSP has various responsibilities, one of which is text synchronization, as previously mentioned. It also needs to parse Ruby files and handle positional requests, where the cursor position matters.

00:17:40.680 Moreover, it handles non-positional requests, which are features computed for the entire file at once. We will primarily focus on implementing non-positional requests, particularly the folding range feature.

00:18:22.220 Folding ranges allow users to expand or collapse blocks of code, and this feature needs to be computed once for the entire file. To do this, we obtain the folding range request, which only includes the document's URI as a parameter.

00:19:10.140 By the time we receive this request, the server must already have an internal representation of the document, as it does not provide the text content. Hence, it must know the document's contents prior to this request.

00:19:57.060 The expected response from the server is a list of ranges, and the minimum necessary to compose a range includes the start line, end line, and category (like 'region' for generic code parts, 'comment', or 'imports').

00:20:20.880 Now, taking a 'post' class as an example, we'd want to identify where we need folding, such as in method and class definitions. However, while synchronizing text from the editor to the server, we only deal with raw text, making it challenging to analyze.

00:20:47.340 To work around this, we parse the file into an abstract syntax tree (AST), which forms an object that describes every component present in the file. The final step of analysis involves traversing the AST and performing analyses needed to collect useful information on where folding can occur.

00:21:22.320 While I don't have time to delve too deeply into parsing today, it's the process of converting the raw Ruby code string into an object representing the structures present in that file—often depicted as a tree or linked list.

00:22:23.700 In our 'post' class, we identify the class node as the entry point of our AST, which contains two child nodes: the 'post' constant and the body. Inside that body, we find a method definition consisting of two child nodes: the 'title' identifier and the method's body. Finally, in the body, we only find the string 'rubyconf'.

00:23:10.680 We want to be able to search through this AST to identify where we want to fold our code. There are numerous ways to traverse an AST, but we'll utilize the syntax tree method that the Ruby LSP employs, which uses the visitor pattern.

00:23:55.440 In the visitor pattern, we start with a parent class called 'Visitor,' from which all non-positional requests inherit. The entry point for our analysis is the visit method, which accepts any node type present in our Ruby file.

00:24:34.740 By invoking 'accept' on each node and passing it the current instance of the visitor, every node type must implement the accept method. This method is analogous to invoking specific visit methods in the Visitor class.

00:25:08.940 For example, if we encounter a class node, we invoke the visit class method from the visitor. The idea is to avoid using a large case statement by letting the node invoke the appropriate process method for its type back on the visitor.

00:25:45.360 To enable traversal of the AST, we set a default behavior for node-specific visits, which defaults to visiting each child node of the current node being processed. This design allows us to traverse the AST and reach every node in the framework efficiently.

00:26:36.420 Once this pattern is established, subclassing the visitor allows us to implement specific requests such as folding ranges. We initialize this subclass with the AST for the current file and an array to accumulate results.

00:27:08.640 Visiting the AST means traversing every node defined in the Ruby file and returning a list of ranges found along the way. When we identify nodes we wish to fold, we can override the default visits in our implementation.

00:27:44.760 For instance, if we find a class node, we take its location information and push a new range for that definition into our result array. Continuing this process, we can pause the analysis as needed; if we discover a specific class node, we can push its range.

00:28:22.140 Completing this process, we also implement folding for method definitions. Essentially, the same method applies as with the class nodes. We capture the location of the method node and add a new range for it as well.

00:29:05.880 By following these methods, we've developed functionality to easily collapse the entire 'post' class that we discussed earlier. This functionality for non-positional requests works similarly; as you're typing, we synchronize your text edits from the editor into the server.

00:29:48.540 When you pause typing, once the debounce period finishes, we parse the file, pass the resulting AST to every non-positional request, and all of these requests are also implemented using the visitor pattern.

00:30:25.920 The visitor subclasses allow each request implementation while utilizing the basic traversal logic, thus minimizing complexity.

00:30:58.920 I invite everyone to try out the Ruby LSP and provide feedback. If you are interested in contributing, remember that it's pure Ruby, and I'm more than happy to help anyone willing to contribute.

00:31:44.640 The features we currently support include format on save, format on type, document highlights, hover functionality, Robocop diagnostics, quick fixes, selection ranges, semantic highlighting, inlay hints, document links, and folding ranges.

00:32:10.200 Looking to the future, I am very excited about what lies ahead for Ruby tooling. We can implement many more features that I mentioned earlier, as we continue to improve.

00:32:55.200 Interactive debugging directly in the editor UI is on our radar, utilizing the Debug Adapter Protocol (DAP), which serves to connect debuggers to editors. Additionally, we can explore gradual typing for more precise features such as go-to definitions and autocomplete.

00:33:48.180 As a result, any editor can become a fully featured Ruby IDE, and we can collaborate towards achieving that vision. Thank you very much!

00:34:28.200 You can connect with any editor that supports the protocol.

00:34:39.740 Oh, sorry, yes, the question was if there's only a client for VS Code.

00:34:51.400 You can connect with any editor that supports the protocol.

00:35:06.080 Some community members have managed to configure Neovim to connect to it.

00:35:17.060 The reason for a VS Code extension is required is that there is no built-in LSP support.

00:35:36.260 However, integrating it with Neovim or other editors supporting the LSP protocol is possible.

00:35:43.600 The question was whether Ruby LSP can work alongside Solargraph.

00:36:05.020 Yes, both can be used together as the language servers can merge responses, even from conflicting ones.

00:36:21.480 At Shopify, we actually use Ruby LSP alongside Sorbet, which also provides LSP features.

00:36:50.640 The question involved whether the LSP could be connected to a REPL environment for executing code.

00:37:06.280 While the specification does not explicitly address that, it is indeed possible to create such a capability.

00:37:35.460 However, challenges may arise when managing larger projects due to performance issues.

00:38:00.660 The following question was if format on save is powered by Robocop.

00:38:12.740 Yes, it hooks into Robocop when available, while providing a fallback to SyntaxTree.

00:38:24.800 We initially supported Shadow environment as our in-house Ruby version manager, but support for RB Environment has been added due to community contributions.

00:38:41.180 Additionally, we have support for CH Ruby and RVM as well.

00:39:00.360 The question was whether the client controls the process running the server and how many server processes are spawned.

00:39:25.640 Yes, the client does control it. The editor spawns the process and holds the standard I/O pipes to communicate.

00:39:45.860 Only one server instance is spawned for each coding session, which remains active as long as the workspace is open.

00:40:05.920 Lastly, the question was raised about potentially integrating another tool aside from the server and editor.

00:40:25.480 Yes, integration of additional tools for providing error highlights or other features is feasible, just like with Robocop.

00:40:39.920 If there are no further questions, I'm here until the conference ends.

00:40:54.000 Feel free to approach me for any discussions about developer experience.

00:41:08.500 I also ordered stickers for the Ruby LSP that haven't arrived yet, so come see me after they do.

00:41:29.500 Thank you again!