IDE development with Ruby

Talks

Soutaro Matsumoto

@soutaro

#ruby

#integrated-development-environment-ide

#language-server-protocol-lsp

IDE development with Ruby

by Soutaro Matsumoto

In this video, Soutaro Matsumoto presents an in-depth exploration of IDE (Integrated Development Environment) development, particularly focusing on tools that support Ruby programming with a static type checker, implemented via the Language Server Protocol (LSP). Matsumoto, a lead Ruby engineer at Square and an active contributor to Ruby's development, aims to provide insights into how IDE features can enhance programming experiences.

Key Points Discussed:
- IDE Definition and Functionality:
- IDEs serve as advanced text editors that facilitate coding through features like error reporting, code completion, and navigation.
- They provide a significantly improved user experience when compared to simple text editors.

Introduction to Steep:
- Steep is a static type checker designed for Ruby, introduced to enhance IDE functionality through type checking.
- Originally a command-line tool, Steep is integrated into IDEs for real-time diagnostics and user interactivity.
Essential Features of IDEs:
- Matsumoto highlights critical text editing features available in leading IDEs, such as syntax highlighting, code folding, and diagnostic reporting.
- Each feature contributes to a more efficient and user-friendly coding environment.
Language Server Protocol (LSP):
- LSP facilitates the development of consistent language support across different IDEs, allowing for shared analysis capabilities.
- It promotes communication between IDEs and language servers, reducing the need for dedicated implementations for each technology.
Architecture of Steep:
- Steep operates through a server-client communication structure where IDEs send notifications to Steep to perform type checks.
- Matsumoto discusses how optimizing type checking improves IDE responsiveness, particularly through increments and prioritizing active documents.
User Experience Optimization:
- Techniques such as incremental type checking and filtering unrelated code during completion requests are discussed as means to enhance user experience and efficiency.

Conclusions and Takeaways:
Matsumoto concludes that developing IDEs requires sophisticated program analyses, with the adoption of LSP allowing for broader collaboration among tools. The integration of advanced features enhances the programming experience, particularly for Ruby developers utilizing Steep as their go-to type checker. The talk encourages developers to explore language servers that best fit their needs and highlights the increasing community interest in LSP.

00:00:00.240 It is time to introduce our next speaker, Soutaro Matsumoto. Soutaro is a lead Ruby engineer at Square, working on static typing for Ruby. He is a core Ruby committer and designed and implemented RBS for Ruby 3. He also developed Steep, a static type checker for Ruby.

00:00:11.599 The integrated development environment, or IDE, is one of the most frequently used tools for programming. It’s a kind of text editor where you type and read the code, but it does more for you, including on-the-fly error reporting, completion, go-to-definition, and more. These features help you write and read the code, making the tools more valuable than simple text editors.

00:00:39.440 Soutaro has been working on IDE development to support Ruby programming with a static type checker. It is based on the Language Server Protocol (LSP) and implemented in Ruby. He wants to share his experience with us today. What is the protocol? How can the LSP features be implemented? You will gain insights into the under-the-hood workings of IDEs, making the tools more familiar to you.

00:01:10.159 Now, let's start with his presentation. All questions can be directed to the stream chat, and Soutaro, it's your turn.

00:01:23.840 Okay, so here I am. This is my first time attending Euruko, and I am very excited to present today. You may notice that my family name is the same as Yukihiro Matsumoto's. You might be wondering if we are related, but actually, we are not. Matsumoto is a really common surname in Japan. Let me introduce myself a bit further.

00:02:19.000 I work at Square, and I live in Tokyo, Japan. I am one of the Ruby committers and have been working on type checking for Ruby, which was released as RBS last year. I also developed a static type checker called Steep, which is based on RBS.

00:02:25.840 My talk today will not be directly about RBS or Steep but will focus more on the development of IDEs. However, before I start, I quickly want to introduce Steep. It is uncommon to implement an IDE in Ruby, but you can install the tool using RubyGems or Bundler.

00:03:01.440 Initially, it was developed as a command line tool, so you can run type checks on your projects with the Steep check command. It will read all the Ruby source code and perform type checking, reporting any errors to the console.

00:03:28.640 However, relying solely on a command line tool is not a great user experience. You write some Ruby code in a text editor, switch to the terminal, run the Steep check, and then return to the editor to fix any errors, which can be cumbersome.

00:03:54.160 To improve this experience, we developed an IDE plugin that integrates with Steep. I developed a Steep extension for Visual Studio Code, allowing you to install the extension from the marketplace. This plugin assists with reading and writing code by providing on-the-fly diagnostics reporting, completion, and navigation between different parts of your code.

00:04:19.680 In this talk, I will focus on what an IDE is and the features it provides. For instance, Visual Studio is a very established IDE, having existed since approximately 1995. In fact, Visual Basic was my first programming language, and I wrote code on Windows using an IDE and did some debugging, among other activities.

00:04:41.920 Of course, one of the most popular IDEs today is VS Code, which I have used for writing Android applications for 10 years. We should also mention the fantastic IDEs released by JetBrains. I really love RubyMine, as it is one of my primary development environments. Other examples include Xcode for iOS development and Android Studio for Android.

00:05:06.960 You might say that Emacs is an IDE; indeed, you can do almost anything in Emacs. Interestingly, I found a funny phrase on the website of Vim, which states that many consider it a complete IDE. Thus, it's apparent that discussing which IDE is better can lead to heated debates.

00:05:41.760 Every IDE has its strengths. There are numerous features that help you read and write code, compile applications, configure settings, run tests, and debug your applications. Most IDEs provide graphical user interfaces to manage these activities more efficiently.

00:06:03.440 In this talk, I will specifically focus on the text editing features of IDEs. These include syntax highlighting, code folding, diagnostic reporting, navigation, and completion features—elements that significantly enhance the coding experience.

00:06:40.960 For instance, I have a screenshot of Visual Studio Code showing its syntax highlighting and folding features. You can see symbols in the left column of the editor, allowing users to fold and collapse specific sections of the Ruby code, making it easier to navigate through large codebases.

00:07:10.399 Diagnostic reporting is also essential when editing code. In the IDE, as you work, the program analyzes your code in the background. If any issues are detected, they will be reported immediately in the editor, allowing you to address them without needing to switch to another interface.

00:07:36.480 Hovering over certain elements in your code can provide instant documentation. For example, if your mouse pointer hovers over a variable or method call, it will show relevant documentation or type information. This feature is implemented by Steep.

00:08:02.379 Navigation features allow users to quickly jump to definitions within their codebase. By clicking on parts of the code, you can easily navigate to the methods or types being referenced, which enhances the coding workflow.

00:08:28.080 Completion features are also incredibly beneficial. When you start typing, the IDE automatically suggests completion candidates, alleviating the burden of typing out lengthy names. Screenshot examples of this functionality highlight how the IDE supports various refactoring tasks, such as renaming methods or extracting code into new methods.

00:09:15.360 However, IDE features are often dependent on an in-depth understanding of the programming language itself. For instance, implementing syntax highlighting or code folding requires knowledge of the language's grammar and syntax. This means that features such as error reporting, navigation, and others must be built on top of rigorous program analysis.

00:09:53.760 Before continuing, I want to outline some levels of program analysis. Some analyses are simple, while others are complex. Text-based analysis is the most fundamental type of analysis; it treats input as a sequence of characters without understanding their meaning.

00:10:21.680 Running an input function may check for issues like forbidden characters or overly long lines of code. More advanced analyses, such as syntactic analysis, understand the structure of the code and can implement features like syntax highlighting or diagnostics.

00:10:53.680 Semantic analysis is one of the most complicated forms of program analysis, as it involves understanding the types of variables and how they are used throughout the program. This level of analysis is necessary for advanced features like navigation, completion, and refactoring.

00:11:23.680 As I continue my talk, it’s essential to understand that developing IDEs is notably challenging. IDEs require their own program analyzers because different technologies and languages often have varying APIs, requiring separate implementations for features common across IDEs.

00:11:47.620 A potential solution is to extract common analysis functionalities from IDEs and use the Language Server Protocol (LSP) to facilitate communication between IDE front-ends and backend analysis engines, thereby reducing the need for individual implementations.

00:12:08.620 The LSP was designed by Microsoft and released five years ago. It allows for communication between development tools and language servers through a protocol designed for interprocess communication, meaning language server implementations are not limited by their underlying technology.

00:12:56.120 Currently, LSP is supported by many popular text editors, including Visual Studio Code and Sublime Text, making it a widely adopted technology. Consequently, IDE development is improving, with language-specific features being developed alongside shared analysis engines.

00:13:29.760 Numerous language servers for Ruby exist, such as Steep's language server and Stratum, which is another Ruby language server. Steep was developed as a static type checker but supports the Language Server Protocol. My presentation will focus primarily on the implementation of Steep.

00:14:18.920 The architecture of Steep involves various IDE front-end tools communicating with Steep's language server via standard input/output. When events occur, such as text document changes, the IDE sends notifications to Steep, which processes them, performs type checking, and sends any errors back to the IDE.

00:15:06.239 For example, when a user types a new character, a notification is sent from the IDE to Steep in JSON format. The server then recognizes the change, updates the source code, and begins type checking. After this, any resulting type errors are communicated back to the IDE.

00:15:47.520 The server-client communication structure can be visually likened to that of a web server and browser, but LSP allows messages to be sent back from the server to the IDE, such as publishing diagnostics to inform users of errors as they code.

00:16:15.039 Here’s a simplified version of the language server implementation. The server receives events and checks if they indicate a change in text documents. If so, it updates the relevant source code and conducts type checking. However, a naive implementation may encounter responsiveness issues.

00:16:46.720 To ensure the language server remains responsive, I implemented some optimizations, the first being incremental type checking. This approach allows the server to focus on the changed file rather than re-checking the entire project, leading to faster type checking.

00:17:19.600 For instance, if there are 100 Ruby files and you make changes to only one, we can skip checking the other 99 files, which dramatically improves performance. This method involves a fast pass, where only the affected files are type-checked.

00:17:55.020 Conversely, if changes are made to files that define types and interfaces (such as RBS files), it may take longer to type check, as many functions depend on those definitions. This is similar to editing header files in C programming, triggering potential long re-compilation times.

00:18:35.760 To enhance user experience, I introduced prioritized type checking for open files. This strategy emphasizes checking files currently in the editor so that developers receive immediate feedback, facilitating a smoother coding experience.

00:19:12.920 Another key optimization involved dropping unrelated code during completion requests. Completion responsiveness is vital, as users will wait for potential candidates while coding. By filtering out methods unrelated to user input, we can speed up the search for relevant completions.

00:20:02.520 In conclusion, I have discussed IDE features that require advanced program analysis and how the LSP assists in developing those IDE features across various programming languages. I have been working on Steep as a static type checker for Ruby, and I shared insights into enhancing IDE responsiveness.

00:20:56.800 Thank you for your attention! That concludes my talk.

00:34:04.240 Oh, the joys of live events! As you said, Euruko 2021 is indeed lively. Thankfully, nothing alarming happened, and we can continue discussing the presentation.

00:34:24.800 The audience's response has been overwhelmingly positive, with many thanking you for the talk, and there's a question about which language server you would recommend for VS Code users.

00:34:45.200 My recommendation depends on what you're trying to achieve. If you're using Steep, it would be a great choice. Otherwise, Sorbet might be suitable, too. I personally love using Steep, as I develop features for the RBS gem.

00:35:16.160 There have been several compliments and interesting questions from the audience. For instance, one person expressed how your talk made them even more interested in LSP. I appreciate the community engagement and hope to continue the discussions on Discord or stream chat, as more questions may arise.

EuRuKo 2021