00:00:00.000
ready for takeoff
00:00:18.600
um hello everybody Welcome to improving the development experience with language servers
00:00:24.960
I'm Vinnie I'm a part of the Ruby developer experience team at Shopify we do a lot of work with developer tools
00:00:31.679
all of them somehow connected to the editor so we're involved with gradual typing using survey and tapioca we
00:00:40.800
recently started doing some contributions to Ruby debug which is a debugger that can connect to your editor
00:00:46.739
for interactive debugging through the UI and I'm going to tell you a little bit more today about the Ruby LSP which is
00:00:53.340
another editor tool that we're working on and if you want to know what I'm up to you can find me at Vini stock
00:01:01.620
so to understand language servers we first got to take a look back at how language features were added to editors
00:01:08.880
before they existed so that we can appreciate the problems they were trying to solve and by language features that
00:01:15.180
means stuff like formatting uh go to definition autocomplete that type of
00:01:20.520
thing so imagine you were a Vim user and you wanted to get those language features
00:01:25.799
for Ruby inside of Vim well in order to do that you would need a ruby plugin
00:01:30.960
implementing those features for vim and if you also used JavaScript say like in
00:01:36.420
the rails application and wanted to get specialized features for JavaScript you
00:01:41.520
would need a JavaScript plugin implementing those features and the same thing for any other programming language
00:01:47.159
that you wanted to get specialized features delivered in the editor and
00:01:52.259
this is this separation is great because you get to pick and choose only the support for the languages that you work
00:01:58.320
with but then imagine somebody else used a different editor like Sublime and they
00:02:04.079
also wanted to get specialized Ruby features inside of sublime Unfortunately they would not be able to
00:02:11.099
reuse all of the features that had already been implemented for vim and there are a few reasons for that but
00:02:17.580
mainly because plugins made for different editors are not made using the same programming language so for example
00:02:23.700
vs code uses JavaScript and Vin uses vimscript but even more impactful than
00:02:29.640
that the internal design of the editors can be completely different the way they handle buffers and files and windows and
00:02:37.080
so the API they end up exposing to their plug-in echo system can also be completely different so there's really
00:02:43.260
no hope of taking a plug-in made for one editor and just sharing it as is with a
00:02:48.720
different one so you would need to recreate that Ruby plug-in and re-implement all of those features and
00:02:54.780
the same thing for all of the other programming languages and if we add as many programming languages or editors
00:03:00.060
here the scenario the story repeats is always the same this meant that we couldn't come together as like a
00:03:06.540
community like the Ruby community and collaborate in making the developer experience better for everybody because
00:03:12.120
your developer experience was tightly coupled with which editor you used
00:03:17.220
in addition to that there was no standard set of features that you could expect from a language plugin there's no
00:03:22.800
guidance as to what it meant to implement language plugins so if you compared like Ruby in vim and Ruby and
00:03:30.420
vs code you could get a completely different set of features present in your editor and the difference could be
00:03:36.300
even larger if you compared across programming languages so you could have a very rich experience for JavaScript in
00:03:42.659
vs code but an incomplete experience for go in sublime or something like that
00:03:49.980
and so what ended up happening is that you actually didn't have one Ruby plug-in for each editor you ended up
00:03:56.220
having a most popular Ruby plug-in and then smaller plugins would emerge around it to implement slightly different or
00:04:03.840
specialized functionality like better syntax highlighting or uh go to definition or linting and this
00:04:10.379
fragmentation is not a great experience because it means you can't install a single plugin and get everything you
00:04:16.799
want better is included to work with Ruby you actually need to install a combination of plugins and make sure
00:04:22.560
they're configured the right way so that they don't conflict to tackle this problem the language
00:04:28.740
server protocol or LSP for short was proposed and the idea is that it's a
00:04:34.259
specification on how to write a background server so a process continuously running on the developer
00:04:39.900
machine that can communicate to any editor via Json and in addition to how to write the
00:04:46.800
background server it also provides us with this list of features that you can expect from any language server no
00:04:52.919
matter which programming language it is implemented for so the idea here is that the editor is
00:04:59.580
the client kind of like uh the browser in a web application and our backend is
00:05:04.800
the language server the process continuously running in the developer machine and they communicate with each
00:05:10.979
other through standard in standard error and standard out by using Json so for
00:05:16.740
example if you were to go to the definition of a method in the editor the editor knows how to translate that
00:05:22.320
action into a Json request which is sent to the server the server figures out
00:05:27.600
where the definition of that method is Returns the response as Json to the editor and the editor then knows how to
00:05:33.960
translate the Json response into whatever Behavior needs to occur which in the case of go to definition would be
00:05:39.960
to jump to the right file at the right position where that definition is located
00:05:45.900
so after language servers our story here changes a little bit uh the editors can
00:05:52.380
now Implement a client layer that knows how to do those Json translations from
00:05:57.720
actions to requests and from responses into behavior and that client
00:06:03.660
can then connect to a Ruby language server that implements all of the Ruby specific features and all of the other
00:06:10.380
editors can then Implement their own client layer which knows all of the editor specific design and can do all of
00:06:18.479
those translations but they can all connect to the same language server providing those features so you decouple
00:06:24.479
the editor specific parts from the programming language specific Parts all
00:06:29.880
of the editor parts are in the client layer and all of the features are implemented in the language server so we
00:06:35.819
can all collaborate on improving the experience and because the clients only deal with
00:06:41.580
Json they can connect to any language server it doesn't matter uh for which programming language they were made they
00:06:48.720
all communicate with Json so you can connect to any of them as a reference this is what a request in
00:06:55.979
the LSP looks like you have an ID which is an incremental number uh identifying
00:07:01.680
this request and it is used for the server to inform the editor which requests you are responding to because
00:07:07.740
your server may be multi-threaded so there's no guarantee that their responses are going to be sequential
00:07:13.440
uh the method is the feature that you're trying to compute and I like to think of
00:07:18.479
that kind of like a a route in a web application you're making a request to this route uh and then the set of
00:07:25.500
parameters of that request which in this case only includes the URI of the text document
00:07:31.440
another form of communication available to LSPs are notifications and the only difference between a request and a
00:07:38.340
notification is that the notification doesn't have an ID because it doesn't expect a response back so this is the
00:07:44.340
editor notifying the server that something has happened or vice versa the server notifying the editor but there's
00:07:50.880
no response involved so in this particular example a text document did open is a notification sent from the
00:07:57.840
editor to the server when a file has been opened in the UI and the parameters are the URI of that document and the
00:08:04.680
current content of the document if we were to take a look at a sequence
00:08:12.120
of interactions between the editor and the language server it would look like this once you open your editor in a ruby
00:08:20.400
workspace the first thing is activating the language server and to do that the
00:08:25.440
editor sends an initialize request to the server what it expects back as a response is
00:08:31.860
the set of features that your server currently implements so you don't have to implement the entire specification in
00:08:37.860
one go you can implement it gradually and your server has the opportunity to broadcast to editors where you currently
00:08:44.880
support so that you only receive requests for those features and that's commonly referred to as the capabilities
00:08:50.519
of the server once it's been activated then we can start with like regular coding
00:08:56.640
operations like opening a file if we open a file the editor is going to notify the server that that file has
00:09:03.180
been opened and it is the responsibility of the server to maintain an internal representation of that document it needs
00:09:09.839
to know that for that URI for that file path in the system there exists a file
00:09:15.180
and it needs to know the contents of that file at all times and this process is known as text synchronization we need
00:09:22.500
to keep to keep the test text synchronized from the editor into the server state
00:09:29.760
right after we receive that notification then the editor is going to want to compute features for that file you just
00:09:35.279
open so that they're available for you so you get a wave of language feature requests such as folding range one after
00:09:42.720
the other and the server has to respond to that and to close our example here if we were to edit the food.rb file and add a
00:09:50.399
method definition then the editor would notify the server that the contents of that document have changed to give it a
00:09:57.540
chance to update its internal representation and because the contents have changed then the features the
00:10:04.080
results of the language features may have also changed so we get another wave of language feature requests one after
00:10:09.899
the other and server has to recompute those
00:10:14.940
the Ruby LSP is a new language server that we're working on for Ruby uh there are two repos here on the slide one is
00:10:22.019
vs code Ruby LSP that one's the vs code extension that connects to the server and the other one Ruby LSP is the server
00:10:29.459
gem but I want to stress out the fact that the Ruby LSP gem can connect to any
00:10:34.860
editor that supports the protocol it doesn't have to be vs code and also the
00:10:40.140
server is entirely made with Ruby so if you want to follow along with the examples that I'm going to show now you
00:10:45.899
can just open the repo and and follow along but what does the Ruby LSP currently do
00:10:52.740
so a few other features we currently support are folding ranges so these are the little icons there on the gutter of
00:10:59.940
the editor to allow you to collapse or expand code document highlights so when you click on
00:11:06.360
a ruby entity it highlights the other occurrences of that entity in this case the name instance variable
00:11:13.320
RoboCop Diagnostics so if you make a violation it will surface that in the editor and it also supports quick fixes
00:11:20.459
through code action so you can fix that directly in the UI we have hover for displaying uh
00:11:28.820
documentation links for rails DSL methods so like has one if you hover
00:11:33.839
over that it will pop up the link to the online documentation if you click it just takes you to the website
00:11:42.000
we have a few inlay hints so like for example if you rescue without specifying a specific error class the default
00:11:48.779
implicit behavior is to rescue from a standard error and so the Ruby LSP will tell you that with an inlay hint
00:11:56.220
we have Auto formatting so if you save the file with formats on Save and fixes
00:12:01.500
your violations and finally the last example is a format
00:12:06.959
on type so in this case we're adding an if statement when you break the line it automatically closes the end token for
00:12:13.560
you so let's see how it actually works and
00:12:19.200
how we Implement these types of features in Ruby as a reference they have of the
00:12:25.260
overall architecture of an LSP it is an infinite Loop that keeps reading Json
00:12:30.540
requests from the standard in pipe and then based on that method attribute which is the feature that we're trying
00:12:36.720
to compute we have to decide what to do so depending on which type of request you got you need to Define what it means
00:12:44.100
to execute that request we need to run that request and then after we uh have a
00:12:49.620
result we have to write that result as Json into the standard outpipe which
00:12:55.200
returns that to the editor so that it can serve the features like any language server the Ruby LSP
00:13:02.399
has a few different responsibilities that it needs to account for I just want to call attention to what we're going to
00:13:08.160
be taking a look at so it does text synchronization as mentioned in a previous example it needs to parse Ruby
00:13:15.959
files it implements positional requests where the cursor position matters so
00:13:22.440
like however you need to serve the request based exactly on where you're hovering in the file and it also
00:13:28.860
implements non-positional requests which are features that are computed for the entire file at once and we're going to
00:13:35.880
be focusing on those non-positional requests more specifically we're going to be focusing on folding range so again
00:13:43.380
these are the little icons next to the line numbers that allow you to expand our collapse code and this feature is
00:13:50.339
computed once for the entire file so you find everywhere you need to you want to
00:13:56.160
fold your code once and you return that to the editor and then the edit populates those icons for you
00:14:02.880
so how do we Implement something like floating range as a reminder this is what the request
00:14:09.720
looks like we get the folding range method and the only parameter we get is
00:14:15.839
the URI of the document so by the by the time we receive this request the server
00:14:21.180
already needs to have an internal representation of that document because you don't get the text content of it so
00:14:27.899
it needs to already know what is the content of that file in what the editor expects back is a
00:14:34.019
list of ranges and the bare minimum to compose a range is the start line the
00:14:39.120
end line of the range and the kind which can be region for generic parts of the
00:14:44.459
code comment or Imports which in case of Ruby would be more like requires
00:14:52.079
so if you take this post class here we can visually inspect it and we would
00:14:57.899
probably want to realize that we want to be able to fold it in the method definition and in the class definition but when we're synchronizing texts from
00:15:05.519
the editor into the server we're only dealing with raw text it's just a string with all of your Ruby code inside and
00:15:12.540
that's very difficult to analyze so we got to be able to go from the raw Ruby code into something that has a little
00:15:18.180
bit more information for us we want to be able to assert what are the Ruby structures and entities present in that
00:15:24.779
string of Ruby code so for that we're going to be parsing the file into an abstract syntax tree so that we actually
00:15:31.380
have an object that describes everything that's present in the file and then the last step in analyzing and
00:15:38.639
implementing these requests is to go through everything that is defined in the file so Traverse the AST and then
00:15:45.000
perform some analysis in the case of folding range we're going to be collecting the ranges where we want to be able to fold the code
00:15:53.220
so what is parsing unfortunately I don't have enough time to go into a lot of detail into what is parsing if you were
00:16:00.480
at rubyconf mini a couple of weeks back Kevin Newton gave a great talk on syntax
00:16:06.000
tree which is the gem that the Ruby LSP is based on and explained parsing in a little bit more detail than I will go
00:16:12.480
into but in very basic terms parsing is the process to go from the raw string of
00:16:19.800
Ruby code into an object that describes that file in terms of the Ruby
00:16:25.920
structures that are present in it so it's a kind of like a linked list tree structure that describes describes
00:16:33.000
everything that's present in that file so we have in this case in our post class we have a class node as the entry
00:16:38.699
point of our AST with two child nodes the post constant in the body of the
00:16:44.100
class inside of that body we only have a method definition with two child nodes the title identifier and the body of the
00:16:51.060
method and finally inside of it we only have have the string rubyconf so we want to be able to take that ASD
00:16:58.199
take that object representation go through everything that is defined in our file and then identify the places in
00:17:04.380
which we want to be able to fold the coach so we want to be able to find the class definition and the method
00:17:09.780
definition node which means we need a way in which we can take that ASD and go through
00:17:16.199
everything that is defined in it study all of the Ruby structures present in the file
00:17:21.799
so that we can collect information about the folding ranges we want to be able to
00:17:26.900
return to the editor and there are multiple ways in which you can Traverse an ASD and go through everything that is
00:17:32.820
defined but we're going to take a look at how syntax 3 allows you to do it out of the box which is how the Ruby LSP
00:17:38.580
does it by using the visitor pattern if we were to implement it we would
00:17:44.280
begin with a parent class visitor for which all of our non-positional requests
00:17:49.799
will inherit from and the entry point of our analysis is the visit method which
00:17:54.840
receives any type of node that can be present in our Ruby file so it could be class definition a method definition
00:18:00.900
string doesn't matter and we invoke accept on it passing to it the current
00:18:06.480
instance of the visitor in order for this to work every single type of node has to implement the accept
00:18:12.960
method and the implementation is analogous to each one of them you take
00:18:18.120
the visitor and you invoke a specific visit for that node type on the visit on
00:18:23.880
the visitor sorry so in this case we have a class node we're going to be invoking visit class
00:18:28.980
if it was a constant node we would reinvoking visit cost the idea is that instead of having a big case statement
00:18:36.600
where you do when it's a class node do something if it's a method definition
00:18:41.700
node do something else the idea is that the visitor asks the node hey I don't know which node type you are can you
00:18:47.940
invoke the right method on me that knows how to process your node type that's basically the whole idea of the double
00:18:54.120
dispatch pattern here and to finalize our implementation to make sure that we can actually go
00:18:59.940
through all of the nodes that are present in the tree we just need to Define what the default behavior is for
00:19:05.880
those node specific visits and they all default to the same thing which is taking each one of their child nodes for
00:19:12.960
that particular node we're processing and then visiting each single one of them and it may not be clear why this
00:19:20.220
allows us to go through the AST so I have a bit of a visualization here on the right we have our post classes AST
00:19:27.000
and on the left the visitor code that is currently being executed
00:19:32.160
so we would begin visiting from the top the entry point of our AST the class
00:19:37.860
node visiting it invokes accept on it which then triggers the specific visit class
00:19:43.919
method back on the visitor and that defaults to just visiting all of the child notes so that moves us to
00:19:51.120
visiting the cost node which then invokes accept on that node triggering visit cost back on the
00:19:58.020
visitor which again defaults to visiting all of the child nodes but cost doesn't
00:20:03.299
actually have any child notes so we're actually done with that side of the tree and we can move on to the next child of
00:20:09.059
the class node the body node and then again we invoke accept on it which triggers the visit body
00:20:16.760
method in the visitor which defaults to going to the child node so it moves out moves us to the method definition node
00:20:23.700
and we continue doing this until we have exhausted the entire AST and been to
00:20:28.980
every single node every single Ruby structure present in our file
00:20:34.020
so the idea here is that you can with this pattern you can separate the traversal logic from the actual request
00:20:40.980
logic and it's going to be a little bit clearer when we get down to implementing uh folding range but it's kind of like
00:20:46.620
in innumerable how you don't care about how each is implemented you just care about the specific logic of the block
00:20:53.220
that you're passing to each and with the visitor we are ready to
00:20:59.100
implement voting range to do it we're going to be subclassing the visitor so that we get all of that
00:21:04.620
visiting behavior from the parent and we're going to initialize it with the AST for the current file and a list of
00:21:11.580
ranges where we're going to be accumulating our results so we're going to be going through the Ruby structures and accumulating information in that
00:21:18.539
array running the request means visiting the ASD so going through everything that is
00:21:24.900
defined in the Ruby file and then returning the list of ranges that we found along the way
00:21:31.020
and the last step in order to implement the folding range request is we need to be able to associate logic to when we
00:21:39.179
find a specific node type we want to be able to fold the class definitions and the method definitions and for that we
00:21:46.080
can override those node specific visits in the request implementation so we know
00:21:52.919
that this method is only invoked when we find a class node and so that's one of
00:21:58.140
the nodes we're interested in folding we can take the location information for that node where it is defined in the
00:22:04.559
file and push a new range for that class definition notice that if we were to stop right
00:22:11.580
here every time we found a class node while going through the Ruby file we would immediately stop our analysis
00:22:17.940
because we're overriding the default behavior from the parent which was to go through every child node and visit them
00:22:24.000
but we can easily get that back by invoking super and I really like the flexibility that this pattern gives us
00:22:30.539
because we can decide if we want to go through the child nodes first or process the current class definition first we
00:22:38.280
can also decide to stop the analysis if we find a certain type of node with a certain type of content so I really like
00:22:43.860
the flexibility of the visitor pattern in this case and finally to implement the method
00:22:50.159
definition folding it's completely identical with the only caveat that you need to override the
00:22:56.700
method that is related to uh to method definition nodes so we overwrite visit def but the body of the
00:23:04.080
implementation is just the same we just take the location of that node in the file and push new range for it
00:23:10.500
and that already allows us to completely fold our post class here which was our
00:23:15.659
example that we were trying to implement the way all of these non-positional
00:23:20.700
requests work in the Ruby LSP is the same while you're typing we are synchronizing your text edits from the
00:23:27.480
editor into the server and once you stop like once the debounce runs off on your typing then we parse that file and pass
00:23:35.820
along the AST to every single non-positional request and they're all implemented with visitors so basically
00:23:42.240
they all subclass the visitor parent class and they just override the node
00:23:47.940
specific visits for whatever they're interested in so they're all very similar in their implementation
00:23:55.140
I want to invite everyone to try out the Ruby LSP provide us with feedback and if
00:24:02.220
you want to contribute to it again it is pure Ruby and I'm happy to take a look with you if you're interested in taking
00:24:08.039
a a stab at contributing the features we currently support uh are
00:24:15.059
here on the screen so we we support format on Save format on type document highlight which is highlightings
00:24:21.720
occurrences for the same entity where the cursor is uh hover RoboCop
00:24:27.539
Diagnostics quick fixes through code actions selection ranges which is also
00:24:33.780
known as smart ranges it's basically expanding and and reducing a selection
00:24:38.880
based on the Ruby code semantic highlighting which is highlighting your file consistently
00:24:45.120
according to Ruby's understanding of it we have inlay hint document link
00:24:51.000
document link is attaching links to specific pieces of syntax so you can jump to documentation or jump to another
00:24:59.100
file uh folding range which we've seen and implemented and document symbol which
00:25:06.120
allows you to fuzzy search uh Ruby entities in the file so if you want to find a class definition or a method
00:25:13.080
definition you can fuzzy search that in the file I'm very excited about the future of
00:25:19.799
Ruby tooling we can have all of those features that I mentioned plus a lot more because we're not done implementing
00:25:26.640
the entire specification we can have interactive debugging directly in the
00:25:31.919
editor's UI thanks to the debugger adapter protocol which is the LSP equivalent for debuggers to connect to
00:25:38.700
any editor we can do gradual typing for more accurate features like go to
00:25:44.100
definition and autocomplete with complete type checking so that it's actually more accurate than without
00:25:51.480
types and it can all work cross editor thanks to the LSP Proto the LSP
00:25:57.059
specification and the DAP or in other words there is a future in which any
00:26:02.340
editor can be a fully featured Ruby IDE and we can collaborate uh getting there
00:26:20.900
you can connect with any editor that supports the protocol
00:26:25.980
oh sorry yes the question was if there's only a client for a vs code
00:26:31.279
you can connect with any editor that supports the protocol so I know some people from the community uh configured
00:26:39.659
neovim to connect to it the reason vs code there's a vs code extension for it
00:26:45.779
is because you actually need one for vs code it doesn't have like a built-in LSP
00:26:51.179
thing that you can just configure uh but you should be able to use it within neovim whatever other editor that
00:26:58.140
supports the protocol right so the question was if you could use uh Ruby LSP and solar graph together
00:27:06.679
yes you can the the way the language server works is it merges responses uh
00:27:13.559
so if you have the same response coming from different language servers the editor will merge them and in fact at
00:27:20.159
Shopify we use the Ruby LSP with survey which is also also provides you with an
00:27:25.260
LSP so if you have conflicting responses they're just going to be merged in the editor
00:27:30.900
right so the question is if the LSB supports uh being connected to a rebel environment with with sexually executing
00:27:38.279
your code um I don't think the specification has anything that is
00:27:43.980
like that explicitly tells you about that support but I'm I'm sure you can
00:27:49.080
like try to implement something like that to provide the outer completion although I do feel like you would face a
00:27:56.520
few challenges in trying to get that for especially for large projects it might be a little bit slow uh
00:28:02.460
I'm not sure but that's an interesting thing we can explore right so the question was if format
00:28:08.220
unsafe is powered by RoboCop uh we actually support two formatters uh
00:28:13.279
RoboCop if you have that in your application is going to be the default um so yes it does hook into Robocop and
00:28:20.640
formats using it and if you don't use RoboCop then it will fall back to syntax tree which actually has a formatter
00:28:27.120
built in as well so why do we choose a shadow environment
00:28:32.640
instead of RB environment um well basically the shadow environment is
00:28:38.820
our in-house Ruby version manager so we supported that first but actually a member of the community uh made a pull
00:28:46.200
request and implemented support for RB environment so that should be working already and there's also support for uh
00:28:53.580
CH Ruby and rvm right so the question is if the client
00:28:59.580
owns the process that's running the server and how many server processes are
00:29:04.620
spawned um yes it does own it so the plugin the editor will spawn the process and then
00:29:12.299
keep a handle for the standard in pipe and the standard out pipe so that I can communicate
00:29:18.200
but it only spawns one server for your like coding session so the moment you
00:29:25.500
open in a ruby workspace it will spawn the server and it will not it will reuse the same server for all files for
00:29:32.039
everything that you do until you switch workspaces or close the editor or something like that
00:29:38.820
uh so the question is if we could integrate something like a a third thing
00:29:43.860
not only the server and the editor uh to provide error highlight or any other
00:29:49.559
features into the editor uh I think it is possible it's kind of the same thing that we do with RoboCop which is an
00:29:56.460
external tool to the Ruby LSP it should be possible to do that type of thing
00:30:02.580
okay if there are no more questions if you have any other questions and you want to uh just find me I'm gonna hear
00:30:09.120
until the end of the conference come come chat I'm happy to talk about developer experience also I did order
00:30:14.940
stickers for the Ruby LSP but they haven't arrived yet so come see me if you want a sticker once they arrive I