00:00:00.240
Hello RubyKaigi! I started learning Ruby and Rails in 2012.
00:00:02.800
As one does, I created my first Rails application with 'rails new' and ran 'bundle install' to ensure everything was in place. But what good is an empty Rails application? So, I created a User model with two string attributes: name and role. After the generator was done, I opened the 'user.rb' file with excitement and found an empty class definition. That was not what I had expected.
00:00:15.040
I thought something had gone wrong and that the name and role fields I had specified had not been generated since they were not in the User class definition. It took me a while to understand that Rails, or rather Active Record, did things in a slightly different way from other ORMs. While most ORMs would have static attributes and methods that map directly to database columns, Active Record actually discovers your database columns at runtime and generates the required attributes dynamically. This was a fundamentally different and effective approach to what an ORM could look like, leveraging the dynamic nature of the Ruby language.
00:01:00.879
However, it was still missing a large piece for me. As a developer, I would never know what attributes my models had just by looking at the model file itself. This made understanding code written by others or even my earlier self really hard. It also complicated reasoning about the code, as the inner workings of what Active Record was doing were opaque to me. It made me think that some magic was happening in the background. Nine years later, I now know that there is no magic involved. What seemed magical was really just the power of the Ruby language and its ability to represent various metaprogramming techniques or domain-specific languages very effectively.
00:01:54.560
So today, we will be talking about DSLs with the aim of demystifying them. Hello again, my name is Ufuk Kayserilioglu, and I'm an engineering manager on the Ruby and Rails infrastructure team at Shopify. You can reach me on various social media platforms using the handle 'paracycle'. Now, what is a DSL? DSL stands for Domain-Specific Language, which is a programming language specialized for a particular application domain. This contrasts with the general-purpose programming languages that we use every day, like Ruby or JavaScript. In the previous talk, Martin gave a good example of a DSL: regular expressions.
00:02:24.160
The DSLs we write in Ruby leverage Ruby's metaprogramming capabilities. Ruby itself is a general-purpose programming language, but it supports very rich metaprogramming techniques. This means you can write code in Ruby that generates other Ruby code at runtime. Any method in Ruby can perform an `include` or `extend` on a class or define methods dynamically. Using these techniques, we can easily write domain-specific language primitives in Ruby. In fact, many common Ruby gems that you use daily make use of DSLs in one form or another.
00:03:06.720
For example, the Gemfile syntax that Bundler uses is a Ruby file with seemingly new instructions like 'gem' or 'group', etc. However, these are just methods implemented by Bundler in Ruby to perform higher-level operations. Rake is another example of a DSL that most of you are probably using. Instructions like 'namespace' and 'task' in Ruby files are normal Ruby methods implemented by Rake as a domain-specific language. Another case is RSpec. All the 'describe', 'context', and 'it' instructions in RSpec are again plain Ruby methods that set up and execute your test cases properly. If you're using Rails, you're constantly using DSL patterns as well, from defining routes to configuration, from defining associations to defining helper methods in controllers. Rails provides a very rich domain-specific language for building web applications.
00:04:49.520
So why are DSLs so popular? There are actually many benefits to providing a DSL as an interface to consumers of your code. The primary benefit is that you prevent users from writing boilerplate code. Can you imagine having to write all the code to define an association in Active Record for every association over and over again? Secondly, it creates a more natural API that your users will learn and remember more easily. Developers tend to focus more on what they want to do rather than how they want to do it. For instance, developers want to declare that there is a 'belongs_to' association rather than teach their models how that association should be found.
00:05:41.440
Now that we know what DSLs look like, let's explore how we can build something like that. Imagine we're writing an application that processes sensitive data that needs to be encrypted as soon as it is populated. We want a sort of an encrypted version of an accessor, encapsulating the logic to implement it in a single place. In this case, that will be an 'encryptable' module. The way we want to use it is, for example, to declare a 'CreditCard' class that includes the 'encryptable' module. This module would grant us access to the 'atter_encrypted' method, which we can call with a symbol representing the name of the attribute we want.
00:06:32.480
Let’s see how we can implement this. First, we’ll create the 'encryptable' module that will provide this DSL. The first step is to add an 'included' method, since the 'encryptable' module will be incorporated into our classes. By doing this, we can add class methods through the module itself as Ruby only preserves instance methods when a module is included in a class. To add the 'atter_encrypted' method on the credit card class, we will need to use the module's included hook to dynamically extend the target class with a class methods module. This module only defines one method, 'atter_encrypted', which is all we need.
00:07:56.800
The 'atter_encrypted' method is the main focus of our implementation. It takes an attribute name and performs its magic. When invoked, it first defines an accessor on the class using the name of the provided attribute. For our credit card example, the attribute name would be 'number', thus creating a 'number' getter, etc. Next, we construct the name of the encrypted attribute and dynamically define two methods: a getter for that encrypted attribute and a setter. The getter, in our credit card case, would be 'number_encrypted'. All the getter does is call the base attribute to retrieve the clear text value, encrypt it, and return the result.
00:08:55.679
The second dynamic method is a setter for the encrypted attribute. Again, in our credit card scenario, this would be 'number_encrypted=' method. This method essentially reverses the first one: it decrypts the value passed in and assigns it to the clear text attribute. Now we just need to establish how encryption and decryption should work. For this basic example, I've chosen to convert the string to and from its hex encoding, meaning there's no real encryption happening here. So please do not use these routines in your actual code for real encryption—consider this a warning.
00:09:50.160
That's it! Our DSL is implemented. Now, let’s see it in action. We create a new instance of our 'CreditCard' class and assign a credit card number to the number attribute. At this point, if we print the card number, we can see that it is correct. But we can also ask the object for the encrypted number and confirm that we receive the encrypted value back. Additionally, we can assign an encrypted value to the 'number_encrypted' attribute, and when we print the card number, we can see that it has changed.
00:10:20.360
This demonstrates that everything is functioning correctly and that our credit card class definition is nicely compact. We successfully encapsulated all the logic in the encryptable module, removing a lot of boilerplate code we would normally need. However, despite these many benefits, there are some drawbacks to using DSLs as well. Not everything is sunshine and roses. I would argue that DSLs have two major issues. The first is that they make it really hard to statically analyze a program. Since DSLs utilize metaprogramming to generate code dynamically, many elements are only available when the program is executed.
00:11:13.600
If the program is statically analyzed, this means it hasn’t run yet, and those features won’t exist. Secondly, it complicates understanding for newcomers who might not know what’s happening behind the scenes. This is why many people who are new to Ruby and Rails tend to think certain things are magical. While DSLs reduce cognitive load when declaring our classes, they make it challenging to understand what’s happening when we use the resulting code. To illustrate these points, I will refer to a gem named 'smart_properties' that provides a rich property definition API as a DSL.
00:12:01.440
An example usage would look like this: a 'Message' class including the 'smart_properties' module. Perhaps you are already familiar with the smart_properties gem. If so, please put yourself in the shoes of someone who isn’t. Can you clearly determine what the 'Message' class definition is doing? While you might guess it will create getters and setters for 'subject', 'body', and 'time'—since they’re declared as properties—will it also create predicate methods like 'body?' Moreover, what distinguishes a property call from a property bang call?
00:12:38.640
Questions like these pose difficulties for developers unfamiliar with the smart_properties gem, as well as for static type checkers. Is this code well-typed? Are any errors in its usage? In fact, the code has an error and will generate a NoMethodError exception at runtime. How could we catch this error without executing the file? If we know the smart_properties gem well enough, we’d realize it does not generate the 'body?' predicate method in the first place, allowing us to flag this error during code review. However, it’s likely that an additional question mark might be overlooked during a code review.
00:13:19.040
This highlights the necessity for an automatic verification method. This is precisely the strength of a type checker—but it also lacks knowledge about what properties do behind the scenes. Can we educate it? It turns out we can. Most gradual type checkers allow you to create files depicting portions of the code that are not visible statically. For Sorbet, this is done via Ruby Interface files, or RBI files, while for Steep or Type Prof, this is accomplished through Ruby Syntax or RBS files. I’ll provide examples for RBI in this talk, but the RBS equivalents would essentially serve the same function using different syntax.
00:14:35.360
To begin generating an 'message.rbi' file, we indicate to our static type checker (which is Sorbet in this scenario) that the Message class declares getters and setters for subject, body, and time attributes. Note that an RBI file is simply a Ruby file with missing method bodies—method bodies aren’t essential to the external interface of the Message class. If you’re familiar with C, this resembles a header file compared to the corresponding C file, where method definitions are located.
00:15:15.680
With this RBI file, our type checker can now let us know what issues exist in our original code. It would indicate that the 'body?' method does not exist on the Message class. This forward-looking approach poses another question: are we required to do this manually for every property in every class that utilizes smart_properties? If that were the case, would we need to keep it synchronized for every modification to any of these properties or their additions? This seems error-prone and fragile. However, we might be able to automate this process.
00:16:28.960
Indeed, we can, by using a gem called Tapioca. Three years ago, I joined Shopify and initiated the Reform team, which aimed to adopt gradual typing on Shopify's enormous monolith. At that time, the leading tool for implementing gradual typing on Ruby codebases was Sorbet. We began collaborating with the Sorbet team to learn how to use it effectively. By mid-2019, we could successfully run Sorbet on our codebase and enforced it as a mandatory check on CI. However, this initial adoption boasted low coverage of about 48%, and many files were essentially not subjected to type checking at all.
00:17:30.080
This is both a blessing and a curse of gradual typing: you can commence relatively easily, but your type coverage won't improve unless you continuously push forward. We wanted to keep advancing, so our team convened to discuss strategies for tackling the rest of our codebase. This involved understanding the barriers to our earlier adoption, where we quickly recognized that the primary impediment was a lack of static artifacts, like those I showcased from many DSL usages in our codebase. This gap was acutely problematic because our monolith is a massive Rails application.
00:18:43.200
However, Rails DSLs were not our only issue. We encountered many smaller DSLs arising from various gems, alongside some DSLs created within the repository. Thus, we needed an effective way to generate RBI files for all these DSL patterns automatically and in an extensible manner. By that point, we had developed the Tapioca gem to generate RBI files that delineate type exports from gems.
00:19:15.680
Since Tapioca already knew how to create RBI files, we decided to teach it to comprehend these DSL patterns to automate the production of their corresponding RBI files. We built this process as a generator pipeline where each DSL pattern is managed by a single DSL RBI generator class. This class decodes the DSL implementation and surfaces missing methods and constants within an RBI file. Each DSL generator must inherit from the Tapioca compiler's DSL base class and implement two abstract methods: 'decorate' and 'gather_constants'.
00:20:15.680
The generator pipeline is a two-step operation. First, all DSL generators in an application are discovered, and their 'gather_constants' method is invoked. Each generator is accountable for identifying the constants that utilize their specific DSL pattern. For instance, regarding smart_properties, the generator identifies all classes including the smart_properties module.
00:20:59.920
In the next phase, all gathered constants are processed sequentially and passed to the 'decorate' method of each DSL generator for further handling. The DSL generator receives an RBI tree and the relevant constant, at which point it’s the responsibility of the DSL generator to append necessary nodes to the RBI tree based on the provided constant. This is the rationale behind the method's designation as 'decorate', as a single constant might employ multiple DSL patterns, thus the corresponding RBI file may be 'decorated' by various DSL generators.
00:22:10.720
Let’s consider what a simple generator for smart_properties could resemble. The 'gather_constants' method identifies all classes that include the smart_properties module. It conducts this by an inefficient approach, examining the object space, but this is executed only once during boot time. Thus, we don’t have to worry too much about efficiency at this stage. Subsequently, the 'decorate' method utilizes the 'properties' method defined by smart_properties to determine which methods were defined and then adds a class definition node to the RBI tree.
00:22:54.080
Next, it appends getter and setter method definitions for each property associated with that class definition node. That’s all it takes to automatically generate RBI files for the missing smart_properties methods. It’s noteworthy that the RBI generator for smart_properties closely reflects how smart_properties itself might be implemented. Instead of defining a method at runtime on the class, we define a method on the RBI tree corresponding to that class.
00:24:08.640
If we execute this generator on the previously mentioned 'Message' class example, we should receive an RBI file that resembles this. Note that we include the getters and setters for all properties represented in the Message class here. We also incorporate type signatures but without typed definitions since we never included them.
00:24:45.680
The framework and numerous generators are already integrated into Tapioca, and we have successfully implemented DSL generators for many of the common DSL patterns utilized at Shopify today. Our primary focus has been on covering most of the Rails DSL patterns, as we are a significant Ruby on Rails organization at Shopify. Executing a simple 'bin/tapioca dsl' command in a Rails application at Shopify should generate RBI files for all Rails DSL usages in that application. As I’ve mentioned, we are currently heavily investing in Sorbet and thus exclusively generate RBI files at Shopify.
00:25:51.200
However, that doesn’t indicate that there are no solutions available if you prefer to use RBS instead. Poke from the Ruby core team, who has a talk tomorrow, has a repository called 'rbs_rails' that accomplishes a similar task of generating missing methods for various Rails DSLs as RBS files. I encourage you to examine it and consider contributing if you work with RBS.
00:26:58.240
So why should anyone care about all of this? As I previously mentioned, our motivation for this project was to expose runtime-defined methods to enable static analysis. We soon discovered that the same artifacts that facilitate type checkers in analyzing the codebase also enhance developer tooling.
00:27:14.080
The RBI files we generated have empowered developers to discover methods that typically have no traces within the codebase. Similarly to what Mamoru said during the open keynote today, our work has allowed for better static analysis, improving understanding and readability of the codebase too. Static analysis is not an end; it's a means to enhance your developer productivity. I want to give you a glimpse of the possibilities by revisiting my initial Rails app challenge from nine years ago.
00:27:55.840
Recall when I generated a user model, but it was empty. Let’s see if we can improve that experience. After adding Tapioca to the project Gemfile and initializing it, let’s execute the Tapioca command to generate the DSL RBI file for the user model. The command loads the Rails application, eager loads all DSL generator classes, and constructs the DSL RBI file for the user into the 'store/base/rbi/dsl/user.rbi'. By checking this generated file, we can observe that the setters and getters for 'name' and 'role' are present, along with other default attributes like 'id'.
00:29:19.920
We not only have all the methods defined for the user class, but we also receive type information for those methods. Soon, we will also be adding code comments to these method definitions, granting developers insight on why a method was generated and which DSL produced it. This will render these generated methods even more useful to developers.
00:30:00.240
You can observe this feature in action in an editor. This is a 'UserDecorator' class that takes a user instance and redefines its name and role getters. Notice how hovering over the name getter for the user instance provides full documentation about it. The documentation specifies that this name getter was added by Active Record because the users table contained a name column, reflecting the same applies for the role getter.
00:30:19.920
Additionally, if we right-click the role method and select 'go to definition', it directs us to the RBI file containing its definition along with its signature and accompanying code comments. Even though we don’t generate code comments at the moment, I wanted to mention this possibility for the future.
00:30:36.800
That’s actually very interesting. Thank you for listening to me! We will have a Shopify after-hours AMA tomorrow at 3 PM JST, and you are invited to come and ask any further questions regarding my talk or any other talks by Shopify engineers.
00:30:43.680
Here’s the link to register for the AMA. Thank you for your attention! I hope you enjoy the rest of RubyKaigi. I am truly appreciating it. I will be available in the chat to answer any questions you may have, and I look forward to seeing you around.