Zeitwerk: A new code loader

Talks

Xavier Noria

Zeitwerk: A new code loader

by Xavier Noria

The video "Zeitwerk: A new code loader" features Xavier Noria's presentation at RailsConf 2019, where he introduces Zeitwerk, the new default code loader for Rails 6. Noria discusses the inadequacies of the previous Rails autoloading mechanism and how Zeitwerk aims to resolve these issues.

Key Points Discussed:

- Introduction to Zeitwerk: Zeitwerk is a gem that provides autoloading, eager loading, and reloading of code, designed to function independently of Rails and with any Ruby project. It is built on a strict project structure where file names correspond to constant paths.

- Implementation and Usage: The basic usage involves setting up Zeitwerk by specifying the root directories of the project, allowing developers to instantly have autoloading capabilities. Noria emphasizes that this structure helps to maintain cleanliness and eases the loading process.

- Eager Loading: Emphasizing performance, eager loading is explained as a method to load all code at boot-up, ensuring that everything required is available when needed. This is particularly beneficial for Rails applications in a production environment.

- Comparison with Existing Autoloading: The session outlines historical issues with Rails' traditional autoloading that led to the development of Zeitwerk. These issues included fragile requires, potential name errors, and limitations tied to constant resolution.

- Enhancements Over Old Methods: Zeitwerk employs a module autoload technique instead of Rails’ previous methods, allowing for better management of dependencies and a reduction of issues stemming from module loading failures or overrides. This change dramatically simplifies the coding process and adheres to Ruby’s standard practices.

- Integration in Rails 6: Zeitwerk will be implemented by default in new Rails 6 applications, allowing users to eliminate old require dependencies without needing explicit requires. Noria defines steps for upgrading existing Rails applications to adopt Zeitwerk seamlessly.

In conclusion, Zeitwerk resolves many long-standing issues with Rails autoloading, simplifies development, enhances application performance, and maintains compatibility across various Ruby project structures. Developers transitioning to Rails 6 should expect a cleaner and more efficient coding experience, propelling Ruby development further.

00:00:21.890 So, this is the menu that we have before lunch. We are going to introduce what Zeitwerk is, the motivation I had to work on it, how Rails autoloads and has autoloaded since the beginning, and compare that with the new solution. We will also cover how this is integrated into Rails 6, as Zeitwerk is going to be the new autoloader by default.

00:00:34.559 Alright, so what is Zeitwerk? It is a gem that provides autoloading, eager loading, and reloading of code. Importantly, it has no dependencies, so Ruby alone is sufficient to solve these problems. Furthermore, it has been designed to work with any Ruby project.

00:00:50.610 You might have heard of it, maybe related to Rails 6, but it is an independent gem. It has been designed to work with any Ruby project, particularly gems and your own private projects. Zeitwerk can also be used by other frameworks, so it is totally generic in that sense.

00:01:21.149 Alright, let's see first how we use this gem, and after that, we will analyze it in a bit more depth. The premise to work with Zeitwerk is that you have a project structure where file names match constant paths, which is standard in Rails.

00:01:38.009 So, it follows a very conventional project structure. For instance, if you have a file named user.rb, it should define the User constant. Similarly, user_profile.rb should define the User::Profile constant. If you have a file called html_parser.rb, the default constant expected from that file would be HtmlParser, written in CamelCase.

00:01:56.250 However, if you prefer to define HTML as an acronym, there is a way to do that. You can define a custom inflector or loader that tells Zeitwerk to treat 'HTML' as uppercase for that specific case. Zeitwerk allows independent inflectors for different gems, ensuring no interference between them.

00:02:21.750 To start using Zeitwerk, you only need to set up the loader by specifying the root directories of your project, which in Rails would be the autoload paths. For a Ruby gem, this would typically be the lib directory.

00:02:34.430 After that, you just call setup, and then you are ready to go. In the case of gems, if you have a structure similar to those of a gem but do not need a gemspec, it simplifies the setup. By convention, a gem's name is typically in all uppercase, which Zeitwerk recognizes.

00:03:07.370 Now, eager loading is another feature we need to discuss. The first call is an instance method, while the second one is a class method in Zeitwerk that eagerly loads all constants. The primary point of eager loading is that Zeitwerk maintains a registry of all loaders instantiated during the application runtime.

00:03:26.960 In a Rails application with multiple dependencies, it's often convenient to load as much code as possible during boot in a production environment. For example, if your Rails application has ten gems using Zeitwerk, it will call the eager load method, ensuring that all relevant code is preloaded.

00:04:13.460 Even if your Rails application is not using Zeitwerk directly, the integration in Rails 6 ensures that any dependencies utilizing Zeitwerk will still be eager loaded. In Rails 6, you don’t need to deal with this API directly since Rails handles it for you through the standard configuration.

00:05:01.460 When it comes to reloading, you must opt-in. This optional feature serves two purposes: it acknowledges at the API level that reloading is one of several use cases for this gem, and it allows for better performance for the default case, which does not require reloading. Some applications, especially web applications, may find reloading useful.

00:05:45.420 By acknowledging the API's functionality, we can improve performance by reducing unnecessary metadata storage during the default cases when reloading isn't needed. In web frameworks, reloading typically only applies in development environments.

00:06:36.570 As for why I worked on Zeitwerk, I initially aimed to address some of the limitations and oddities of the classic Rails autoloading. While it worked well for many years, the underlying technique had certain issues that I wanted to improve.

00:07:19.100 During development, I faced personal pain points, particularly regarding the brittleness of loading requirements in arbitrary Ruby projects. I noticed that it was easy to forget requires, which often leads to confusing NameErrors. In Ruby, if you forget a require, it could work sometimes if another part of your code has already required it.

00:07:58.570 However, this behavior can lead to complications in production scenarios where the required file was not loaded as expected. I've spent considerable time navigating Rails' codebase to find missing requires, and it was ultimately a pain point that I hoped to resolve with Zeitwerk.

00:08:32.870 A specific example of this brittleness can be seen in class definitions where including a module may lead to NameErrors if the module has not been required. The issue arises because requires have global effects, which can lead to discrepancies in behavior depending on the execution context.

00:09:10.710 Zeitwerk, on the other hand, abstracts this by loading classes and modules automatically following a standard naming convention, eliminating the need for manual requires. For example, in the static site generator Nano C, rather than cherry-picking requires, we could have a structure that pulls in everything needed on boot time.

00:09:57.270 This means that developers can concentrate on writing their code without needing to manually handle requires, streamlining the programming process.

00:10:43.780 Now let’s examine how Rails autoloads. To understand the limitations of the Rails autoload technique, we should brush up on some basics about constants in Ruby. Unlike many programming languages, Ruby treats class and module definitions as constant assignments with their own unique behaviors.

00:11:12.260 For example, when creating a class/module, we are essentially assigning that class/module to a constant, and constants themselves can be changed, leading to complexities. Constants are more than just variables; they are mappings to objects that the Ruby interpreter manages.

00:11:56.000 As such, when considering a structure that involves nested modules and classes, understanding that these constants can behave differently based on their context is important. In Ruby, the interpreter uses a nested structure that influences how constants are resolved.

00:12:32.550 When looking for a constant, the resolver checks the nesting first for potential matches and falls back on the ancestors' chain if the constant isn't found directly. This distinction is crucial for understanding how Rails autoloads files using this algorithm.

00:13:19.890 When Rails encounters a constant that is not found during execution, it utilizes the autoload paths to find the file corresponding to that constant. It constructs the expected file name conventionally, typically using underscores for the constant name.

00:14:01.460 If it locates a matching file, Rails will require it and continue executing. However, this classic technique has certain intrinsic limitations.

00:14:29.600 Active support doesn't have all the context that Zeitwerk has when resolving constants, so there can be discrepancies when it may not deduce a constant's location accurately.

00:15:07.840 Additionally, if a constant can be found elsewhere, it will not trigger a 'const_missing' for the intended scope, leading to the possibility of unexpected behavior. For example, if a constant is defined in a broader context rather than where it was expected, it can create complications.

00:15:42.960 This technique is fundamentally limited, which is why the implementation behind Zeitwerk adopts a different approach. Zeitwerk incorporates the concept of module autoload, enabling a more systematic and modular approach to file loading.

00:16:15.680 This novel approach allows for lazy-loading where the system only fetches the files when they are required, allowing for a cleaner and more efficient application architecture.

00:16:54.750 Thus, instead of requiring files upfront, you can define when you'd like a certain part of the system to load, which drastically enhances performance and reduces unnecessary loading during boot times.

00:17:29.180 For instance, during the setup phase for the application, Zeitwerk scans the directories for any constants that need to be loaded and sets them up accordingly. It uses the namespacing conventions established, making sure that everything is organized.

00:18:05.800 There’s a technical challenge when it comes to explicit namespaces in that each namespace should ideally have a structure that reflects its hierarchy within the file system. If you created a class within a module, Zeitwerk knows how to find it.

00:18:47.360 The crucial point is that for every directory you define, Zeitwerk translates that directly into its namespace structure, enabling easy retrieval of the associated files when needed.

00:19:32.350 In conclusion, Zeitwerk not only simplifies the loading of Ruby projects but also increases performance through its autoloading and eager loading capabilities.

00:20:13.970 As Rail 6 integrates this feature by default in new applications, it enables developers to remove cumbersome requires from their codebases. This great enhancement improves clarity and reduces friction within Ruby projects across the board.

00:20:52.700 So, as you consider upgrading your existing Rails applications, the transition should ideally be straightforward. You'll have to ensure that your classes and modules are well-structured to minimize the need for manual file requirements.

00:21:30.250 The reliance on convention also means developers can focus on writing functional code without getting bogged down in the intricacies of managing code load sequences, allowing for a smoother development process.

00:22:10.890 That's it for my talk on Zeitwerk. Thank you for your attention!

RailsConf 2019