Opening Keynote: The Journey to Zeitwerk

Ruby

Xavier Noria

Opening Keynote: The Journey to Zeitwerk

by Xavier Noria

In the opening keynote of RailsConf 2022 titled "The Journey to Zeitwerk," Xavier Noria discusses the advancements in autoloading within the Rails framework, particularly the introduction of Zeitwerk, which replaces the Classic Autoloader. Noria, a core Rails team member since 2011, outlines the history of autoloading in Rails, explaining the challenges faced with the Classic Autoloader and the transition to Zeitwerk.

Historical Overview: Noria begins by explaining autoloading, which allows Rails applications to operate without requiring explicit loading of classes and modules. He reflects on the limitations of the Classic Autoloader that had been in use since Rails' inception and its evolution from Rails 6.0 to the dropping of the old implementation in Rails 7.
Technical Insight: He provides a technical deep dive into how constants work in Ruby, emphasizing that constants are tied to class and module objects, which complicates the autoloading process. The presentation describes how the Classic Autoloader struggled with constant lookups and the inability to handle the nuances of Ruby’s constant resolution effectively.
Development of Zeitwerk: Noria shares his journey in developing Zeitwerk, highlighting significant issues such as module autoload behavior and namespace handling. With Zeitwerk, he uses built-in Ruby mechanisms for more reliable autoloading. For instance, he explains how the new autoloading method is designed to work seamlessly with Ruby’s resolving of constant lookups, allowing for more efficient management of application classes.
Collaboration and Community: Throughout the keynote, Noria emphasizes collaboration within the Rails community, specifically acknowledging contributions and support from other core team members and projects like Shopify, which helped refine Zeitwerk through extensive testing and implementation efforts.
Significant Outcomes: The transition to Zeitwerk has significantly reduced the number of issues related to autoloading in Rails applications, marking a notable improvement in developers' experiences. Noria concludes by celebrating the successful integration of Zeitwerk into Rails and the future it promises for Ruby programming. This progress illustrates the importance of continuous improvement in software development practices.

Overall, Noria’s keynote encapsulates the journey of refining Rails autoloading through the lens of community collaboration and technical ingenuity, aiming to enhance the Ruby developer experience.

00:00:00.900 foreign.

00:00:12.080 And now for our opening keynote speaker, Xavier Noria, who comes to us from Barcelona. He's been on the Rails core team since 2011. He is a part-time lecturer, a lifelong learner, and a Ruby Hero Award winner. He's done tons of Rails co-work on Zeitwerk, among other things. Please help me give a warm welcome to Xavier.

00:01:02.100 All right, let's talk about autoloading. For those of you new to Rails, autoloading is a feature of the framework that allows you to program with all your application classes and modules available everywhere without having to use 'require.' This feature is also related to reloading and eager loading. Autoloading has been in Rails since the beginning, but in Rails 6.0, we shipped a new implementation, which coexisted with the previous autoloader for versions 6.0 and 6.1. This was a transition period that took us to Rails 7, where we finally dropped the previous implementation because the new one is better.

00:02:09.360 This is the commit where this happened; this is my commit dropping the support for the previous autoloader, which we called the classic autoloader. This commit is highly symbolic because there are years behind that commit, and it represents an entire journey that I would like to share with you today.

00:02:21.900 Let’s go back in time for a moment. Back in the day, constants were not very well-documented features of Ruby, and Rails didn't even have documented auto loading. Years ago, I was interested in these topics, so I conducted my own research trying to reverse engineer how things worked. As a result of that research, I was able to share some of the things I had learned through talks and an autoloading guide. This autoloading guide contained one infamous section detailing the pitfalls of the classic autoloader. Here’s a screenshot of that section, which was so long that I had to display it diagonally on the slide.

00:03:46.340 Why did this happen? Well, the technique used by the classic autoloader is based on fundamental limitations. This section was essentially documentation stating, 'This is the way it works.' It's not technically a bug, because the pitfalls arose as a consequence of the limitations of the technique itself. I would like to leverage this presentation to explain to you why we included that section about pitfalls in the previous autoloading guide. Let's do a very quick recap of the most important things we need to know about constants in Ruby.

00:04:31.139 The first thing we must understand is that constants belong literally to class and module objects. This is quite particular to the Ruby programming language because, in most languages, constants are often treated like variables whose values are protected against changes. However, in Ruby, the topic of constants is very deep. Constants belong explicitly to class and module objects; each class and module object has an internal hash table that maps constant names to values. This is a unique characteristic of Ruby's model. Furthermore, there's an API to manipulate that hash table—known as the constants API—allowing you to query a module or class object about the constants it stores, as well as to dynamically set and remove constants.

00:05:25.139 Top-level constants belong to the Object class. For instance, if you define a constant at the top level in a Ruby script, that constant is stored in the Object class. This is important to understand; it’s not just a pure identifier like a variable. It has more significance because it is stored in the internal hash table of the Object class.

00:06:19.440 When referring to a constant in code, Ruby must perform a lookup to find that constant. There are two algorithms that perform this lookup in different ways. The first is the algorithm used when the constant reference is relative. A relative constant reference does not have a '::' to the left. For example, in the source code, if we have 'admin_users_controller' and an action 'new' that calls 'user.new,' we have 'user.' Here, 'user' is a constant, not a type like in other programming languages. This is a regular constant that evaluates to a class object.

00:07:24.000 How does Ruby resolve this constant? To explain this, we must first introduce the concept of nesting. At any given point in the source code of a Ruby program, the interpreter maintains an internal collection called nesting, which reflects the nested classes and modules at that specific point. For instance, in one example, we have module A and class B. In the nesting, we can see how this structure affects the resolution of constants. In another example with class A::B, the nesting would be different because there is no module A present.

00:08:46.860 To briefly explain how Ruby looks up the 'user' constant, when Ruby checks its nesting and doesn’t find it, it looks up the ancestor chain. It checks the superclass of 'admin_users_controller', which is 'application_controller.' After checking the entire hierarchy up to 'Object', if the constant is still not found, the interpreter returns a fallback known as 'const_missing.' This fallback is called if the lookup fails. All classes and modules respond to 'const_missing,' and while the default implementation raises a 'NameError,' it is possible to override that.

00:09:30.000 The second lookup algorithm involves absolute constant lookup, which is simpler. When looking up a qualified constant (like 'Product::Job'), the algorithm checks the specified class first. If the constant isn’t found, Ruby checks the ancestor chain of that class. However, if the inner namespace is a module, Ruby manually checks Object for the constant.

00:10:05.840 Now that we understand how constant lookups work, we can see why the classic autoloader faced challenges. It adhered to file naming conventions where file names matched constant paths, with namespaces represented as directories, and defined a 'const_missing' hook.

00:10:34.00 However, 'const_missing' does not know if the missing reference was relative or qualified, which means it cannot effectively emulate how Ruby resolves constants. Even if it had that information, there were still limitations. If you attempt to autoload something and find a constant with the same name elsewhere, Ruby could resolve it in a different namespace than expected without triggering 'const_missing.' Thus, the classic autoloader worked for 15 years, allowing development without requires, but it couldn’t adequately match the way Ruby handles constants.

00:11:58.760 Another method in Ruby is 'autoload,' which allows you to specify that when looking for a particular constant, Ruby should load a specified file. The key observation I made was that Ruby executes 'autoload' when checking class and module objects. If the class or module doesn’t have the constant but has an 'autoload' defined, the execution takes place.

00:12:32.720 This realization inspired an alternative way to implement autoloading. Instead of relying on 'const_missing,' we could proactively scan the project tree before executing the application. If we found a file corresponding to a constant, we could directly define an 'autoload' for that constant to load the file.

00:13:15.760 However, I faced technical challenges implementing this concept. One major challenge was that 'autoload' relied on an internal require, meaning I couldn't track what was being autoloaded. To address this, I developed a thin wrapper around the 'require' method.

00:14:09.600 Eventually, Aaron Patterson helped to patch 'autoload' in Ruby, which allowed me to monitor the autoload process as originally intended. This patching unified the handling of required files, allowing us to track when files were being loaded. And so, we moved forward.

00:14:48.020 The next challenge was how to implement reloading, as Rails users often use reload during development. Reloading would reset the application's state, but since 'autoload' performs requires, if a required file was previously executed, reloading would not trigger it again. I had learned from my experiences with other languages that you could manipulate the loaded features collection by removing files from it. This discovery allowed me to enable reloading functionality within the new autoloading system.

00:15:19.920 Another blocker I faced was the need to remove autoload definitions upon reloading, as there was no API for this. Fortunately, it turned out that if you remove the constant, the autoload definition is also gone, allowing us to keep the system clean.

00:15:58.020 The next hurdle was handling implicit namespaces—those that exist without a Ruby file defining them. Rails automatically defines a module for a directory, but for my implementation, I needed to set an autoload when a directory was found to avoid issues later. This posed further technical challenges.

00:16:28.300 The last major blocker was the presence of explicit namespaces. When a namespace is defined in a file, such as 'Hotel' in 'hotel.rb,' Ruby must respect this definition during execution. The challenge was updating these namespaces during the lookup without breaking functionality.

00:17:02.700 Eventually, I received invaluable support from Rafael Francois from the Rails core team. During a hackathon at Shopify, we discussed how to tackle some of these challenges, leading to new ideas and approaches.

00:17:39.600 Rafael shared with me the idea of using TracePoint—a tool that allows defining callbacks for certain events during program execution. By using TracePoint to trigger actions when a class or module is defined, I could manage autoload definitions dynamically, promoting better integration.

00:18:09.300 Despite its reputation for performance overhead, TracePoint showed no measurable difference during tests, which reassured me that the implementation was on the right track.

00:19:05.000 I also realized that I needed to move away from the idea of an implementation directly confined to Rails and create a gem that could support coexisting autoloaders throughout various Ruby projects. This external gem would allow Rails to integrate seamlessly without causing braking changes.

00:19:42.780 At this point, we had an independent gem and an integration within Rails. However, there were still transitions for users. We decided to ship both the new and classic autoloaders, letting users migrate at their own pace while providing them with a fallback option if they encountered issues.

00:20:34.500 As we prepared to launch, I emphasized the importance of maintaining the functionality of the classic autoloader to ensure user peace of mind and ease of migration.

00:21:06.600 With the new autoloader, we aimed for it to run seamlessly in applications while providing thorough guides and documentation to help users transition. App developers could easily adopt the new approach without risking the stability of their existing Rails applications.

00:22:05.000 On February 25, 2019, the new autoloading framework was officially announced, and I was thrilled to see the community’s positive response. This was just the beginning of our journey into improving the way constant lookup and autoloading works in Rails.

00:22:28.170 As we witnessed users embracing the changes, we started to see significant reductions in issues related to constant lookups. Many users migrated their projects effortlessly to the new system, finding it much cleaner and easier to manage their dependencies without dealing with prior pain points associated with the classic autoloader.

00:23:38.720 From this project and its successful integration into Rails, there was a stirring improvement in the user experience, as developers reported that it just works. I monitored feedback, and it was evident that libraries and frameworks outside of Rails also began implementing similar structures, ultimately benefiting the broader Ruby ecosystem.

00:24:15.430 By sharing my journey here today, I aimed to engage with the Ruby community, to highlight the power of collaboration, feedback, and ultimately, problem-solving. This is what drives innovation in open source and fuels our collective advancement in software development.

00:25:02.200 I want to wrap up by emphasizing the importance of open dialogues within our community. Our discussions and feedback were crucial in developing this new tool, showing that together we can tackle even the most challenging issues.

00:25:40.600 It's essential to remain respectful in these discussions, especially as we navigate the intricacies of enhancing our tools and frameworks. When in conversation about proposed changes or upgrades, it is crucial to appreciate other perspectives and experiences, ultimately promoting a healthier open-source environment for everyone.

00:26:26.640 As I conclude, I invite everyone here to continue participating in these dialogues, sharing your own experiences and solutions, and collectively pushing our community forward into new horizons.

00:26:59.100 I appreciate your attention today, and I look forward to collaborating toward making Ruby and Rails even better for everyone.

RailsConf 2022