How resolve Gem dependencies in your code?

by Hiroshi Shibata

In this video presentation, Hiroshi Shibata discusses the intricacies of resolving gem dependencies in Ruby code during his talk at the Euruko 2023 conference. The discussion highlights the importance of managing libraries effectively in Ruby applications using tools such as RubyGems and Bundler.

Key points covered in the presentation include:

Introduction to RubyGems and Bundler: Hiroshi explains that RubyGems is the standard package manager for Ruby, enabling users to install and manage libraries easily. Bundler, on the other hand, enhances RubyGems by providing version locking, ensuring consistent environments for applications.
Understanding Gems and Gem Specifications: A 'gem' is defined as a unit of external library software, with a 'gem specification' providing essential metadata about the gem, such as its name, version, and dependencies. Hiroshi elaborates on the process of creating and using gemspec files, which define these specifications.
Mechanics of Bundler: Bundler not only resolves dependencies but also adjusts load paths for libraries as specified in a Gemfile. Hiroshi shares how Bundler improves performance despite a potential slowdown due to its extended mechanisms compared to the traditional Ruby require methods.
Dependency Resolution: Hiroshi delves into how Bundler resolves gem versions through its unique P-graph resolution engine, which is designed to manage dependencies efficiently, particularly when faced with version conflicts. This engine iterates through potential resolutions and prioritizes performance to streamline dependency management.
Challenges in Dependency Management: The talk addresses the complexities involved in maintaining a clean and functional resolution system, especially as codebases grow with multiple gems and dependencies. Hiroshi emphasizes best practices for handling these challenges effectively.
Future of Ruby & RubyGems: He concludes by encouraging ongoing support for the Ruby ecosystem, underscoring Ruby’s commitment to being a user-friendly programming language for developers.

In conclusion, Hiroshi's presentation offers valuable insights into Ruby's dependency management system, equipping developers with a better understanding of RubyGems and Bundler's functionalities and how to navigate potential complexities. The talk inspires continued engagement with the Ruby language, symbolizing the community's strength and dedication to improving its ecosystem.

00:00:12.280 I'll be speaking at a conference outside of Japan for the first time since COVID-19, and I am happy to present my talk at Euruko. Today's topic is how to resolve gem dependencies in your code. My name is Hiroshi Shibata; please call me Hiroshi. I primarily work as an open-source software developer at Anot. I am a member of the Ruby C team and the RubyGems and Bundler teams, working on maintaining the Ruby ecosystem and its infrastructure every day. Thank you for using Ruby, RubyGems, and Bundler.

00:01:06.280 Let me introduce my company, Anot. Anot provides software products for the construction industry, and our service is built with Ruby, Ruby on Rails, and JavaScript. Anot is a supporter of the Ruby ecosystem, and they hired me as an open-source developer. I've traveled over 18 hours from Japan to be here at EUR today. Japan is a significant place for the Ruby language, and I am excited to attend in Lithuania.

00:01:35.320 I would like to introduce RubyKaigi, one of the major conferences for the Ruby language. It is a great opportunity to meet many Ruby developers and experts. You will be able to engage in discussions about Ruby and enjoy delicious food and drinks. RubyKaigi 2024 will take place in Okayama, Japan, a beautiful area with scenic views of the sea and islands. I look forward to seeing you again at RubyKaigi in May 2024. Please submit your proposals!

00:02:16.440 Now, let me proceed to introduce the first topic of my presentation: dependency resolution with RubyGems and Bundler. First, I will provide an overview of RubyGems and Bundler. A package manager like Bundler serves two primary functions. First, it offers a user-friendly interface for users to install, upgrade, and uninstall packages. Second, it undertakes the task of resolving dependencies among package names.

00:03:40.720 Additionally, a crucial feature is version locking. This functionality ensures that a consistent list of packages is available regardless of the unique processes. RubyGems is the standard package manager for the Ruby language. It is developed within the GitHub organization, consisting of two primary repositories: one is RubyGems, which is an application for hosting Ruby libraries, and the other is the command-line tool for RubyGems.

00:04:49.320 My main focus remains on the RubyGems C repository, where I work with it daily. Next, I will introduce some terminology. A 'gem' is a unit of external library software or frameworks, and a 'gem specification' is a class that defines metadata, including the gem name, version, platform, and dependencies. A gemspec file describes the gem specification and can be created manually for gem releases or automatically at the time of gem installation.

00:06:05.880 Now, let’s take a look at Bundler. RubyGems provides a user interface and dependency resolution, but it does not have the capability of version locking. Bundler provides this version locking functionality, which is achieved using a Gemfile. You may be familiar with the Bundler application. Let's look inside the Ruby libraries and how Ruby loads gems. You can check the gemspec, which is the metadata of a gem, using the 'gem specification' method with the Gemfile.

00:07:20.919 For example, the Rack gem contains various metadata, with the most important being the dependencies section. In this example, the Rack gem has four dependencies: 'minitest', 'minitest-spec', 'rack-test', and 'rake'. These dependencies are marked as development dependencies, meaning you can use Rack without needing the other dependencies. Developer dependencies are unnecessary for your application or code, as they show the metadata that RubyGems and Bundler use to resolve all the gems and versions required.

00:08:15.360 In Ruby, there is a slightly different kind of gem called default gems. Default gems are gems that can be required regardless of the Gemfile, and they can be updated with the 'gem update' command. You can check if a gem is a default gem by referring to the gemspec. For instance, you can require the RSS library and check its specification using `gem specification`. For example, when checking RSS, it returns false indicating RSS is not a default gem, but for OpenSSL, it returns true, confirming that OpenSSL is a default gem.

00:09:17.880 Furthermore, Ruby has 'bundled gems' which are automatically installed during installation without needing to run 'gem install' or 'bundle install'. This is exactly the same as your local installation of gems such as Rails or other libraries. I work on organizing the bundled gems in each Ruby release package every year. Bundler only requires gems from the load path environment variables.

00:10:00.960 However, Bundler can automatically add the gem path even if the path does not exist in the load path variables. Bundler extends the original require method of Ruby so that it injects the required paths and libraries. As a result, the require process is slower compared to the original Ruby require methods, creating a typical slowdown. Bundler has mechanisms to address this slowdown, which will be discussed in this talk. Bundler also extends various classes and methods of RubyGems, affecting the moment Bundler is initialized. For example, the extension defines the directory of the essential library in the gem specification class.

00:11:22.799 Bundler must locate the gem repositories, which can be specified in the gemspec or Gemfile. This section introduces the architecture of RubyGems and Bundler. Both have similar functionality and structure, using resolution engines for dependency management. RubyGems uses the Morino resolution engine, while Bundler uses P-graph, which is a different implementation. There are also differences in the command line interface and runtime architecture between RubyGems and Bundler.

00:12:06.639 Resolution is the process of creating a condition based on the combination of multiple libraries. The resolution engine provides a low-combination result based on library names and versions. Finally, the gem resolver is a mechanism that abstracts the necessary data for the resolution engine, providing a list of libraries to be installed for RubyGems and Bundler.

00:12:45.600 Now let's understand how Bundler resolves gem versions and updates the load path. We will use a Gemfile that specifies a single dependency on the RSS gem, which itself has a dependency on rexml. When you run 'bundle exec' with this setup, it will call the Bundler setup method to set up the environment variables and modify certain environment configurations such as the load path variables. After verifying the Gemfile, Bundler will install any gems defined and listed in your lock file.

00:14:26.680 Bundler also performs cleanup on unnecessary paths in your load path when using Bundler. If unused paths remain, those will be rejected. Bundler utilizes all directory specifications associated with names and versions as they are imported from the Gemfile. This process allows Bundler to resolve the necessary gem dependencies. In this case, Bundler should resolve and return all dependencies for RSS, rexml, and itself as specified in the Gemfile.

00:17:30.200 In the second part of my talk, we will explore how Bundler expresses the gems needed for your application and updates the necessary versions. Let’s take look at how the resolution engine, P-graph, experienced various updates and developments. P-graph is built to resolve dependencies quickly while handling the complexity of packages. It was originally developed for a different package manager, but has been implemented in many languages, including Ruby.

00:19:15.000 P-graph is designed to skip resolving conflicts in the graph efficiently, allowing it to run faster than Morino. An example of how P-graph works is when there is a conflict amongst the defined versions of gems, it gives up easily when the dependencies cannot be found due to version incompatibility. P-graph prioritizes efficient resolution over complexity, making it easier for developers to manage their gem dependencies.

00:22:47.360 This example showcases gem dependencies defined within a Gemfile. If certain dependencies have conflicting version requirements, P-graph will provide explicit exceptions to indicate that resolution cannot occur due to incompatible dependencies. As a result, Bundler must iterate through the potential resolutions, reset the process, and attempt re-routing to resolve any dependency conflicts.

00:24:41.320 As you have seen, maintaining a clean and functional resolution system for gems can be deeply complex. Not only does Bundler involve itself in managing the dependencies, but it also updates the required gems dynamically as needed. Dependency management in Ruby is challenging, especially when applications can scale into numerous gems and dependencies. This framework encourages best practices and optimal handling of library dependencies.

00:28:24.000 In conclusion, I would like to summarize the essential points from my talk about RubyGems and Bundler. I appreciate the opportunity to present today and share insights about making Ruby a better language for developers. Thank you for your attention; I'm excited for the continuing support of the Ruby ecosystem. Remember that Ruby is built to be a programmer's best friend, and I will ensure it continues to be regarded as such. Thank you all!