Talks
Building Homebrew in Ruby: The Good, Bad and Ugly
#homebrew
#ruby
#command-line-interface-cli
#dependency-management
#developer-experience-dx
#open-source
#bundler
Summarized using AI
Building Homebrew in Ruby: The Good, Bad and Ugly
by Mike McQuaidIn the presentation titled "Building Homebrew in Ruby: The Good, Bad and Ugly" by Mike McQuaid at RubyKaigi 2019, the speaker discusses the development and challenges of Homebrew, a widely used package manager for macOS, built in Ruby.
Key points from the presentation include:
- History and Development: Homebrew was initially created in Ruby to ensure no external dependencies were needed on macOS. The speaker highlights the journey and evolution of Homebrew over the past ten years.
- Version Control Challenges: Homebrew's reliance on the Ruby version provided with macOS led to issues, particularly when starting on outdated versions like Ruby 1.8, highlighting the difficulties of maintaining compatibility and transition to newer Ruby versions.
- Adoption of Tools: The speaker discusses the utilization of tools like Bundler for managing dependencies and RuboCop for enforcing code style. This reflects a move towards a more structured development environment that enhances both developer and user experiences.
- Unique API Structure: Homebrew opts not to provide a Ruby API, instead keeping its command-line interface as the main interaction point, which contributes to consistent functionality and stability.
- Performance Optimization: Emphasizing efficient startup times and tailored command performance is crucial given Homebrew’s scale of users, with a focus on reducing delays that could hinder user experience.
- Future Collaboration: The talk touches on potential collaborations and exchanges of ideas with related projects like CocoaPods, indicating an openness to learning from other package management methodologies.
The presentation concludes with a recognition of the community's contributions to Homebrew's success, encouraging involvement and questioning to enhance the collective knowledge and functionality of the package manager. McQuaid expresses gratitude towards the Ruby community for their support in Homebrew's ongoing journey.
00:00:00.030
Hello everyone! Can you hear me up there? I'm going to talk today about Homebrew and how we've built it in Ruby.
00:00:07.020
I'm going to ask for quite a lot of hands in the air because it helps me be sure that you're listening and provides feedback. Plus, it prevents you from falling asleep.
00:00:15.570
So, is anyone here using Homebrew? Awesome! Has anyone ever submitted a PR to Homebrew? Was it merged? That's sad, and it’s probably my fault. Does anyone like our new logo? Can you tell the difference? It's really different from our old logos.
00:00:34.110
In fact, who thinks the new logo is on the left? And who thinks the new logo is the one on the right? There was a designer who worked on Homebrew who explained that the glass on the left isn’t a real beer glass, whereas the one on the right is.
00:00:51.180
So, I'm Mike. I've met some of you already, and if I haven't met you yet, feel free to come and talk to me any time. I’ll be here until Saturday night.
00:01:10.860
I've been working on Homebrew for ten years now, which is quite exciting. I’m the project leader, and I’ve been at GitHub for just over five years, working on improving GitHub for open source.
00:01:22.500
I’m on a team working on some fun secret stuff that will ship in the next few months. If you're interested in talking about GitHub, open source, or Homebrew, then give me a shout. I've got my email address there, and feel free to email me—I'll reply unless you're being mean.
00:01:49.079
My coworker convinced me it's generally better to not reply and avoid arguments with people on the internet. So, feedback is always welcome.
00:02:02.219
As I mentioned, I want to talk today about Homebrew and Ruby.
00:02:07.920
You may or may not know that the first version of Homebrew was written in Ruby right from the outset. A big part of the reasoning for that was that Ruby was one of the languages included with macOS.
00:02:19.640
Max Howell, the creator of Homebrew, wanted to build something that didn’t require any dependencies to run on macOS. He had played around with several different languages and, much like myself, fell in love with Ruby, seeing it as a good fit for this project.
00:02:33.769
However, because I am Scottish, I can't start with nice positive things; we need to start with the negatives.
00:02:41.060
The first problem Homebrew has is a result of our desire to use the Ruby that is provided with macOS, which means we don’t really get to control our own version.
00:02:53.450
So, who here is running Ruby 2.6 in production at the moment? You lucky people! Ruby 2.6 is great—there’s lots of good stuff in there.
00:03:05.090
Unfortunately, until 2016, Homebrew had to work with Ruby 1.8. Even until earlier this year, our installer script, also written in Ruby, had to support Ruby 1.8.
00:03:15.709
Is anyone here still running Ruby 1.8 in production? A couple of people. Shout out to those at the back.
00:03:22.010
Ruby 1.8 is great, to be clear, but you know it’s 2019 now. We’re all meant to be in flying cars and using Ruby 2.x versions.
00:03:27.650
Currently, we are running Ruby 2.3, which is the version that ships with macOS. This is a step up from 2.0, which was the previous version we were using, and it elevated us from 1.8, the version before that.
00:03:45.859
And if you run ‘/usr/bin/ruby’ on macOS, that’s the version you get. We try to use this system version where possible.
00:03:56.060
If you have a sufficiently modern version of macOS, you can benefit from that; however, where you don’t, we have our solution: portable Ruby. This is a binary build of Ruby that we developed to avoid hard-coding paths anywhere.
00:04:20.659
You can install it anywhere on your macOS system, and it will behave nicely and just work.
00:04:26.510
Again, we still prefer to use the system stuff where we can.
00:04:31.640
Unfortunately, there are some quirks associated with this. For instance, people like me have to remember to compile and build this to ensure version compatibility, which can be a bit of a pain.
00:04:48.560
There’s also some weird stuff regarding gem installations. If you install gems with one version of the portable Ruby and then remove that version after upgrading macOS, they don’t always play nicely together.
00:05:10.640
For a while, frozen string literals have been an area of frustration for me, alongside some bugs that were eventually fixed. While I could request inclusion of these fixes, who knows if Apple will actually do it.
00:05:38.960
So for now, I’m living with having several comments at the top of my files for frozen string literals.
00:05:50.390
One of the nice aspects of being within the Ruby ecosystem is that although we are unique and operate in our own space, we rely on many tools that are commonplace.
00:06:06.720
For instance, we use IRB. If anyone here uses it too, please raise your hand. We also utilize the ‘ron’ gem for generating man pages.
00:06:27.400
We also work with ‘rubyprof’, a useful profiler, among other things. We’ve created small wrappers around these tools because of the way Homebrew operates.
00:06:40.220
Since Homebrew runs its own Ruby processes, everything has to be initialized with a variety of environment variables, initially set by bash, followed by executing the Ruby processes in the expected manner.
00:07:11.900
Additionally, who has heard of RSpec? Many of you, I see. Does anyone use Bundler’s standalone install mode? This was a recent revelation for me!
00:07:27.400
While we now use Bundler to install our gems, we previously had a convoluted process of people downloading packages, unzipping them, and adding them to GitHub, leaving us without version control. The initial intent was to ensure that Homebrew users didn’t have to know about Ruby or the process of manually installing gems.
00:08:24.440
Consequently, all Homebrew users should be able to install software seamlessly. It’s only when developers delve deeper into Homebrew or create a formula that they need to be aware of these technicalities.
00:09:00.290
I also learned about ‘gel’, a new tool that allows bundler to run faster, which I’ve recently started to explore to improve our speed.
00:09:12.440
We really want to minimize any manual installation burdens for users, as this can slow down the invocation of the Homebrew process.
00:09:28.959
Ultimately, improving the user experience is paramount, and while we previously had one gem file, we’ve managed to reduce it down to two gem files in our repository.
00:09:57.600
This includes all the necessary runtime dependencies, plus various development tools.
00:10:17.600
Through Bundler, we now generate a file that configures these dependencies seamlessly, without users needing to worry about runtime overhead or gem installations.
00:10:40.820
I should highlight that in 2019, Homebrew operates as a command-line application rather than a web application, which emphasizes the necessity for fast startup times.
00:11:06.920
For us, a quick initialization is essential, especially considering how often command-line tools are run.
00:11:30.220
Unlike Rails projects where overhead isn’t excessive, we need to prioritize speed to create a more efficient user experience, which is why we avoid checking in binaries.
00:11:56.829
Native extensions that require compilation do not go into the repository, which simplifies our gem management. Thus, for user-level operations, we ensure that no gems with native extensions are necessary.
00:12:30.149
For development tasks that require native extensions, Bundler will facilitate installation. This process is an initial setup that is easily manageable after the first run.
00:12:50.919
Thanks to this approach, we can now use ActiveSupport within Homebrew, with features like string presenting. Although it may not be to everyone’s liking, I appreciate these tools.
00:13:21.600
Homebrew's unique structure means that we can utilize many gems in a way that minimizes friction, allowing us to develop with cleaner implementations.
00:13:44.540
Ruby-style coding has significantly improved our approach to development, making upkeep easier and coding cleaner.
00:14:13.770
A significant shoutout goes to RuboCop, which we’ve integrated widely to automate style checks. When someone runs ‘brew style’, it checks for discrepancies in our codebase.
00:14:41.010
By running checks through RuboCop, any issues can be identified by the users before they submit a pull request, ensuring consistent code quality.
00:15:05.320
As our files are within a hierarchy we control, we can integrate tools effectively so that users can reap the benefits without any additional setup.
00:15:37.080
For instance, our developers can rely on IDE integration to get immediate feedback, which facilitates an efficient workflow. This makes the coding experience smoother and quicker.
00:16:03.780
While maintaining a focus on developer experience, it allows us to streamline the contributions and reduce the manual tasks involved.
00:16:25.590
One unique aspect of Homebrew is that, despite its Ruby foundations, we intentionally do not provide a Ruby API. If you're writing a gem or application, you cannot directly interface with Homebrew through Ruby.
00:16:50.730
We have had attempts at establishing Ruby APIs, with some projects that resorted to monkey patching classes and modifying Homebrew indirectly, but this is not sustainable.
00:17:26.309
Our intended API for Homebrew is the command-line interface itself. We maintain stable outputs, treating them as an API, which respects the functional behaviors of our commands.
00:18:03.350
When you run commands like ‘brew search’ without arguments, it lists the installed formulas on your machine.
00:18:42.490
While this may seem odd, there are enough scripts on the internet that rely on this behavior, and we've opted to keep it intact to avoid breaking functionality for users.
00:19:20.389
We do provide APIs such as ‘brew info’, which returns a machine-readable version of the formula file when you pass a JSON argument, allowing for integration with other tools like jq for JSON parsing.
00:19:50.249
Additionally, Homebrew operates online through a static JSON generated by GitHub pages, allowing users to access package information through API endpoints without server maintenance.
00:20:04.780
Regarding performance, we prioritize startup time within Homebrew. Commands like ‘brew shell’ quickly produce output for sourcing in your terminal, which is essential for rapid operations.
00:20:43.020
Such performance is crucial as, with a million users, even a delay of 0.1 seconds can impact overall efficiency.
00:21:09.680
Thus, we constantly assess our performance with benchmarks to refine our commands, so they are efficient and swift.
00:21:40.750
Homebrew’s DSL for formulas allows users to create recipes easily with a simple syntax, making it highly readable and approachable.
00:21:54.369
The formula design is gratifying in its simplicity compared to other package managers I have used over the years. There's less overhead in formulation.
00:22:20.800
Recently, many have been influenced by this model, and other package managers have started adopting similar styles, but the Ruby syntax remains distinct in its beauty.
00:22:45.200
So, I think Homebrew today is a product of the Ruby community and its contributions since inception, leading to success and further development.
00:23:10.940
I’m thankful to everyone here for the roles they have played in Homebrew’s journey and their contributions to the community.
00:23:31.620
Now, I'm going to open the floor to questions.
00:23:48.050
Have you checked CocoaPods, compared how it works with Homebrew and any decisions they made in relation to package management? Have you collaborated or discussed in any way?
00:24:59.300
That's a great question! We have exchanged ideas with CocoaPods regarding managing package managers effectively and identifying pain points that come with it. Although I have limited experience with CocoaPods, I recognize its popularity and well-received nature.
00:25:57.390
Their approach seems inclined towards manual installation steps that involve deeper integration with Ruby, which can be cumbersome for users who just want to use the packages without diving into Ruby.
00:26:36.730
Nonetheless, CocoaPods has made strides towards compatibility with Homebrew to allow installations!
00:27:12.850
As for Homebrew on Linux, our team has taken steps to ensure its use while still needing to continue taking advantage of system-level tools already provided by the operating system package manager.
00:27:53.440
In summary, we adapt to fit the functionality of core system tools while providing users with the most seamless experience possible.
00:28:34.920
Thank you very much for having me!
Explore all talks recorded at RubyKaigi 2019
+69