RubyGems
Using "modern" Ruby to Build a Better, Faster Homebrew
Summarized using AI
Using "modern" Ruby to Build a Better, Faster Homebrew
by Mike McQuaidIn Mike McQuaid's presentation at RubyKaigi 2024, he explores how modern Ruby can be utilized to enhance Homebrew, a popular package manager for macOS. With 15 years of experience working on Homebrew, McQuaid outlines Homebrew's long-standing relationship with Ruby and discusses the ways in which they are evolving together.
Key Points Discussed:
- Homebrew and Ruby's Relationship:
- Homebrew has relied on Ruby since its inception in 2009.
- Initially, installations were done via direct Ruby commands, but this evolved to the current curl | bash installation method.
- Homebrew's Formula and Cask DSL:
- Introduced the concept of formulas (which define how to install packages) and casks (for binaries and installer files).
- Demonstrated the readability of Ruby-based formula syntax and its impact on community adoption.
- Modernizing Homebrew:
- Transition from relying on Apple’s older Ruby versions to incorporating Portable Ruby, moving to newer Ruby versions like 3.1 and 3.3.
- The introduction of Sorbet for type-checking and RuboCop for code style consistency.
- Performance Improvements:
- Highlighted recent speed enhancements, such as making the 'brew list' command faster than PIP list.
- Suggested further performance enhancements through Bash for certain commands to reduce execution time.
- Security Enhancements:
- Discussions around ongoing security audits and the use of GitHub Actions for attestation in binary packages.
- Work Brew:
- Briefly described the new startup effort - Work Brew, aimed at improving how Homebrew works in enterprise environments.
Conclusions & Takeaways:
- Emphasized the importance of community contributions for Homebrew's growth.
- Encouraged the Ruby community to help improve Homebrew, focusing on its performance, security, and overall experience.
- Invited attendees to reach out for assistance or to get involved, highlighting the evolving nature of Homebrew to better support users and developers alike.
00:00:17.760
So first, put your hand in the air if you use Homebrew. Good! Put your hand in the air if you have ever contributed to Homebrew, made an issue, or a pull request or commit. Can we get a clap? Thank you! Put your hand in the air if you love Homebrew. Now put your hand in the air if you hate Homebrew.
00:00:49.039
So today I'm going to talk about using modern Ruby in Homebrew to build a better, faster Homebrew. First, some background.
00:01:03.559
This is not the right background. Put your hand in the air if you hate Windows XP. Good, it is too old. This is almost right, but this Mac version is also too old, and Homebrew does not support it. This also makes me sad.
00:01:21.680
Ah, this is better! We have the right background and we can continue the talk now. Homebrew supports this.
00:01:28.119
I am Mike McQuaid. I have worked on Homebrew for 15 years and I am the project lead. I'm the CTO of a startup called Work Brew, which I will talk a little more about later. We're doing stuff to make Homebrew better at work.
00:01:43.680
I was at GitHub for 10 years and left as a principal engineer last year. After the talk, I have Homebrew and Work Brew stickers, and I found my sticker coverbed again. I have a bunch of GitHub stickers that you can have as well.
00:02:03.479
So today I'm going to talk about three things: first, Homebrew and Ruby and our relationship together; next, Homebrew's Ruby tooling that we use and how that's possibly changed in the last four or five years; and finally, what we're doing to build a better Homebrew.
00:02:26.280
First, Homebrew and Ruby. Put your hand in the air if you knew that Homebrew was already using Ruby. Okay, some people, not everyone.
00:02:31.680
So this text is very small, so you probably can't read it right now. But this is the original first commit of Homebrew, which was 15 years ago next week exactly, which is kind of funny. It was by a guy called Max Howell, up here on May 2009, and the first commit was to this readme file where he talked about how Homebrew was going to work.
00:03:12.080
The important bit for us is that Homebrew uses Ruby. From the beginning, Homebrew has relied on Ruby, and it always utilized the Ruby that was provided by macOS. So, Homebrew and Ruby have been very closely aligned from the start. In fact, Homebrew in the very early days looked a lot more obvious that it was using Ruby than it does today.
00:03:38.239
To install Homebrew back then, there wasn't a brew command yet, nor a nice website with an installer. You just ran a Ruby command after checking out Homebrew on your disk. Now, it's a curl to bash. I think Homebrew might have been one of the first projects to adopt this. Put your hand in the air if you hate curl to bash. Yes, some people. Put your hand in the air if you love curl to bash.
00:04:08.000
Most people don't care, it seems. Installing Wget is a nice example of a simple Homebrew package. It used to look a bit like this: you would just run Ruby and pass the actual formula file directly to run that formula script with Ruby.
00:04:34.720
Who knows what a formula is already? A few people, not everyone. Nowadays, you just type 'brew install Ruby', making it easier to forget that you're using Ruby with Homebrew at all.
00:05:05.320
Now, I'm going to talk a little bit more about the formula. This is the DSL (domain-specific language) that we use for writing packages in Homebrew. In my opinion, this is one of the biggest reasons that Homebrew has become so popular compared to maybe APT or MacPorts or Debian.
00:05:51.919
It's very easy to write a Homebrew package, and one of the big reasons for this ease is Ruby. This is a slightly simplified version, so we can fit it all on one slide. This is a minimal example of what a formula can look like; it's valid as is.
00:06:21.639
However, if you go and look at the actual formula, you'll see there are a few more details for boring reasons I won't get into right now. In my opinion, this is extremely readable without being overly verbose.
00:06:38.880
I still think that Ruby is the best language for writing beautiful code like this. For this formula, we have 'Wget' which is the name of the class, the package, and the file on disk. This is a type of formula. We've got a description here, 'Internet file retriever'. We have the homepage where we say, 'This is where to go if you want to learn more about Wget'.
00:07:02.080
We have the URL from which it's going to be downloaded, and in this case, we're simplifying it; you would actually download it directly from Homebrew. We provide binary packages for you; I'll talk more about that later. Homebrew will download this on our CI systems; we've got a checksum to ensure that this is what we think it is, the license, and then some commands we run.
00:07:40.480
Again, if you've written a bit of Ruby, the system command will already be familiar for you. We're just passing this through; the prefix is the special place where we install the software, and then we run 'make install'. So even from this 'def install', it's very easy to see what commands are needed to install Wget under Homebrew.
00:08:06.599
As a result, I've seen other people who don't use Homebrew trying to figure out how to build a tool go look at the Homebrew formula because it's often the easiest thing to read if you don't know anything about the package manager itself.
00:08:31.680
Here we've got an example of a little test where we're saying we want Wget to dev now because we don't care what the output is. We just want to see if Google.com is up, and hopefully, if it's not, then it's just Homebrew's Wget that's broken and not Google.
00:09:04.840
Although maybe this should be ChatGPT nowadays instead. Raise your hands if you know what a Cask is. This is even less widely known than a formula. Yeah, fewer people.
00:09:59.000
A Cask in Homebrew used to be part of a separate project called Homebrew Cask, but that was merged back into Homebrew, and now we have them all in the same place. A Cask is generally related to an upstream project where the developers create the packages, binaries, or installer files and we download from them.
00:10:17.180
So it works a little differently; we're not having to create build instructions, but we're more concerned with the packages we get and where to put them and how to uninstall them.
00:10:50.160
I'm not going to go through this in as much detail because I don't think it's quite as interesting, but we have different things for the different architectures. This is for one password; we've got different checksums for the different architectures.
00:11:04.040
We're using a different download depending on the system, whether on Apple Silicon or on Intel x86. We've got the URL for downloading, the name and description similar to a formula, and we're stating that it auto-updates.
00:11:52.319
This means that this version might not always be correct because one password can update itself after you've installed it. We also indicate the Mac OS version it depends on and the app file it installs.
00:12:20.720
This is a bit minimal; the actual thing is slightly larger, but not by much. Another thing worth noting is that Homebrew now fully runs on Linux. Most packages can be installed and work nicely on Linux as well, but Casks are, for now, only available on Mac OS.
00:12:47.360
We also write Casks in Ruby. This is a nice modern example of what we're doing nowadays using Sorbet, a Ruby type checker that has come out of Stripe. We have this command; if you run 'brew docs', you can specify the arguments.
00:13:21.120
This command doesn't actually have any arguments but does have a description for help output and all this type of stuff. We also want to run a browser to go to Homebrew's website; even if you don't know much Ruby, it's quite nice and readable.
00:14:06.000
I mentioned Homebrew was initially released in 2009, back when it was macOS 10 Leopard, and it came with Ruby on the system, which was pretty cool at the time. In 2009, that was Ruby 1.8.7, which was released in 2008, so it wasn't too old and was just right.
00:14:41.600
It meant that Homebrew didn't have to figure out how to install Ruby to run on a Mac system before you could start using it; Apple did that all for us. Unfortunately, as we were relying on the Apple version, by 2016, we were still running Ruby 1.8.7.
00:15:10.960
In 2016, my face would look like this every time I had to write with Ruby 1.8. I'd find some lovely new feature released ages ago in Ruby 1.9 or something and every single time... Raise your hand if you still have to run Ruby 1.8 anywhere in 2024? Oh, just a few people. Sorry for you.
00:17:17.960
By 2017, things improved a bit. I still have my eyebrow raised because I'm not quite as happy as I would like to be. Ruby 2.4 had already been released, but Apple released Ruby 2.3 with macOS. So we could rely on this version, and it was not too old.
00:18:44.199
By 2019, however, we were falling into the same pattern again and having to rely on older versions. At the last RubyKaigi, I talked about running Ruby 2.3; does anyone still run 2.3 in production? Fewer than 1.8, interesting! By 2019, macOS finally came with a newer version.
00:20:13.360
But still, by 2023, we were stuck on Ruby 2.6.1 on my lovely Apple Silicon laptop running the latest version of macOS today. This situation makes me very sad, but we have some people at Apple we talk to, and they reminded us that the Ruby on the system is deprecated and we shouldn't be using this anymore.
00:21:13.320
We said, 'Yes, but that's not us, right?', and the rules don't apply; we are Homebrew people. But they advised us to fix the problem ourselves. So, with tears in my eyes, I had to say goodbye to macOS Ruby entirely for now.
00:22:39.600
We asked Apple for a newer version, but we didn't get what we wanted. However, we managed to get a slightly newer Ruby ourselves and moved onto Ruby 3.1.
00:23:15.520
During this conference, I heard discussions from people like Jay Hathorne and Aaron Patterson, who showcased a new profiler for Ruby. It looked really shiny, and I wanted to set it up with Homebrew, but I was disappointed to find it needed Ruby 3.2.
00:24:12.720
I thought, 'Can I get Homebrew upgraded to Ruby 3.3 during RubyKaigi?'. I was happy to get the pull request merged 48 minutes ago.
00:25:39.880
Now, this didn't get updated in the last 48 minutes. We have this thing called Portable Ruby in Homebrew. This originated back when we were on Ruby 2.3 or 2.6 because we had to support older versions of macOS.
00:26:12.720
We wanted to use the newer Ruby, so originally Portable Ruby was created to install a Ruby version compatible with all supported macOS versions. Now, with Ruby 3.1, we are moving entirely to our own Portable Ruby.
00:27:10.760
So now, if you're on Linux, you can supply a Ruby version from the system, but if you're on macOS, you're always running Portable Ruby. This is great because it gives us more control over the Ruby versions in Homebrew.
00:28:40.920
So, I've talked a bit about Homebrew's relationship with Ruby and how we moved around the versions. Now, I'm going to discuss Ruby tools used in Homebrew. RuboCop deserves the first mention because we use it widely for many things in quite cool ways.
00:30:06.920
Initially, we used RuboCop for Homebrew's internals to enforce some rules and ensure code consistency, which helped us review pull requests. Homebrew has had about a thousand contributors over 15 years.
00:30:49.760
With around 30 people now, we have a lot of work to keep up with. We try to automate as much as possible, so RuboCop has become an essential tool for that.
00:31:43.760
Over time, we built it into Homebrew. If you type 'brew style wget', it checks the contents of the file on disk against our style guidelines. We also have custom RuboCop checks that enforce ordering in formula files and a lot of them have autocorrection support.
00:32:50.360
If you're using an editor like VS Code with auto-correct enabled, it will modify things according to Homebrew's style guidelines. I encourage other open-source projects to consider using RuboCop and auto-correct to improve the quality of contributions.
00:33:39.680
'brew style' checks Homebrew's code and we run it in our CI to ensure we meet our standards. It reduces the need for internal discussions about styling rules in pull requests, shifting the focus to what rules should be defined.
00:34:17.560
Since 2019, we've started using Sorbet, the type checker for Ruby from Stripe. We initially brought it in for specific files, but we're moving to use it more extensively.
00:34:56.720
We are increasingly removing runtime checks in favor of Sorbet checks and have enabled it by default for developers. Now, we have the 'brew type check' command, which checks things via Sorbet.
00:35:46.520
We expect to make much of Homebrew type strict because we found it helps catch many errors, providing better documentation for our Ruby DSLs.
00:37:02.520
Regarding our test suite, who here has a test suite with 100% coverage? No one? One person? Okay! I'm coming to check to ensure it's legit!
00:37:50.440
Ours has around 70-75% coverage, which is decent, catching many edge cases since Homebrew has been around for a long time. It doesn't catch everything, but we're finding Sorbet really helps catch issues RuboCop may have missed.
00:38:40.720
Speaking of tests, we use RSpec for that. There was a survey outside showing many people use RSpec compared to MiniTest.
00:39:18.080
If you run 'brew tests', that's what we do there, and thanks to the parallel tests gem, we run in 10 processes, which speeds things up. It takes about six minutes on my two to three-year-old Mac.
00:40:05.680
We've achieved a good balance with the Homebrew testing system, enabling us to verify that things work without having a test suite that takes hours.
00:40:49.920
In this final part of the talk, I'm going to discuss what we're doing in Homebrew to improve things better.
00:41:12.320
In particular, I will address what we aim to enhance in the coming year, so if I'm at RubyKaigi next year, feel free to tap me on the shoulder and ask why those things didn't improve as I suggested.
00:41:57.680
The first area in Homebrew is improving performance. Who finds Homebrew too slow? Not many, but Google disagrees.
00:42:31.680
So, we talked about why it's written in Ruby. I can't help with why it might not be installing correctly. Let's discuss the slowness issues a little bit.
00:43:46.880
I decided to make a last-minute addition to my slides, as I tend to procrastinate. Initially, I included a line suggesting people love Homebrew; however, with the latest updates, we need to improve the speed.
00:44:50.720
Recently, I made Brew list much faster, and compare that to PIP list; Brew list is now 1.28 times faster!
00:46:28.080
A lot of discussions about performance optimization occurred during the conference. How do we make Homebrew much faster?
00:47:04.160
Is it porting Ruby to Rust? People are using Rust for improving the performance of their Ruby applications, or porting to C? I don't know. I've heard the idea of using Bash was raised.
00:48:10.400
Let me show you this new development. When you run Brew commands, the initial execution runs a bit of Bash to clean up the environment before executing Ruby code, which includes a lot of requires. It runs slowly for simple commands.
00:49:34.760
We aim to lessen the dependencies of each command and shift calls to Bash for commands that can be handled the same way.
00:50:20.720
You may have seen Hyperfine discussed in other talks; it's quite useful for benchmarking. It allows comparisons and benchmarks on Ruby versus Bash, and it shows how long it takes to execute each command and variations.
00:50:56.560
We've noted that for 'brew prefix', running in Ruby takes around 400 milliseconds, compared to 26 milliseconds in Bash.
00:51:46.760
Therefore, while Ruby is slow, Bash can outperform it in many command situations due to the overhead Ruby entails.
00:52:19.760
Finally, we are incorporating better security into Homebrew. We've had several security audits over the last few years.
00:52:34.760
We've published one of those audits on Homebrew's blog and are working on another that we hope to publish once we fix a few more outstanding issues.
00:53:51.760
Additionally, GitHub Actions introduced attestation, which if verified will allow you to know whether the binary package provided came from GitHub Actions and CI systems. It is not enabled yet but we hope to see it available soon.
00:54:19.760
In my work on Homebrew, I found the security aspect quite challenging, as we cannot always open-source all of it as big companies might desire.
00:54:43.720
Last year, I started a company called Work Brew focused on making Homebrew work better with tools like MDMs (Mobile Device Management) including Jamf or SimpleMDM. We're also looking to develop engineering specifically innovation features.
00:56:01.760
We have our installer, which works offline, provides portable Ruby, and sets up both Work Brew and Homebrew simultaneously.
00:56:35.760
We have a nice interface allowing you to see every device in your fleet of MacBooks, checking what is outdated and what is ready for upgrades.
00:57:17.480
You can easily upgrade a formula or deploy it across many devices. If you're interested in Work Brew, feel free to talk to me, email me, or visit our website to fill out the contact form.
00:58:06.560
There's a demo video available on our website explaining everything we are doing. Throughout this talk, I shared some insights about Homebrew and Ruby, their long-standing relationship, and how we're keeping up with newer versions.
00:58:38.960
We also discussed the Ruby tools like Sorbet, RuboCop, and RSpec that we rely on heavily at Homebrew. And finally, how we aim to build a better Homebrew in terms of performance and security.
00:59:13.360
Because this is RubyKaigi, with all of you brilliant Rubyists, I invite you to help us on our journey to improve Homebrew. If you need assistance or if our contribution documentation confuses you, please reach out, and I'll be happy to help you get started.
01:00:03.480
Thank you very much for your attention! Feel free to contact me or ask any questions via Mastodon, Twitter, or email.
Explore all talks recorded at RubyKaigi 2024
+51