Keynote: What it takes to keep Ruby gems a thing

Community

Samuel Giddins

Keynote: What it takes to keep Ruby gems a thing

by Samuel Giddins

In his keynote at BalticRuby 2024, Samuel Giddins discusses the challenges and successes of maintaining Ruby gems and the organization behind it, Ruby Central. He highlights the importance of community and sponsorship in supporting the ecosystem.

Key Points Discussed:
- Role of Sponsorship: Giddins emphasizes the need for community sponsorship to support conferences and organizations like Ruby Central, noting his gratitude towards AWS for sponsoring his role.
- RubyGems Growth: The growth of RubyGems.org has been significant, achieving a 20% increase per year over the last 20 years, with staggering statistics such as 11 million unique IP addresses each month and 23.3 billion requests recently.
- Infrastructure Challenges: Giddins shares experiences with spikes in traffic, such as facing 225,000 requests per second, highlighting the necessity for robust web servers.
- Security Initiatives: He details new security features including trusted publishing utilizing OpenID Connect and enhanced gem build processes to bolster the security of the Ruby ecosystem against threats like typo-squatting and dependency confusion.
- Community Contributions: Giddins praises the vibrant Ruby community and the support from companies like Shopify and the German Sovereign Tech Fund that contribute funds for security improvements and maintenance.
- Ongoing Projects: He presents his ongoing efforts in software supply chain security, mentioning a project to track gem changes over time, and notes the significance of a recent partnership with Sigstore for increased security practices.
- Operational Practices: The talk covers the manual deployment processes and challenges faced, including maintaining zero-downtime during updates.

Overall, Giddins stresses the crucial role of community feedback and contributions in sustaining the Ruby gems ecosystem. He closes with an appreciation for the support received from various organizations and a personal note thanking his wife, reflecting on the collaboration necessary to succeed in such endeavors.

00:00:08.080 I want to start off by saying that we absolutely love our sponsors. Whatever company you work at, if there’s someone you can persuade to sponsor something—be it a wonderful conference like this or Ruby Central—please do so. It’s beneficial for everyone. Today, I’m going to talk about what it takes to keep Ruby gems a viable aspect of our community and the organization behind it. It’s a shameless plug for them since they pay my salary. I’ll give you a sneak peek at what we’ve accomplished but haven’t yet announced, as well as the things we plan to do for the rest of the year.

00:01:00.440 A little about me: you might have seen my handle online; it’s the same everywhere. I’m Samuel Giddins, and I’ve been a maintainer of Ruby gems, Bundler, and rubygems.org for about a decade now. I don’t know how that happened! During this time, I've contributed my fair share of bugs, so I apologize for those. Please feel free to laugh; I’m trying to be funny, and it helps when I hear you chuckle. My current role is the Security Engineer in Residence at Ruby Central, and I'm proudly sponsored by AWS. Thank you, AWS! My focus is on the security of the Ruby ecosystem, particularly packaging security.

00:02:00.680 Some goals for today’s talk include entertaining you all since it’s 10:00 in the morning and not everyone has had a chance to drink coffee yet. I’ll try to make you laugh, shock you with large numbers, and provide some slides you can screenshot to show your boss, urging them to support our initiatives. If that feels too honest, I’ll lean into the humor instead. My role often involves introducing industry best practices into the Ruby community and fostering communication between our community and others. We aim to ensure Ruby continues to thrive, supporting both hobbyists and Fortune 500 companies like Shopify who help make using Ruby enjoyable.

00:03:00.000 Since living in Scandinavia, I’ve been able to enjoy the novelty of it being late in the day and still having sunlight outside. Now, onto more serious matters. RubyGems.org is everyone’s favorite gem hosting service, I hope. While it may not have the enormous scale of GitHub or npm, it has grown about 20% per year for the last 20 years. If you remember your high school math, you’ll know that compound growth means the numbers go up quite significantly. It’s an important service with a vast number of users worldwide. Speaking of which, can anyone here estimate how many people are in this room right now?

00:04:40.880 Let’s do a quick show of hands: how many of you have ever run 'gem install'? Okay, now keep your hands up if you’ve ever included a line in a gem file. I see some hands down—I don’t believe you! Guess what? You’ve all contributed to the traffic that RubyGems.org handles, so congratulations, you’re part of the statistics. Here are some quick figures: around 10 to 11 million unique IP addresses per month download gems, and there were 23.3 billion requests made as of a recent month. That’s billion with a 'B'! We peaked at about 20,000 requests per second, with over 167 billion gem downloads cumulatively.

00:06:01.280 But we don’t accept returns or exchanges on gems—I’m sorry; it’s a final purchase! We’ve experienced exponential growth, and if we could monetize each download, I wouldn’t be standing here; I’d be on a private island. Let me share a quick story about the weekend when I got paged and missed brunch because we momentarily hit 225,000 requests per second. Anyone here ever served that many requests per second? It’s not fun if you’re not prepared for it. Typically, we peak at about 20,000 requests; an order of magnitude more requires way more web servers than normal.

00:07:00.599 A picture is worth a thousand words, so let me illustrate this with a graph showing the amount of bytes our Rails application served. You’ll see the numbers fluctuated between weekdays and weekends, but in May of last year, we experienced a significant spike one weekend. That spike was tied to an old API called the dependency API that we had deemed outdated after seven years. We believed it was safe to turn it off because it was cost-inefficient to maintain due to its expensive database queries. Before the final shutdown, we scheduled a brownout period so users could prepare for the change. During the last brownout, we saw that spike, revealing some users hadn’t updated RubyGems in nearly a decade!

00:08:24.760 Some of you may remember Chef, an automation tool that fetches cookbooks through RubyGems. Some AWS services provisioning virtual servers with Chef did not cache anything, meaning it would invoke the outdated dependency API every 30 minutes—downloading every gem spec repeatedly from three IP addresses corresponding to a large public company. In one unfortunate situation, I had to dig through Bundler’s source code to figure out how to raise error messages to guide them. Can you believe that? In the interest of time, I’ll skip ahead from the numbers and focus on what we’ve been doing.

00:09:07.560 This past year has involved some significant accomplishments, including the release of a feature called trusted publishing based on OpenID Connect. We also launched official GitHub actions for secure gem builds and pushes to RubyGems.org, introduced an alpha feature that automatically adds checksums for gems into lock files for security teams, and successfully migrated our asset pipeline to utilize import maps and stimulus instead of sprockets. Even though it sounds simple, it took over six months to implement.

00:09:30.720 Looking ahead, we have exciting plans, including beautiful redesigns for RubyGems.org and building a compliant Sigstore client dependent only on the Ruby standard library. There’s a group in our Slack workspace dedicated to establishing organizational support for companies with numerous gems, like AWS. The idea is for them to have a single organization owning all their gems, ensuring that every gem downloaded truly comes from AWS. We’ve also undertaken some small features, like removing avatars from RubyGems.org, which were initially taken down for privacy reasons.

00:10:20.320 Recently, we reintroduced avatars to enhance the visual appeal of the site. We’ve created a proxy to fetch the avatars securely while making 'gem install' and similar commands faster. This improvement dramatically reduces the time taken to reset gem environments to a clean state, much to my relief! Beyond just delivering these features, we handle daily operations to keep everything running.

00:10:49.760 Our deployment process is somewhat manual and irregular, often involving numerous deployments throughout the day or week. We emphasize zero-downtime deployments to ensure services remain uninterrupted. Additional challenges include zero write locks anytime we need to perform database migrations. A colleague is preparing a blog posting about how he managed to achieve this using tools like PG Bouncer and RDS.

00:11:42.840 We're now operating a 24/7 on-call rotation with members across several time zones, including Australia and the US. This means we sometimes stay up late or wake up early for calls. Managing this is costly; the work we do resembles that of larger companies with considerable support and funding. AWS substantially helps by providing credits that cover infrastructure expenses. We also rely on donations from Fastly, DataDog for monitoring, and error tracking services from Honeybadger.

00:12:55.520 Shopify has been particularly generous, contributing over a million dollars over four years to enhance RubyGems' security and reliability and the German Sovereign Tech Fund has advanced about $860,000 in recent years for both maintenance and security improvements. It’s amusing to think, but at one time, I could say my work was sponsored by the German government! Recently, funding has emerged from the OpenSSF Alpha Omega organization, which has facilitated a security audit and is also financing the addition of organizational features to RubyGems.org.

00:14:26.920 Don’t forget, AWS sponsors my role, giving them a round of applause! They made it possible for me to be here today. As you might expect, all this requires significant manpower. We currently have five engineers on staff, and most work part-time. For many, this is a side job, and they’re grateful to be compensated for the work instead of committing nights and weekends entirely.

00:15:24.640 We also have a supportive network from various Ruby-related companies, including hsbt, Shopify, and others that help with the integration of Ruby Gems and Bundler. There is a vibrant community of individual contributors, too, who support us continually. Similar to how PBS operates in the US, all this work happens thanks to grants and support from the community.

00:16:21.920 We value the feedback and contributions from our Ruby community, especially since maintaining such an ecosystem is challenging. Personally, I've been in this space for a decade, and we’re eager to retain this knowledge and grow our team, particularly during these financially turbulent times, which can make long-term planning difficult.

00:17:09.040 Regarding infrastructure, we face constant updates and deprecations, meaning we need to test, upgrade, deploy, and monitor new versions of services like Postgres and Kubernetes continuously. Each change requires substantial effort and time investment for staging and live testing. Managing a 15-year-old codebase has its challenges, evolving from Ruby 1.8 to now Ruby on Rails 7.

00:18:03.840 As we embrace new industry standards, we adapt to best practices, like moving from Sprockets to Prop Shaft or adopting Tailwind instead of Sass. Transitioning our service and alleviating our service’s limitations takes time and effort. The platform we maintain exists primarily to serve Ruby gems and Bundler, and we introduced the compact index nearly ten years ago to enhance dependency resolution.

00:18:43.440 So, if you’re interested in security, a buzzword you’ll want to latch onto is software supply chain security. That’s my focus here at RubyGems. With all the governmental attention on software supply chain issues, it has become increasingly vital to ensure the integrity of our ecosystem. To that end, we operate a bug bounty program on HackerOne and have received several hundred reports requiring varying degrees of analysis and verification.

00:19:34.240 From those reports, we’ve identified numerous issues, with some being significant concerns. We’re also assisting with documentation focused on security for package repositories, helping to define best practices for maintaining and securing services like RubyGems.org. Our primary goal is to ensure security audits are efficient and cost-effective.

00:20:18.720 A significant aspect of our security efforts involves preventing malicious activity within the Ruby gem ecosystem, such as typo-squatting and dependency confusion. Malicious actors may try to exploit common mistakes that developers make. Let me share an example from a recent time when I was on call: our site was inundated with traffic when a researcher started hitting every endpoint for every gem, causing us to investigate why there was a flood of requests.

00:21:16.440 I'll be candid—we were able to resolve the situation and communicate with the researcher to redirect them to a better resource. In my free time, I’ve been developing a project about Ruby gems, which focuses on incident response across the ecosystem. It allows us to track changes in gems over time to identify trends and possibly malicious changes.

00:22:14.360 If we determine that a file or a gem has exhibited suspicious behavior between versions, we can trace back through the repository history to understand those alterations. There was a notable incident a few months ago when the internet reacted to a vulnerability, and my role was to assess how deeply it impacted us.

00:22:55.920 Being able to search through our entire database of gems allowed for speedy investigation. Ultimately, we were fortunate; the main RubyGems platform was not affected, and we could relay that information quickly to the community.

00:23:48.280 As one of my ongoing projects, I am working on a Sigstore client which focuses on reinforcing software supply chain security. Sigstore, whose motto is "sign, verify, protect," ensures that software comes from legitimate sources, thus enhancing trustworthiness in the Ruby gems we use.

00:24:32.560 We rolled out the trusted publishing feature to automate gem deployment without persistent credentials which many developers find valuable for security. My goal is to advocate for widespread adoption of this feature during our campfire session this afternoon!

00:25:11.840 To wrap it all up, thank you to our sponsors for the contributions that have made this all possible. This includes organizations dedicated to security initiatives and financial support, like the German government, AWS, and Shopify. Before I end, I want to give a special thanks to my wife for supporting my attending this conference just a week after our wedding.

00:26:02.640 Thank you all for your attention; I look forward to hanging out this afternoon!

BalticRuby 2024