Summarized using AI

RubyGems on the watch

Maciej Mensfeld • May 11, 2023 • Nagano, Japan • Talk

In his presentation at RubyKaigi 2023, Maciej Mensfeld discusses RubyGems security, focusing on the complexities of the open-source supply chain and recent security incidents. He highlights the risks associated with package management systems due to internal and external threats. Mensfeld presents key points regarding recent vulnerabilities and the strategies implemented by RubyGems to mitigate these risks. He explains that package versions are supposed to be immutable, yet the system can be exploited by malicious actors targeting the platform itself. \n\nKey Points Discussed:\n- Introduction to RubyGems Security: Mensfeld introduces himself and his work with RubyGems, emphasizing the importance of security in the open-source community.\n- Supply Chain Complexity: The open-source supply chain, while seemingly simple, has complexities that can lead to vulnerabilities. Mensfeld describes the common misconception that versions within a package system are immutable.\n- Recent Incidents: He describes critical incidents involving unauthorized package takeovers and a newly identified vulnerability that could allow malicious access through caching.\n- Monitoring and Response: Mensfeld details the establishment of an external monitoring system for RubyGems to detect changes in package hashes and potential security breaches in real-time.\n- Proactive Measures: He discusses the implementation of tools like 'bundler integrity' and adjustments to gem-creation templates aimed at preventing attacks related to brand jacking and enhancing user safety.\n- User Responsibility: Mensfeld concludes by emphasizing that while RubyGems works to protect users, the ultimate responsibility lies with developers to ensure they are securing their applications.\n\nConclusions and Takeaways:\n- The RubyGems platform must continually improve its security posture due to evolving threats in the open-source ecosystem.\n- Developers should adopt best practices and tools when interacting with RubyGems to minimize risks associated with package management.\n- Increased vigilance and proactive measures are essential in safeguarding the integrity of open-source software and its users.

RubyGems on the watch
Maciej Mensfeld • May 11, 2023 • Nagano, Japan • Talk

RubyKaigi 2023

00:00:01.620 Oh, it's bright. Okay, hello everyone.
00:00:13.860 Welcome to my presentation about RubyGems security. A bit about me: I specialize in supply chain security and work for a company called Mend. I'm part of the RubyGems security team, an active open source contributor, and a creator of the Karafka framework.
00:00:35.820 You may know me from this button here. If you click it, you can get a nice diff of every single release of every package in RubyGems, so I built that. Surprisingly, we're going to talk for a second about that—it’s already fixed, so don't worry.
00:00:56.219 It's quite interesting, right? What a coincidence! RubyGems is not malicious, at least not all of it. Someone reported us to Google because of one of the packages called 'meta Sport payloads.' We know about it; it contains test signatures for antivirus software. Someone is probably just scanning packages and reported that it's not malicious.
00:01:30.060 However, your systems could be affected because it wasn't just on the website; there are lists for ad blockers, and if you have a system that blocks domains on the DNS level, you wouldn't be able to run 'bundle install.' Luckily, folks from Google fixed that just two minutes ago.
00:01:51.960 Now, let's get back to the main agenda. I'm going to give a really quick introduction to the supply chain, talk about the most recent critical incidents we had in RubyGems, what we did to mitigate them, and what countermeasures we are taking. Please keep in mind that your security is your responsibility. We do as much as we can, but in the end, you shouldn't fully trust anyone.
00:02:25.260 There are so many things happening in RubyGems regarding security that we could sit here for a whole day. So I'm just going to talk about what I do within the scope of RubyGems operations. The open source supply chain is quite simple; some people have packages they want to share, and there are people who just want to use them. This is the most common understanding of what an open source supply chain is.
00:02:58.800 However, it’s actually more complex than that. Registries tend to claim that packages are immutable, which means that if you upload version 1.1.0, it should remain the same whenever you download it. Unfortunately, that's not always true. People abuse certain capabilities in RubyGems and other registries, like npm or PyPi, to achieve auto-updating software. It's a really bad practice, but we need to deal with this as well.
00:03:39.300 For us at RubyGems, I have some quite interesting stats. Every second of my talk, there are 162 lines of content being released to RubyGems. That’s quite a lot of changes, if you think about it! Every three minutes, we get a new package version with many downloads and numerous users. If you convert that into a stream of messages, this is how it looks in what I would call real-time.
00:04:19.079 You need to keep in mind that most of your apps aren't actually yours; they’re built on top of blocks built on top of other blocks. That’s why it’s a really good thing for malicious actors to target open source supply chains to try to infiltrate companies.
00:04:56.160 If we consider the source origin of attacks, there are actually only two types: the internal ones, where the legitimate owners of packages do something out of frustration, and external malicious actors who want to take over packages or get into registries like RubyGems. Some malicious actors target our users or their gems, and some target the platform itself. Exploiting the platform would make an actor much more successful, but it's also much harder for us.
00:05:20.040 Recently, there have been a couple of very interesting and critical incidents in RubyGems. We had to issue critical severities on unauthorized package takeovers, which sounds really bad because it was. The codebase of RubyGems had a vulnerability that allowed you to create a different package that could take control of other packages. This meant that you could remove versions of packages you didn't own, potentially replacing them with your own malicious content.
00:06:41.400 Initially, we thought this problem wouldn't be serious since package versions are supposed to be immutable. But the majority of people don’t know that package versions are immutable only within the scope of the platform for which they're designed.
00:07:04.560 The default platform for RubyGems is Ruby, but there are other platforms based on CPU types. This vulnerability was exploited. For instance, if you could remove a Ruby-specific package version 1.1.0 and replace it with a CPU-specific package of the same number, it wouldn't violate Ruby's registration platform restrictions.
00:07:57.480 It's not a problem per se, but if you have a lock file and you don’t run it with the 'frozen' command, sometimes your dependencies can still resolve in unanticipated ways. If you're working in production, you wouldn't expect that because you want reproducibility.
00:08:19.139 We were really scared when this happened because we had to assume that someone might have exploited this for malicious reasons without reporting it to us first. Luckily, to take ownership of a package, one would have to yank all versions first and transfer ownership. This meant that if this was malicious activity, there would typically be a direct jump to the new owner, but that wasn’t the case this time.
00:09:58.080 We picked out all of the packages from last year with any state changes that included a new release or a yank. We focused specifically on packages with names that included a hyphen, as this was a requirement for the exploit to work. Then, we filtered those with specific platform releases.
00:10:56.580 Fortunately, we found three packages, and luckily, they were all only research packages. It was a critical incident, so we wanted to dive deeper into that. We wanted to find all packages that had a Ruby platform package removed but still had other platform-specific versions present.
00:11:29.640 We were able to download and check those packages to ensure they did not contain malicious content. It was a tough couple of days for us. Following that, we considered what we could do to ensure that if our bucket were compromised, we would be notified.
00:12:16.560 We built an external monitoring system that checks RubyGems periodically. The system checks all packages and compares their hashes. If everything is fine, we keep running this check in a loop. If something goes wrong, we get a notification and investigate.
00:12:59.880 Just two weeks later, we received another critical report. There was a new bug in RubyGems that allowed an attacker to release a gem version that crashed our validation system, though the artifact would still be stored, resulting in potentially malicious access.
00:13:57.180 While the version wouldn’t appear due to a 500 error and wouldn’t actually release it, if a new legit version were released later, the malicious package could exist in cache systems around the world.
00:14:30.420 It became evident that a malicious actor could effectively target a specific region, pre-caching that malicious version without alerting us, as we were operating under different conditions.
00:14:58.260 So how do you run a risk assessment in such scenarios to ensure a healthy and stable ecosystem? First of all, having an appropriate platform is crucial. We run checks that the packages had to hit the RubyGems API, which required a record in our backtracker.
00:15:50.520 While we indeed found only one non-research package that went through the entire cycle, it was a coincidence causing similar issues. We weren't entirely satisfied with our assessment, though; there may still be undiscovered packages in S3 containing code for unreleased legit versions.
00:16:55.800 To address this, we downloaded all data from S3, correlated it, and checked how many of those files corresponded to unreleased or non-existing versions. Luckily, we only found about 20 older versions related to different bugs that did not pose security threats.
00:17:48.120 We validated that RubyGems no longer faces this problem, completed cache purges, and confirmed the safety of the packages. However, we still sought to guarantee the safety of all users.
00:18:36.300 We built a small tool called 'bundler integrity' that checks directly with RubyGems for the proper checksums of all artifacts, thereby assuring users that nothing was compromised.
00:19:17.800 In summary, within a month, we faced two critical incidents. We recognize that not all issues receive CVE assignments. We noticed increased activity regarding brand jacking, which denotes the release of packages that creators think contain certain code, but they actually do not.
00:20:07.080 To combat this, we amended our gem-creation templates, alerting developers to ensure they register with RubyGems when they intend to release packages. We often block many packages each week that contain malicious tracing code or related issues.
00:21:17.040 A good ecosystem recognizes that people sometimes make mistakes. Domains can expire, affecting package registrations. To mitigate this, we implemented an automatic system that locks accounts one day before a domain expires, allowing us to handle it more efficiently.
00:22:37.440 I came across an article discussing how malicious attacks led to substantial downloads for a package. We must remain vigilant. Open source software is an excellent way for attackers to get into supply chains, and it can happen very quickly.
00:23:40.440 Malicious actors are shifting their activities earlier in the development cycle, which is much easier than targeting production systems. Many packages aim to capture environment variables and credentials that could eventually lead to unauthorized access.
00:25:10.020 Keep in mind that running 'bundle install' gives access to your computer. RubyGems is not an app store; it's a platform to share code. Thus, the end responsibility is on you as the user.
00:26:10.800 While there are numerous changes every second, there are only a few lines of new additions. We consider this when inspecting changes because the number of changed lines is generally much smaller than the total lines released.
00:27:01.920 Thank you.
Explore all talks recorded at RubyKaigi 2023
+52