00:00:01.620
Oh, it's bright. Okay, hello everyone.
00:00:13.860
Welcome to my presentation about RubyGems security. A bit about me: I specialize in supply chain security and work for a company called Mend. I'm part of the RubyGems security team, an active open source contributor, and a creator of the Karafka framework.
00:00:35.820
You may know me from this button here. If you click it, you can get a nice diff of every single release of every package in RubyGems, so I built that. Surprisingly, we're going to talk for a second about that—it’s already fixed, so don't worry.
00:00:56.219
It's quite interesting, right? What a coincidence! RubyGems is not malicious, at least not all of it. Someone reported us to Google because of one of the packages called 'meta Sport payloads.' We know about it; it contains test signatures for antivirus software. Someone is probably just scanning packages and reported that it's not malicious.
00:01:30.060
However, your systems could be affected because it wasn't just on the website; there are lists for ad blockers, and if you have a system that blocks domains on the DNS level, you wouldn't be able to run 'bundle install.' Luckily, folks from Google fixed that just two minutes ago.
00:01:51.960
Now, let's get back to the main agenda. I'm going to give a really quick introduction to the supply chain, talk about the most recent critical incidents we had in RubyGems, what we did to mitigate them, and what countermeasures we are taking. Please keep in mind that your security is your responsibility. We do as much as we can, but in the end, you shouldn't fully trust anyone.
00:02:25.260
There are so many things happening in RubyGems regarding security that we could sit here for a whole day. So I'm just going to talk about what I do within the scope of RubyGems operations. The open source supply chain is quite simple; some people have packages they want to share, and there are people who just want to use them. This is the most common understanding of what an open source supply chain is.
00:02:58.800
However, it’s actually more complex than that. Registries tend to claim that packages are immutable, which means that if you upload version 1.1.0, it should remain the same whenever you download it. Unfortunately, that's not always true. People abuse certain capabilities in RubyGems and other registries, like npm or PyPi, to achieve auto-updating software. It's a really bad practice, but we need to deal with this as well.
00:03:39.300
For us at RubyGems, I have some quite interesting stats. Every second of my talk, there are 162 lines of content being released to RubyGems. That’s quite a lot of changes, if you think about it! Every three minutes, we get a new package version with many downloads and numerous users. If you convert that into a stream of messages, this is how it looks in what I would call real-time.
00:04:19.079
You need to keep in mind that most of your apps aren't actually yours; they’re built on top of blocks built on top of other blocks. That’s why it’s a really good thing for malicious actors to target open source supply chains to try to infiltrate companies.
00:04:56.160
If we consider the source origin of attacks, there are actually only two types: the internal ones, where the legitimate owners of packages do something out of frustration, and external malicious actors who want to take over packages or get into registries like RubyGems. Some malicious actors target our users or their gems, and some target the platform itself. Exploiting the platform would make an actor much more successful, but it's also much harder for us.
00:05:20.040
Recently, there have been a couple of very interesting and critical incidents in RubyGems. We had to issue critical severities on unauthorized package takeovers, which sounds really bad because it was. The codebase of RubyGems had a vulnerability that allowed you to create a different package that could take control of other packages. This meant that you could remove versions of packages you didn't own, potentially replacing them with your own malicious content.
00:06:41.400
Initially, we thought this problem wouldn't be serious since package versions are supposed to be immutable. But the majority of people don’t know that package versions are immutable only within the scope of the platform for which they're designed.
00:07:04.560
The default platform for RubyGems is Ruby, but there are other platforms based on CPU types. This vulnerability was exploited. For instance, if you could remove a Ruby-specific package version 1.1.0 and replace it with a CPU-specific package of the same number, it wouldn't violate Ruby's registration platform restrictions.
00:07:57.480
It's not a problem per se, but if you have a lock file and you don’t run it with the 'frozen' command, sometimes your dependencies can still resolve in unanticipated ways. If you're working in production, you wouldn't expect that because you want reproducibility.
00:08:19.139
We were really scared when this happened because we had to assume that someone might have exploited this for malicious reasons without reporting it to us first. Luckily, to take ownership of a package, one would have to yank all versions first and transfer ownership. This meant that if this was malicious activity, there would typically be a direct jump to the new owner, but that wasn’t the case this time.
00:09:58.080
We picked out all of the packages from last year with any state changes that included a new release or a yank. We focused specifically on packages with names that included a hyphen, as this was a requirement for the exploit to work. Then, we filtered those with specific platform releases.
00:10:56.580
Fortunately, we found three packages, and luckily, they were all only research packages. It was a critical incident, so we wanted to dive deeper into that. We wanted to find all packages that had a Ruby platform package removed but still had other platform-specific versions present.
00:11:29.640
We were able to download and check those packages to ensure they did not contain malicious content. It was a tough couple of days for us. Following that, we considered what we could do to ensure that if our bucket were compromised, we would be notified.
00:12:16.560
We built an external monitoring system that checks RubyGems periodically. The system checks all packages and compares their hashes. If everything is fine, we keep running this check in a loop. If something goes wrong, we get a notification and investigate.
00:12:59.880
Just two weeks later, we received another critical report. There was a new bug in RubyGems that allowed an attacker to release a gem version that crashed our validation system, though the artifact would still be stored, resulting in potentially malicious access.
00:13:57.180
While the version wouldn’t appear due to a 500 error and wouldn’t actually release it, if a new legit version were released later, the malicious package could exist in cache systems around the world.
00:14:30.420
It became evident that a malicious actor could effectively target a specific region, pre-caching that malicious version without alerting us, as we were operating under different conditions.
00:14:58.260
So how do you run a risk assessment in such scenarios to ensure a healthy and stable ecosystem? First of all, having an appropriate platform is crucial. We run checks that the packages had to hit the RubyGems API, which required a record in our backtracker.
00:15:50.520
While we indeed found only one non-research package that went through the entire cycle, it was a coincidence causing similar issues. We weren't entirely satisfied with our assessment, though; there may still be undiscovered packages in S3 containing code for unreleased legit versions.
00:16:55.800
To address this, we downloaded all data from S3, correlated it, and checked how many of those files corresponded to unreleased or non-existing versions. Luckily, we only found about 20 older versions related to different bugs that did not pose security threats.
00:17:48.120
We validated that RubyGems no longer faces this problem, completed cache purges, and confirmed the safety of the packages. However, we still sought to guarantee the safety of all users.
00:18:36.300
We built a small tool called 'bundler integrity' that checks directly with RubyGems for the proper checksums of all artifacts, thereby assuring users that nothing was compromised.
00:19:17.800
In summary, within a month, we faced two critical incidents. We recognize that not all issues receive CVE assignments. We noticed increased activity regarding brand jacking, which denotes the release of packages that creators think contain certain code, but they actually do not.
00:20:07.080
To combat this, we amended our gem-creation templates, alerting developers to ensure they register with RubyGems when they intend to release packages. We often block many packages each week that contain malicious tracing code or related issues.
00:21:17.040
A good ecosystem recognizes that people sometimes make mistakes. Domains can expire, affecting package registrations. To mitigate this, we implemented an automatic system that locks accounts one day before a domain expires, allowing us to handle it more efficiently.
00:22:37.440
I came across an article discussing how malicious attacks led to substantial downloads for a package. We must remain vigilant. Open source software is an excellent way for attackers to get into supply chains, and it can happen very quickly.
00:23:40.440
Malicious actors are shifting their activities earlier in the development cycle, which is much easier than targeting production systems. Many packages aim to capture environment variables and credentials that could eventually lead to unauthorized access.
00:25:10.020
Keep in mind that running 'bundle install' gives access to your computer. RubyGems is not an app store; it's a platform to share code. Thus, the end responsibility is on you as the user.
00:26:10.800
While there are numerous changes every second, there are only a few lines of new additions. We consider this when inspecting changes because the number of changed lines is generally much smaller than the total lines released.
00:27:01.920
Thank you.