How to take over a Ruby gem

Talks

Maciej Mensfeld

@mensfeld

#rubygems

#security

#dependency-management

How to take over a Ruby gem

by Maciej Mensfeld

In this presentation, Maciej Mensfeld discusses the vulnerabilities associated with Ruby gems and how developers can protect themselves against various attacks. He begins with an overview of Ruby gems, their popularity, and mentions their potential security risks. Key points include:

Introduction to Ruby Gems: Ruby gems, similar to npm for JavaScript, have around 160,000 published gems and face security challenges.
Malicious Installation: Users can unknowingly install gems that perform harmful actions, such as gathering sensitive information.
Post-Install Hooks and Code Execution: Mensfeld explains how malicious actors can utilize post-install hooks and native extension compilations to execute hidden code during the gem installation process.
Typosquatting Threats: He highlights typosquatting, a technique where attackers register misspelled versions of popular gems, leading to the potential for exploiting unsuspecting developers.
Risks of Abandoned Gems: The presenter warns that many gems with high downloads are abandoned, granting attackers opportunities for takeover. He draws parallels to past events, such as the incident with Bootstrap Sass where a malicious code was introduced.
Mitigation Strategies: Mensfeld stresses the importance of adopting security practices, such as two-factor authentication, dependency policies, and conducting manual checks on gem updates.
Conclusion: The talk emphasizes the importance of vigilant code reviews and limited trust in dependencies, encapsulating the overarching need for developers to maintain security awareness when using Ruby gems.

00:00:01.100 Hello, let's start. If you sit in the back and have any problem seeing my slides, because there will be some code examples, you can just open the presentation on your mobile phone or laptop and follow along. I will give you about 10 seconds to copy that or take a photo, just in case.

00:00:07.049 I usually have up to six lines of code, so it should be fine. But in any case, the slides are here. A couple of disclaimers: don't do this at home. There are more problems related to Ruby gems, but I won't be talking about all of them, just a couple. If you're interested, you can find me at every after-party and socialize there. As for the tools I will be showcasing, or any other open source projects, I haven't had time to refine them, and if I opened them now, you might think I am a mediocre programmer. However, I will release them in a month or two.

00:00:26.340 About me: my name is Maciej Mensfeld, but just call me Mattie. I'm just a programmer, and I've contributed to some open source projects. You can find my code in Karafka, dry-components, and Kafka, among a few others. I tend to find peculiar bugs in Ruby from time to time as well. Most of my commercial life is programming; I work at a company called Castle, which provides end-to-end solutions to combat account takeover attacks. To put it simply, our focus is security. That's why I take some of my working hours to focus on the security aspects of developing software in Ruby.

00:01:08.610 Additionally, I'm an organizer of the Krakow Ruby User Group, one of the largest Ruby groups in Europe. This year, we're also organizing a small conference called Color Code RB in Krakow. If by any chance you're in Poland, come visit us! It has been quite a journey, as it took me almost 10,000 kilometers to get here. The travel was fun yet tiring, taking about 20 hours to reach Tokyo, resulting in a bit of jet lag. However, I always enjoy visiting Japan; this is my second time speaking at RubyKaigi, but my first time in Japan. I've been practicing martial arts for a while now and I love 'Fullmetal Alchemist,' both the manga and the anime.

00:01:54.090 If you have any questions or just want to send me a message, you can find me on Twitter. You can also use my email or check out my blog where I write a bit about Ruby as well. I'm definitely not an expert that possesses comprehensive knowledge about security; my area of expertise is somewhat limited to the realm of Ruby gems. If you ask me about really complex, low-level Linux security issues, I might not have the answers. If you find yourself unsure about what to discuss with me during the party, that's completely fine.

00:02:51.430 So let's get started! I hope to leave you with some knowledge about the serious issues we face with Ruby gems, what to do as countermeasures against various threats associated with working with these gems, and how people can approach the dark side if they decide to explore that path to gain certain benefits. However, I strongly advise against doing that! I think the Ruby community has enough problems to contend with already.

00:03:39.459 Let me briefly explain what Ruby gems are, assuming you are somewhat familiar with them. Ruby gems can be thought of as the npm equivalent for Ruby; technically, Bundler is a separate tool related to Ruby gems but let's not complicate things. It got started around 15 years ago by Chad Fowler and Richard Kilmer, and it's still going strong; it's open source. There are around 160,000 gems published, with about 130,000 active users and a staggering 33 billion total downloads, which translates to approximately 71 downloads per second. This means there are ample opportunities to do some interesting work, or at least attempt to!

00:04:27.480 Usually, Ruby gems come bundled with Bundler; they aren't the same, so please keep that in mind as some people might take offense. I remember a time when installing everything manually was common practice with Rails applications which was quite problematic. The session is recorded, so I won't use harsh language, but to put it bluntly, Ruby gems aren't perfect; they indeed come with certain risks. I'll skip the part about making users blindly install gems; my aim now is to operate under the assumption that someone might decide to install our gem purely for simplicity's sake.

00:05:26.340 So let's say we have a user who decided to install our super library. What does that actually mean? The majority of people would argue that it means nothing at all; they've simply downloaded the gem and it's ready to go. But is that true? We could potentially play with this a bit, but I must insist: please don’t do any of this at home. What I’m going to present comes from my work with Ruby gems, and I strongly advise against taking risks by installing any of those gems yourself. While they might not steal user credentials or SSH keys, they could still behave maliciously.

00:07:27.780 Gathering information from an end-user laptop is relatively easy with Ruby. As long as Ruby can run, one can list the directory of files and aggregate the information, potentially providing debugging opportunities if something goes wrong. A shady gem can improve itself with user feedback, but in a typical scenario, you should expect to find everything you're interested in within an RI. Once you've assembled that data, all you need to do is dispatch it. Don’t worry, it’s pretty standard practice.

00:08:18.680 Next, let's consider the possibility of executing Ruby gems without even writing any code. If you require executing a library code, it's already a bit late. It would be more beneficial if we could somehow force someone into installing the library in the first place. The first approach to achieve this is using post-install hooks; they are documented, so there’s no issue there. However, these hooks work only when you are using Bundler and setting things up directly from GitHub. They often fail when imported from Ruby gems, which is a good safeguard.

00:09:15.870 If you want to provide a message to the end user at install time, you use the post-install message which is serialized. We checked this in Tokyo after having some coffee, and it's suggested to be safe. Hence, thumbs up for Bundler for that, and kudos to Ruby gems for disallowing that in most cases. Using GitHub can be quite useful, as it allows for providing certain features in your private libraries. However, it's not recommended to use gems directly from GitHub because any changes made there may have amplified impacts on your projects.

00:10:49.170 Nevertheless, there is a trick one can use to run any code during the installation process while keeping it hidden, so no one notices anything. This is achieved by exploiting the extensions API used for compiling native extensions. One may define a file meant to prepare a Makefile for building a native extension; the limitation is nonexistent. Instead, you can inject any code into it, generating an empty dummy Makefile that reassures the end-user that everything went smoothly.

00:11:39.860 If done correctly, this can take as little as half a second to send data to the server. When simulating this action with Ruby gems, it gives the illusion of compilation, which remains undetected. I'll demonstrate this as long as I have an internet connection. I prepared a simple script that forces the installation of a gem that I pushed to RubyGems. It’s a basic gem, and please don’t even think about trying it. What the script does is send a list of files in the home directory to my server.

00:12:44.020 Just to demonstrate, I set up an empty screen waiting for the network, and it may take a few moments due to slow internet connectivity. As you can see, the gem is being installed, and it is performing native extension installation. Eventually, what happens is that the entire list of files found in my local machine is sent online. I had some friends install this gem for me to verify it would work effectively, despite them thinking it was completely safe.

00:13:50.730 The truth is that merely installing the gem is sufficient for it to function. Now let’s say we don’t want to steal SSH keys; we wish to create a botnet — a more alarming scenario. You might recognize an instance of this code from Bootstrap Sass. I had submitted a CFP for this conference months ago before the security event escalated with Bootstrap Sass, as they used the same method.

00:14:23.223 What this malicious implementation does is monkey-patch a library while looking for a specific cookie to execute whatever code is present. I would refrain from demonstrating due to poor internet quality, but the point is that commands can be sent via a browser, allowing for sophisticated operations. For instance, one could construct a bot that reads incoming request data, buffers it, and sends all this information while concealing exceptions.

00:15:03.110 Interestingly enough, you can utilize this technique to infect other gems while cleaning up your own, given that some people inspect their gems locally or integrate a debugger. It’s astonishingly simple! The method actively searches for other gems to insert the botnet code into, all while ensuring your gem appears untainted. But realize: you get unlimited power, and that indeed poses significant risk!

00:15:58.370 But in practical terms, it isn’t straightforward to just install random gems. One typically needs to be a decent contributor or a core member to have release permissions. In recent years, Ruby gems have introduced two-factor authentication; however, in my opinion, that is still not enough. Malicious actors can leverage techniques like typosquatting to target Ruby gems.

00:16:37.350 For those who aren't aware, typosquatting involves registering gems that are misspellings of popular libraries on Ruby gems. There's a gem available on Ruby gems to generate typos, meaning nefarious actors can create versions that often go unnoticed. As a result, when you look through the Ruby gems catalog, you see that people frequently make mistakes, which is unfortunate.

00:17:45.140 It is shockingly simple to generate such schemes; Ruby does not hinder you from registering gems with different names. To my surprise, Ruby gems previously allowed the same gem name to have multiple variations by simply changing the letter casing. For example, you could have a gem called 'Rails' in capital letters and another gem named 'rails,' all while maintaining both trades in the system.

00:18:46.780 While the system has improved and now discourages this practice, many gems still exist with both attributes. Furthermore, I've encountered instances where individuals have posted issues asking for removal of certain gems, with the response being, 'There's no malicious code here, so no reason to remove it.' This is alarming because it implies that I could game the system, booking vulnerable gems with similar sounding names.

00:19:54.760 Moreover, from a cyber security perspective, bit scoring exploits the likelihood of typing errors. It assumes that when numerous requests reach certain domains, there’s a chance of inadvertently introducing a vulnerability. I purchased several domains that appear confused, only as a proof of concept regarding how easy it is to register such typos and leverage them.

00:20:31.300 Though I did not receive much traffic, I could still observe requests, raising concerns about the potential for malicious activity. The underlying issue arises from malicious takeovers, which are often a product of social engineering—not only attacks on Ruby gems, but the platform as a whole. Additionally, Ruby gems have many versions, with over 5,000 gems having reached 100,000 downloads.

00:21:51.810 With almost 2,000 gems surpassing a million downloads, I utilized the Ruby gems database to identify popular gems, of which approximately 3,500 have not been updated for years. This allows pinpointing gems that seem unmaintained, which, unfortunately, grants attackers direct access. I've found abandoned gems with substantial downloads and checked their GitHub pages, confirming their lack of upkeep.

00:22:36.210 My experience with these libraries demonstrates that the original maintainers often grant access to their gems due to busyness or loss of interest. This situation resembles the NPM event stream injection, where someone mistakenly integrated a Bitcoin miner after addressing a few issues without considering the broader impacts.

00:23:09.310 One crucial takeaway here is that once you gain release permissions, try to maintain some form of reference on GitHub. The main reason Bootstrap Sass was flagged was the absence of any rationale for the gem version being yanked; no updates had been recorded on GitHub, so someone started to investigate and discovered the mishap. Furthermore, one should be mindful that merely being on Ruby gems does not guarantee security.

00:24:32.240 Hence, if you have a GitHub version of your library, be aware that it does not have to mirror what's on Ruby gems. This realization is key for developers, ensuring that you’re adopting verification steps before releasing gems. Though I still have some remaining time, I believe many in this room may recognize a fellow— the Japanese minister of cybersecurity who publicly stated he doesn’t use a computer. Recently, he also admitted his ignorance on cybersecurity issues.

00:25:40.860 In a way, this reflects how everyone can implement safety measures. Just remember, it's unsafe to rely on Ruby gems completely. There's no universal solution to tackle such issues; Ruby’s inherent dynamic nature allows monkey-patching. However, specific strategies can mitigate risks.

00:26:29.770 Though many challenges might not feel pressing for smaller organizations, they become critical when junior developers tackle code. Implementing a typo-squatting scanner, utilizing middleman algorithms, and Levenshtein scoring can help identify authentication attempts more effectively but also come with many false positives.

00:27:55.910 Ruby gems together allow us the ability to track how many are under maintenance, comparing them with known gems. By monitoring their activity, it becomes easier to identify outdated libraries which can lead to vulnerabilities. We maintain a database of gems to track and identify these outdated libraries that we realize need modification or are being taken over by less scrupulous actors.

00:28:54.020 Moreover, implementing a dependency management policy becomes vital to know what you have installed, especially to prevent malicious gems from infiltrating your ecosystem. Be careful with your automated dependency updates. People often remove gem file locks carelessly, leading to untrustworthy executions on CI pipelines where keys might become compromised during the installation process.

00:30:15.840 Instead, rely on manual checks of the gems and their releases. Tools I am developing will facilitate that process, allowing developers to check differences in code and detect deprecated versions. As long as we review code appropriately from Ruby gems, it ensures security against rogue code attacks.

00:31:21.840 Overall, limited trust must be the standard approach to maintain safety. Keeping the limit on how many Ruby gems to use and tracking changes rigorously is crucial. Fortunately, tools streamline this monitoring process, making it a bit less burdensome. If you are a maintainer or owner of a gem, please utilize two-factor authentication across interfaces and APIs.

00:32:45.990 Consider creating isolated CI stages as you build Docker images or perform Ruby gem installations to ensure no environment variables that could jeopardize the system are inadvertently exposed. I will put several of the code samples and conduct further discussions on the platform after.

00:33:19.150 Finally, remember to review code changes on Ruby gems, as it secures your dependencies. Thank you for being this attentive audience, and feel free to reach out with any questions!

RubyKaigi 2019