00:00:00.900
Good afternoon everyone! I'm excited to talk to you today.
00:00:22.380
Did you all know that? I didn't! But clearly, I've thoroughly researched this. I am definitely team Portland, though; and that's not just because we’re physically here.
00:00:30.120
Anyway, welcome back from lunch everyone! We're really excited to have you all join us for a fun-filled half-hour of learning, laughing, and maybe a little bit of groaning if none of our jokes land. But if you make it through this whole talk, we have a fun little surprise for you at the end. So, if you’re ready, let’s hear some verbal bat signals! Whoop whoops, or anything!
00:00:58.620
Nice! We're getting lots of whoop whoops and caca, so let's get started. Hello everyone, my name is Betty, and I'm feeling super lucky to grace the stage with the effortlessly smart and wonderfully kind Ashley Ellis Pierce.
00:01:09.180
Now, gassing ourselves up, I honestly think the world of Ashley. I think she’s fire or, as some kids say, Ashley is lit. Quite literally too, as she just lit herself on fire the other day! I'm not even joking. That’s her favorite shirt!
00:01:28.020
Anyway, I'm so excited to do this talk with her because we channel the energy of a badass dynamic duo. I’d compare us to the likes of Batman and Robin or Anna and Elsa. But then, I started reflecting on it and realized we might be more like Pinky and the Brain.
00:01:34.740
Ashley brings the brains, and I bring a lot of other things to this relationship too. And that’s not a knock on myself by any means! I also want to quickly share that I'm a first-time conference speaker.
00:01:43.740
Thank you! Speaking at RailsConf is definitely a career bucket list item for me, so it’s an honor to be here but, I’m not gonna lie, my palms are a bit sweaty, knees weak, arms are heavy. There's vomit on my sweater already—mom's spaghetti! I'm a bit nervous.
00:01:50.220
But I'm working on channeling all that energy into some pure, unadulterated excitement, so bear with me if I’m a bit of a loose cannon today. Hi Mom! Also, hello, Sandy Metz! The YouTube algorithm is probably gonna get her to watch our talk at some point, right?
00:02:00.420
I’m really taking us far off base right now. Ashley is probably going to rein me in soon, but before we do that, we want to build some positive energy in this room. So, can everyone just face their neighbors and give them a socially distant but enthusiastic high five?
00:02:12.060
Great! Now can you send some high fives up here to Ashley and me as well? Thank you! I like to call these y5s because it’s like a high five over Wi-Fi!
00:02:23.520
Okay, now that this is all out of my system, let’s get to the good stuff! Ashley, what are people here for?
00:02:29.160
Great question, Betty. The title of our talk is 'Gem install: What could go wrong?' and spoiler alert—the answer is: a lot. We put a lot of faith in gems and we want to make sure that they're safe.
00:02:44.340
Betty and I work together at Shopify on the Ruby conventions team. Our team works to ensure that Shopify's many hundreds of Ruby apps are all in tip-top shape. So, for example, if there’s a Ruby CVE, we make sure that none of the over 600 Ruby apps are running the vulnerable version.
00:03:04.660
In that work, we started to consider how to ensure that none of these apps download an unsafe gem. This led us into the all-encompassing multi-faceted world of supply chain security, and that’s what we're here to discuss today.
00:03:19.180
When you install a gem, what could go wrong? That gem looks so cute and friendly, right? It’s tough to imagine that it could be harmful. But it turns out a lot could go wrong; the security of the gems we use is incredibly important.
00:03:32.700
When we run gem install, we are putting unquestioned faith in whatever we're installing, and the security of our gems is becoming more critical every year. According to Sonatype's 2021 State of the Software Supply Chain report, there was a 650% increase in software supply chain attacks aimed at exploiting weaknesses in upstream open-source ecosystems.
00:03:47.260
In 2020, there was a 430% increase over 2019. So, every year, this is accelerating and there are more and more people trying to make sure that when you run gem install, something indeed goes wrong.
00:04:07.020
There are a couple of notable supply chain attacks from last year that you might have heard of. One was the Codecov attack. Codecov runs a code coverage report on your CI environment and then uploads that report to its servers.
00:04:21.480
But between January and April 2021, its stock range was modified. For Codecov's 23,000 customers, that uploader also sent their environment variables to the hackers' remote server.
00:04:39.259
Another supply chain attack from 2021 was the dependency confusion attack. Unlike the Codecov attack, this wasn’t taking advantage of one company in the supply chain and going after all their customers.
00:04:53.340
This was taking advantage of a broader vulnerability in how companies structure the way they get packages. Basically, it exploited when you say gem install XYZ; how do we know where to go to get that gem from? Is it from RubyGems or from some other gem hosting server?
00:05:02.980
We’re going to talk about this in more detail later, but suffice it to say that dependency confusion is bad and we want to avoid this. These examples are just two small pieces of evidence that confirm that these attacks are indeed happening.
00:05:12.560
So, why aren’t all developers up in arms doing something about this? Well, one reason may be that we want to leave it to the experts, right? In the past, security is not one of the areas that I have personally felt most empowered to tackle.
00:05:24.000
It's really hard to know where to start looking. Web developers often rely on security-focused teams to be the keepers of this knowledge. But the thing is, we can be keepers of this knowledge, too, because the most common attacks are actually quite easy to execute.
00:05:37.860
And that’s why they’re common. Most often, there aren’t any super hacker tactics required. As much as NCIS would love for us to believe that the way to defend ourselves from attacks is to have two people fiercely keyboarding away on the same keyboard, that is sadly not the case.
00:05:48.000
Otherwise, Ashley and I would single-handedly—dare I say double-handily—defend against all attacks at Shopify. Attackers prey on the fact that developers are imperfect human beings.
00:06:00.480
They exploit common mistakes. For instance, a common exploit is where attackers take advantage of common spelling mistakes or when a gem is hosted on a different source repository.
00:06:10.620
In the most classic tactic from the hacker handbook, they prey on the fact that some of us are lazy humans. Many of us are guilty of reusing our passwords, right? I’m sure people can relate.
00:06:22.140
The good news is that over the next 20 odd minutes, we’ll unpack some types of attacks, how they happen, and, most importantly, you’ll walk away with strategies to defend against them.
00:06:31.440
Our goal is that you no longer need to solely rely on security-focused teams and that you can be the first line of defense.
00:06:37.420
So, as Betty said, before we get into how to mitigate attacks, it's probably good to understand what kinds of things can happen when malicious code gets into our gems.
00:06:40.620
I mean, really, what's the worst that can happen? What can attackers actually do with compromised gems? Developers are secretive people; we live a double life. Part of it involves sharing code with others, while the rest of the time, we deal in secrets.
00:06:55.720
This is why our servers are so big—they're full of secrets! You may have things like your AWS deploy tokens, RubyGems access tokens, and all sorts of secret data stored on your local machine or servers.
00:07:05.700
That’s a lot of trust to put into random gems, like inviting strangers into your home and hoping none of them are thieves. A bad actor could put code into a gem to steal these secrets and send them back to the attacker.
00:07:20.179
This isn’t just hypothetical. When the Rest Client gem was compromised in 2019, this was one of the things it did—it checked to see if it was on any environment but localhost. If so, it would take all the environment variables, add them to a hash, and then post that information back to the hacker's own website.
00:07:33.700
As you can see, this is pretty simple code. The hacker isn't some genius furiously typing away, seeing who can type faster. The good guy or the bad guy?
00:07:46.939
Another type of attack is ransomware, where instead of taking something from you—like your secret credentials—they hide your own data from you. They'll encrypt your data, so now you can't access it, or maybe they'll lock you out of your system entirely.
00:08:01.860
You'll only gain access back to your files or system once a ransom is paid. There was the attack called Reuke a few years ago, which was a widely publicized ransomware attack between 2018 and 2020. The FBI estimated it caused $60 million in damages worldwide.
00:08:19.759
If an attack like that happens 700 times, that’s enough money to acquire Twitter, too soon! So, ransomware is pretty detrimental. Similar to ransomware, where the goal is to extract money from you, there are also lots of crypto-based hacks.
00:08:32.020
Maybe they want to install a Bitcoin miner on your machine to use your server resources instead of paying for their own, or maybe they want to just straight-up steal any crypto that you might have. One example was in 2020 with two Ruby gems called Pretty Color and Ruby Bitcoin.
00:08:53.100
Pretty Color was a fraud gem; it was an exact replica of the Color gem which lets you add some color to your console. If you were looking for a gem to perform this action, you could have easily downloaded the Pretty Color one without realizing it was malicious.
00:09:02.500
With these attacks, they would take control of your clipboard, so that if you ever copied a cryptocurrency wallet address, it would replace it with the hacker's wallet address instead. You might ask someone to send you money and try to send them your wallet address, but you’d actually be sending the hacker’s wallet address.
00:09:19.470
This is a niche and fairly small potatoes attack, but it happens. Another type of attack is when an attacker adds a backdoor. This is a broad, arbitrary term; it generally means when a malicious script is installed to give the hacker a persistent gateway into your system.
00:09:36.420
An example of this was a hack of the Strong Password gem in 2019, which was also part of the Rest Client gem hack we discussed. This is a common attack method. Attackers added a lot of clutter to make it seem confusing to anyone looking at the code.
00:09:47.220
However, they had simply added code to check if the Rails environment is production; if so, they would eval open whatever was on Pastebin. This is a backdoor because they could use it to steal secrets, like they did with Rest Client.
00:09:57.440
But they could also use it to do whatever they wanted; they could keep changing the code on Pastebin, and whatever is there would run. Here’s another example from a 2019 attack on the Bootstrap Sash gem. A specific cookie sent by the client is base64 decoded and then evaluated at runtime, effectively allowing for remote code execution.
00:10:10.379
This is very similar to what happened with Pastebin. They can put anything they want into the cookie, and it will run on your servers. So, this is another example of a backdoor.
00:10:20.240
As I mentioned, backdoors are a pretty broad category because they can steal credentials or install ransomware, but they can also do just about anything else, like install a crypto miner. The sky's the limit.
00:10:36.020
So now that we've explored what's the worst that can happen, let's unpack how malicious code actually gets into our systems. One way is through installing the wrong gem.
00:10:52.680
Before I get into that, I want to do a quick survey. How many of you have ever made a typo? I see some hands still aren't up; I’m assuming you just abstained from typing to keep your typing record pristine! Just kidding.
00:11:10.200
For our friends watching this from home during the RailsConf Home Edition, basically the whole room more or less had their hands up, which isn’t shocking. This brings us to a weakness that attackers try to exploit.
00:11:19.860
They're counting on our propensity to mistype something. Instead of typing 'gem install RoboCop', you're typing 'gem install Robocop', or 'gem install Robocoop'. I hope you get the point.
00:11:29.520
Sneaky attackers squat on gem names that are just a typo or two away from legitimate gem names. Their goal is to trick you into erroneously installing their malicious code in your applications and running it on your local machine or production systems. At that point, they could be crypto mining or doing something even worse.
00:11:49.080
It’s just bad news bears all around. This type of attack is called typo-squatting, and there’s also a variation of it, or what I like to call the cousin of typo-squatting, called combo-squatting. An example would be if you wanted to include a gem called 'cool beans' but instead installed the fake gem called 'beans cool' because you didn’t remember it properly.
00:12:06.420
At that point, it definitely will not be 'cool beans', am I right? What’s interesting about this class of attacks is that it's the most common attack on software supply chains.
00:12:23.940
I’ll let that sink in—the most common attack on software supply chains exploits our inability to type or remember sequences of words accurately. Another way a wrong gem gets installed is when the source repository isn’t explicitly stated.
00:12:35.400
If it's not clear where the dependency should be downloaded from, for instance, let’s say we have an application with a bunch of gem dependencies. Some are hosted publicly on RubyGems, while others are hosted privately on our private server.
00:12:52.940
Honing in on one of our private gems, let’s say we have one called 'Soho House' because you all know how private and exclusive Soho Houses are, right? An attacker learns of this private gem and finds out there's no public version.
00:13:07.780
So they proceed to create a public version of 'Soho House', and of course, their version will have all the malicious code in it. But the crucial part is what happens when we run 'bundle install'. Well, Bundler needs to decide where to download 'Soho House' from.
00:13:19.920
Will it be from our private server or the public server, which is RubyGems? The thing is, it depends. Bundler will decide based on whatever gem is the latest version; it will download from that source repository.
00:13:37.020
This is the attack Ashley was mentioning earlier, called the dependency confusion attack—a very aptly named tactic, I’d say. On the note of confusion, account ownership can also be quite confusing.
00:13:53.620
Specifically, the confusion is over who actually owns an account. I highlight this as a problem because the second most common attack on software supply chains is account takeovers.
00:14:07.440
This affects us because it’s possible for people who are the maintainers of our dependencies to have their accounts stolen. The attacks on gems like Rest Client, Strong Password, and Bootstrap SASS all happened through account takeovers.
00:14:20.340
Once an account has been compromised, an attacker can perform a range of privileged actions, like yanking a gem or pushing a new release of a gem with malicious code.
00:14:34.659
It’s like you thought Aaron Patterson created a new version of Rails, but it turned out to be done by Aaron Batterson. Did you like that one? I was really proud when I came up with that.
00:14:50.940
Another way malicious code can get installed is if RubyGems itself gets compromised. Now, this is far less common than the previous attacks I mentioned, but it's the most dangerous for all our Rails applications.
00:15:02.700
If RubyGems is compromised, attackers could just change the files on package repository servers without maintainers even realizing it. This type of attack is called repository compromise.
00:15:15.540
The attackers basically have the ultimate keys to the kingdom. In fact, just recently, RubyGems addressed a critical vulnerability where any RubyGems user could yank and replace certain gems that fit a particular criteria.
00:15:31.920
Fortunately, WhiteSource completed an analysis and concluded that no gems were compromised. So, while this attack is uncommon, it is still a risk that exists when we install and use gems.
00:15:45.640
Now that we've sufficiently built up our paranoia, let me clarify: this isn't a PSA for y’all to stop using gems. Gems are incredibly useful—they're convenient and help us iterate faster.
00:16:11.720
We don't need to constantly reinvent the wheel. Besides, Rails is a gem, and we’re not here at RailsConf advocating for you to stop using Rails! Stand on the shoulders of giants; it accelerates your development and innovation.
00:16:27.960
But, as we've discussed here today, there are some very scary things that can happen if we don't secure our gems. There are some inherent risks to using gems. Our goal is to understand the risks and be diligent about mitigating them.
00:16:49.260
I mean, driving can be risky, right? We could all approach it like a game of Grand Theft Auto, but we don't. We wear our seatbelts, follow traffic laws, and take safety precautions to make our journey as safe as possible.
00:17:07.680
We need to adopt a similar approach to supply chain security to ensure our applications aren’t compromised. So, how do we actually secure our gems?
00:17:24.480
Now, let’s talk about countermeasures. For starters, when I say countermeasures, I mean, the first countermeasure we can get into is combating typo-squatting and combo-squatting. You're probably thinking, what can save me from my erroneous typos? Like a Grammarly for gems?
00:17:43.500
The answer is: yes, kind of! In the sense that Grammarly uses a defined list of words and grammar rules, you and your team can create a list of vetted gems. That list can essentially become your allow list.
00:18:01.300
So, you’ve got an allow list, but how do you enforce it? One option is to create a GitHub action that verifies whether the gems added to your Gemfile are allowed or not. If you're curious on how to get started, we built a scrappy prototype to help you get the ball rolling.
00:18:21.280
But to be clear, it is far from feature complete. However, we hope it can serve as inspiration. In addition to the allow list, we also recommend practicing the 'campsite rule'—the one that says leave things better than you found them.
00:18:33.779
So, if you come across a suspicious package, just report it! How about dependency confusion? How do we counteract that? Last year, the Bundler team released a fix for how it prioritizes sources.
00:18:49.920
This was originally fixed in version 2.2.10, but we recommend being on at least 2.2.22 for all the best goodness related to this issue. If you look at the bottom of your Gemfile.lock, you'll see a 'bundled with' version.
00:19:02.170
If it's not at least 2.2.22, update your Bundler and run 'gem install' again. With this fix, you can now specify a block source in your Gemfile, enabling Bundler to prioritize resolving the gems and their transitive dependencies to the specific source listed.
00:19:14.000
Even if you don't explicitly list the sources, once you upgrade and run 'bundle install', your Gemfile.lock will change from the old format to a new format where the remotes are listed one by one.
00:19:26.500
Each gem will then know exactly where to pull that gem and its dependencies from, making it consistent on every bundle install. So, you’ll never have to worry about Bundler being dependent on confusion again.
00:19:37.359
You should also check the Bundler version used to generate your Gemfile.lock. If it's lower than 2.2, you should update to get this protection. This was a problem we had to solve at scale at Shopify since we had many apps to update.
00:19:46.680
One of our teammates wrote a blog post on how this problem was tackled, and if you're interested, you can check it out on the Shopify engineering blog.
00:20:05.760
Now let's talk about how to counter repository compromise, as Betty mentioned before. Someone hacking into RubyGems and changing the gems directly is not a super common scenario, but it is very destructive.
00:20:23.020
Luckily, there are several things that can mitigate that risk. One is gem signing. Maintainers cryptographically sign their gems, and then when you install, you can use high or medium security flags to verify the signature and ensure the gem hasn’t been changed.
00:20:37.160
This feature has been built into RubyGems for quite some time now, but unfortunately, it’s cumbersome to use and so effectively, no one uses it. There’s an RFC open on RubyGems right now to make signing easier.
00:20:48.779
If you’re interested, please check that out and feel free to share your opinion. Last but not least, let's talk about account takeovers. There are two parts to this problem.
00:21:04.020
Firstly, how can you assess your risk exposure to account takeovers related to your application's dependencies? Secondly, how can you contribute to account integrity to improve intent related to account integrity on RubyGems?
00:21:17.320
When discussing account takeovers, I’d like to stress these four words: enable multi-factor authentication! It’s such an easy thing to do, and it’s incredibly effective.
00:21:31.660
How effective? Microsoft released a report stating that MFA blocks 99.9% of automated account takeover attacks. That level of protection is more effective than the awesome N95 masks we're all wearing right now.
00:21:49.060
We may not have N99, but we’ve got MFA 99, right? Who wouldn’t want a piece of that? Now, back to our two key questions about gem maintainers. Is there a way for us to determine whether their accounts or any accounts have MFA enabled?
00:22:03.160
If you go to RubyGems.org and check out Aaron's profile, you probably won't see any indication of his MFA status. This lack of visibility is by design. While we might have the best intentions, publishing MFA status publicly is like letting a thief know which home is easier to break into.
00:22:20.840
We don’t want to advertise which home or account is easier to break into, and that’s why MFA isn’t displayed on profile pages. That said, gems can be configured to require all maintainers to enable MFA before performing privileged actions.
00:22:34.620
If you're a gem owner and want to set that up, you can check out the documentation. If you're not, but want to verify whether this configuration is actually set up to enforce MFA on a gem level, you can find that information by navigating to a gem's show page.
00:22:53.219
For example, with Rails, you’ll find that it requires its maintainers to have MFA enabled. Additionally, the team Ashley and I work on is very invested in the health of the RubyGems ecosystem.
00:23:05.000
Recently, we've submitted an RFC and had it accepted by the RubyGems team to roll out an MFA enforcement policy for the top 100 most downloaded gems. This policy aligns with what other package managers are doing.
00:23:19.120
For instance, npm has already rolled out a similar policy, and GitHub recently announced that they will require all users to enable MFA by the end of 2023.
00:23:36.380
We’re excited that, in the near future, maintainers of some of the most popular gems will be protected from the Aaron Battersons of the world!
00:23:50.740
Now, how can you personally help improve account integrity on RubyGems? If you have a RubyGems account, enable MFA. In the words of my bestie Beyoncé: 'If you like it, then you better put a ring on it.' If you like your account and want to keep it, you better put a figurative ring on it.
00:24:10.680
We are big proponents of MFA—so much so that we are riding the MFA train all the way home! So we’d extend the ask of enabling MFA to not just your RubyGems account but to everything. MFA all the things: npm, GitHub, Shopify!
00:24:23.660
If a service offers MFA, enable it! For those of you who do end up enabling MFA on your RubyGems and/or GitHub account, Ashley and I want to give you this adorably limited edition sticker.
00:24:37.800
This is to celebrate our collective love of MFA. So, in closing, if you loved our talk or our amateur doodles, tweet at us and, I don’t know, CC Sandy Metz if you want to.
00:24:51.740
We honestly would love to hear from you, though. Tell us if you learned something new or how much impact we've had. We can shamelessly collect receipts for our next impact review.
00:25:05.000
Thank you so much! Remember, MFA all the things! Come find us for stickers, and we’ll share a final Wi-Fi to all of you.