Security
Hacking with Gems
Edit
#hacking
#rubygems
#security
#dependency-management
#malicious-code
#ethics
#ruby-on-rails
Summarized using AI

Hacking with Gems

Ben Smith • June 24, 2013 • Earth

In the talk titled "Hacking with Gems," presented by Ben Smith at the Ancient City Ruby 2013, the speaker explores the risks associated with trusting third-party gems in Ruby applications. The central theme revolves around the potentially malicious behavior that could be embedded in Ruby gems, and Ben Smith aims to raise awareness about the importance of scrutinizing code dependencies in software development.

Key points discussed include:

- Introduction to the Topic: Ben differentiates between ‘hacking on gems’ and ‘hacking with gems,’ the latter focusing on how malicious code could be introduced into applications through trusted dependencies.
- Confession of Non-Expertise: Ben acknowledges his lack of formal security training while explaining concepts in a manner accessible to a general developer audience.
- Concept Demonstration: The talk features several hypothetical gem examples that appear innocent but include harmful functionalities, demonstrating how easy it is to disguise malicious code:
- Awesome Rails Flash Messages: A gem that seems to enhance Rails flash messages but secretly harvests user credentials by logging sensitive data to a web service.
- Net HTTP Detector: A gem that detects HTTP requests made by other gems, potentially logging sensitive data transmissions.
- Better Date to US: A gem that claims to modify date formats but could actually compromise application source code.
- Be Truthy: An empty gem that pretends to offer functionality but executes harmful code upon installation, allowing remote access to compromised systems.
- Viral Malicious Gems: Ben explains the potential for maliciously coded gems to spread throughout the Ruby community via trusted dependencies. He prescribes measures to prevent such occurrences:
- Caution in Dependencies: Developers should thoroughly vet gems before installation using tools to track network activities and filesystem changes.
- Sign and Trust your Gems: Leveraging signing mechanisms using public/private keys to ensure the integrity of the code being used.
- Monitoring and Sandboxing: Encouraging the use of isolated environments to test gems before full integration.
- Conclusions and Advice: The talk finishes with a warning about the risks of installing untrusted gems. Ben strongly advises developers to:
- Favor writing custom code over relying on external gems.
- Monitor the behavior of gems during and after installation to detect any malicious activity.
- Use tools like Coal Mine Canary to identify and respond to security risks efficiently.

Ultimately, Ben hopes to empower developers to critically evaluate the dependencies they incorporate into their projects, fostering a more secure programming environment within the Ruby community.

Hacking with Gems
Ben Smith • June 24, 2013 • Earth

What's the worst that could happen if your app has a dependency on a malicious gem? How easy would it be to write a gem that could compromise a box?

Much of the Ruby community blindly trusts our gems. This talk will make you second--guess that trust, and show you how to vet gems that you do choose to use.

Ancient City Ruby 2013

00:00:00.680 My name is Ben Smith. That's actually my real name. I've been told before that I should use some hacker alias for this talk, but honestly, Ben Smith is pretty common as it is.
00:00:05.920 I'm actually going to be talking about hacking. A lot of people say, 'Oh, you're talking about hacking on gems,' right? I'm actually talking about hacking with gems. I'm talking about getting credit card information or rooting boxes.
00:00:17.119 Just to be clear on that, I've actually thought about renaming my talk to 'How to Get Rich Quick and Maybe Not Go to Jail.' With that in mind, my legal counsel has asked me to read you the following statement: I cannot be held accountable for anything that might happen to you if you install my gems or anyone else's gems.
00:00:30.519 So real quick before I get started, I want to tell you a little bit about who I am. I'm a Rails developer. I've been doing Rails since 2006, I work at Pivotal Labs, and I live in Boulder, Colorado. Now, just a bit about what I'm not: I'm not a security expert at all.
00:01:03.720 I have a fascination with security, but I don't have any background or training. I like going to DEF CON every year, but 90% of the content goes way over my head.
00:01:10.200 As I'm showing you these gems and these examples, keep in mind that they're written by a complete noob. Anyone could do this—it doesn't take an elite hacker to do any of this stuff. That being said, please don't try this at home. It would actually be better if you just forgot everything here that I'm about to tell you, except for the last few slides.
00:01:39.000 This picture actually looks like a horrible idea. The funny thing is, a lot of these gems are kind of interesting and fun. You start thinking about it, you're like, 'Oh, I could do that, that's kind of interesting,' and then you have another idea, and you go like, 'Oh, I could actually do this,' but don't, just don't. This talk is a little bit conflicted. On one hand, I'm telling you how to avoid bad things happening to you, and on the other hand, I'm showing you how to write malicious gems.
00:02:19.360 With knowledge comes power—use that wisely. So how did this all start? A couple of years ago, I had a client come to me and he wanted to know what was in his app. He wanted to know what dependencies were there. He just wanted to have some sort of handle on what was going on. He wasn't super technical, just a little bit, and so I showed him the gem file. That satisfied his curiosity, but it got me wondering, 'What's the worst that could happen? What could a malicious gem do?'
00:02:38.840 What would be the worst possible thing that could happen to that client? Rails has somewhere in the order of 39 different dependencies, so what if one of those dependencies in Rails was actually malicious? If you're the visual type, this is a dependency graph for Rails. A standard Rails project right out of the box is what you get, so what if something in there was bad?
00:03:10.120 I was thinking that the worst thing I could imagine happening was that customers' private data could get compromised and all their customers could lose faith in the product. Then you know the whole thing collapses, so I thought, 'How hard would it be to write that gem?' So I decided to write this gem. I called it 'Awesome Rails Flash Messages,' and I decided right away that no one would install a gem that was openly malicious.
00:03:43.680 Thus, I had to create something that appeared to do something different. It's arguable whether or not this gem actually does anything useful at all, but at least it's just not hacking you. What it does is it takes typical Rails flash messages and makes them more awesome—that's all it does! All caps, exclamation points, someone’s littered in there. Totally useful, you should definitely use it, but there are some side effects.
00:04:28.880 If you cracked open the source of this gem and started poking around, you might see something like this. This is kind of odd, right? If this evaluates to true, then it writes your params to something called 'development.log' in your public directory, which doesn't sound quite right. On top of that, it takes your params and posts them off to a web service somewhere.
00:05:02.680 If we go back, this line, 'if params to us matches something,' is actually looking for a password. So if the params your Rails app got includes like 'password' somewhere in there, then it's going to do all this stuff. So anytime you have a user logging in or signing up and passwords are passed to it, it's going to write something like this to this development.log file: clear text emails and passwords.
00:05:36.840 Since it's in a publicly accessible place, I can just go to your app's development.log and just see it all. Of course, I don't even have to do that; it's also posted somewhere else on the internet, so no problem there, and now I have usernames and passwords.
00:06:00.480 There's an internet meme you're probably all aware of: step one, you do something; step two, you do something else; step three is always these question marks—you never know what you're supposed to do there—but step four is always profit. That's where we want to get. So if I were to fill this out with some real examples, here it is: step one is write a gem that does something. Alright, I got that. Step two is add code to harvest emails and passwords.
00:06:50.640 Check use emails and passwords on banking websites to transfer some money, and I got profit. There you go, no problem! But I think there's like one more step that needs to happen, and I think you've got to flee the country. Yeah, I don't think the FBI likes you stealing money from people. I'm not an expert on the subject, but according to the internet, the US doesn't have extradition laws with 54 different countries.
00:07:28.480 So if you want to come visit me later, I'll probably be at one of these places. But before I leave, I thought, 'Well that was easy, what else can I do?' So I decided to write another gem. I called it 'Net HTTP Detector.' I wanted to detect the hack from my Awesome Rails Flash Messages. This gem logs calls to Net HTTP.
00:08:02.639 If I were using that Awesome Rails Flash Messages gem and I added this into it, I would get these logs. Basically, it's not formatted well or anything, but it's basically telling me that something got posted out to some Heroku app on the internet, and I should be aware of that. If you look at how it works, it's just defining post form and allows you to log what's going on.
00:08:30.560 It also defines a valid post form, so if you wanted to do some get or post on your own, you could do that without it having to log everything. But it does one more thing. In one of the files somewhere in this gem, the very last line grabs some code from somewhere on the internet and deals it.
00:09:02.320 So what does that code do? Well, it's a little before filter that looks for this 'DB console' param, and if it sees it, it does some ActiveRecord type of stuff. So let's look at how that affects our app. If you take a normal Rails route, such as user sign-in or any other route that you have, and you add this little 'DB console' param, you'll get a nice interface with database access.
00:09:44.000 Now you can do things like show all the users, make yourself an admin, or create a database user. The moral of the story is to be careful of wolves in sheep's clothing. If we go back to our five-step program again, we can fill this out. I wrote a gem, I added code to get access to the database this time, and with that database access, I can hopefully get some personal info on people, apply for a boat loan, profit, and then get out of here and go to a beach somewhere.
00:10:14.560 When I was looking up these countries, I made a low Google Map, but right when I typed into Google 'Which countries do not have extradition laws with the United States' and hit enter, I realized I probably should have done that on a public computer. Now I am on some government watchdog list somewhere.
00:10:49.640 Anyway, so I got database access—awesome! And I thought, 'Well that was easy, what else can I do?' So I wrote a third gem called 'Better Date to US.' It claims to strip extra white space off of the to-sd formats in Rails. It's something that annoyed me, but that's not the only thing it does.
00:11:33.680 You'll notice a theme here: none of these things actually do what they claim to do. So what does it actually do? Well, it calls this method 'set date formats for' and passes the Rails environment and the Rails route, which is strange. You probably don't need that for date formats.
00:12:01.040 So let's see what that method does. It's a bit hard to tell what's going on there because that's compiled C code. This gem doesn't have the source; it's pre-compiled for your platform and it's not distributed with the source at all. Since I'm nice, I'll show you what the source was before it was compiled.
00:12:53.920 This is what it looks like. Can anybody tell me what that does? If you edit this gem and deploy it into production, you'd end up with something called 'assets.tar.gz' in your public directory. If I were to download that and extract that tarball, you'd see something like this, and that's your Rails source.
00:13:05.920 That's the source code of your application. So now I have all your source code, all your intellectual property. A little bit of truth about this: this gem doesn't actually work; it could, but I'm pretty lazy. It's what you call a fat gem, which is a gem that's pre-compiled for specific platforms and can be distributed without the source code.
00:13:50.760 But it took me more than a couple hours to try to figure out how to do it, and I was like, 'Ah, this is hard,' and not only that, but what am I going to do with your source code? Can I sell it to the Russians? Is that still a thing or is it the Chinese now? What do I put that on eBay? So I thought, 'Ah, screw all that, that's too hard, what else can I do that's easier?'
00:14:37.760 I wrote another gem called 'Be Truthy.' How much truth do you think this one will have? This is what it does: ARPC matchers are kind of liars—it's a pet peeve of mine. It doesn't assert against true or false; it's truthy or falsy. What this gem does is add some matchers that make things like 'user.new.should be true' fail because that's not actually true. True should pass; 'user.new' should be truthy. Yeah, that sounds good. So this fixes that little pet peeve of mine.
00:15:35.559 But let's talk about what it actually does. It actually doesn't do anything; it's just an empty gem. There's no functionality there. It could have been because I didn't get around to finishing it, or probably because I got tired of writing useful code and just went straight to the hack. I'm lazy, like I said. But if there's no code there, where is the hack? So if we poke around, the source looks fine.
00:16:38.720 The source code looks good; there's actually, like I said, no code there. So if there's no code, what's the catch? When you install this gem, it says 'building C extensions, this could take a while.' What this means is there's usually output when you're compiling some C extensions, but if we look back, there weren't any C files, and you probably don't need C to build an RSpec matcher.
00:17:27.440 So let's dig some more. If we pop open the gem spec of this gem, you see this line: 'gem.extensions = rake file.' What this is doing is saying to run the rake file at install time. Keep this in mind: this is at install time. This isn't when you require the gem; it's not when you execute the code; this is at install time.
00:18:13.760 But if we look back, there is no rake file. So where's it go? This stuff happens at install time. So if you ran gem install or bundle with this gem, something's already happened. So if we look at the source before you install it, this is what it would look like; it's a little bit different. We have a rake file and we also have this 'temp.rb' file.
00:19:05.840 So what does the 'temp.rb' file do? Remember, this is a rake file that runs at install time. It copies this 'temp.rb' file into the user's home and hides it as '.temp'. It then proceeds to alias 'sudu' in your bash profile and points it at this '.temp' file, and then it removes itself. The last line of the rake file says 'remove yourself,' and then it's gone. So right after the gem gets installed, the file's gone.
00:19:50.240 And the only evidence that anything happened was the one line in the gem spec and the one line of output when you did the 'bundle' or 'gem install' command saying it's compiling C extensions. So the question now is, what does sudu do? I aliased it, right? It points to that temp file. So what does it do? Well, it kind of does what sudu normally does: prints out a warning, grabs a password, and then runs the real sudu with whatever you wanted to run it with. Of course, it grabbed your password.
00:20:47.080 Now I can do more stuff, and it does: it enables SSH, creates a user, and sets their password and tells some web service that there's a box that's ready to be SSH'd into. So now what can I do? Yep, I've rooted your box.
00:21:36.640 So what's the takeaway from all this stuff? Don't use gems. Don't use my gems. I started presenting these for the first time at a lightning talk at Mountain West a year ago, and the biggest takeaway from everyone was, 'Ah, I'm never going to install any of your gems. I don't care what it is; I'm never installing anything that you write.' I was like, fair enough!
00:22:17.760 I've actually garnered a reputation now where literally no one will install my gems. I wrote a gem, and I asked friends, like, 'Hey, could someone test this out for me?' and the only place I installed your gem is in a VM.
00:22:57.360 Sounds like a challenge to me! So how can I get you guys to install my gems? You guys are smart; you're only going to install gems that you trust. You're obviously not going to install my gems, but you will install gems you trust, like Rails, RSpec, Sinatra. Those all seem safe, right? Everybody uses those. You have to trust all their dependencies as well, but it's probably a safe bet.
00:23:44.720 So how can I add my code to some already trusted gems? If you're not going to install my gems, then I might as well just skip that whole step. I don't think this is going to be as easy as too many pull requests, but I had this thought and I went back to my Be Truthy gem. I did one more thing: at install time it grabs your gem cutter credentials and your list of installed gems and posts them to a web service. Now I own your gems!
00:24:37.040 Hopefully, you guys are some trustworthy gems because now I'm about to clone your repo, add my own malicious code to your gem, build your gem, and push it up to RubyGems, and the kicker is you won't even know that this is happening because there's no notification that a new version of your gem has been pushed! You would have to go onto rubygems.org and check the version number or update your own gem, and who does that?
00:25:41.680 I was able to do all that just by grabbing your little gem cutter API key from the publicly readable gem credentials file. Now, do people trust your gems? It's not about me anymore; it's about you guys. Do people trust your gems? Do people who install your gems have trustworthy gems and so on and so forth? It becomes viral! I add my code to your gems, and your friends install your gems, and it goes bigger and bigger and bigger until I hit one of the dependencies of Rails.
00:26:43.040 But there's still one problem: how do I get the first set of people to install my gems? How do I bootstrap this entire viral aspect? I can't really ask my friends to install my gems anymore; that doesn't work. I tried being popular for a while, so I wrote this social experiment gem and I wrote a script that constantly downloaded it all day to keep it at the most downloaded today gem. It didn't work, though! I got one download and actually this leaderboard is gone now, so that doesn't work.
00:27:29.360 So how else can I get people to install my gems? Conferences are good, people talk about libraries and gems at conferences. I've seen several libraries thrown around at this conference. WebMock was mentioned a couple of times. Respect given to that one—it can't be harmful! But giving a talk is hard, and I'm lazy, so how could I get you guys to install my gems without actually giving a talk?
00:28:50.240 So I came up with an idea. Have you guys seen these? You guys seen these? Who did that? I did that! That's my gem! Don't worry, it doesn't do anything harmful except for harm your ego. I did the same thing at Aloha RubyConf and I got 5% adoption, which was decent. Who installed my Ancient City Ruby gem? A few people? I know there are more of you out there.
00:29:38.440 That's who installed it! So I got 7.5%. I'm doing a little bit better; I'm working my way up. The question is, are any of you guys gem authors? If I had done this for real, you know I could have owned your boxes, owned your gems, and this is how I could have started this whole viral aspect.
00:30:30.840 I did kind of trick you guys. I made that gem, and I said, 'Oh yeah, there's a free $50 off PIV tracker gift card,' which is true. I'll give those away. I might pay for them with your own credit cards, though; just watch for that on your statement.
00:31:35.920 So what happens now? Let's say someone did do this for real, you know, at this conference or another conference. What would happen? My guess is, RubyGems would probably go down after they realized other people's gems are getting compromised. Heroku deploys, that'll go down, and I would end up on a beach somewhere. This is basically what happened a couple months ago with the YAML vulnerability getting exploited on RubyGems.
00:32:37.040 RubyGems went down, but only for a moment, and then they came back up again. They said the only thing you can't do is push. You can continue to download stuff, which, to be honest, is kind of weird because they were worried that gems were being changed, but they still allowed people to download and install gems already up there.
00:33:13.680 I think they should have just stopped everything—stopped all downloads. Heroku deploy went down for a minute, and then they said, 'Alright, you guys can deploy, but you're going to have to use your own custom build pack.' They basically made it a little bit harder for you and said you can do it, but you should be aware that there are issues here.
00:33:59.760 The recovery from that attack was that RubyGems knew when the exploit started. They knew at what time, and they were able to compare gems prior to the attack to ones that were current. They said, 'Prior to the attack, these gems were good, and then after that, we don't know what to think.' So they compared them and said, 'Alright, nothing's been changed, everything's good now!'
00:34:55.920 In my case, if I'm stealing gem cutter credentials, there's no way to really do that; I'm basically stealing usernames and passwords. There's no single point in time that could occur over weeks or months, and it would be very hard for RubyGems to say, 'Well this is when it started, and we can say that everything before this is good and everything after this is bad.' It'd be quite difficult.
00:35:32.560 So what now? How do we keep all this from happening? Well, some of the gems I talked about have easy things you can do to detect that they're doing something crazy. The Awesome Rails Flash Messages gem posted out to a web service, so there's network traffic there. You can use a tool like Little Snitch to monitor your network traffic.
00:36:24.040 In this case, you can see that iTerm is running my Ruby process that made a post out to some Heroku app. It's fairly easy to monitor this stuff and keep an eye on it to make sure nothing weird is going on. The Be Truthy gem is the one that modified your bash profile, right? So this made file system changes, and in this case, FS Eventor is a great tool for monitoring your file system.
00:37:14.560 In this case, it's showing that there are changes to the profile and the .temp file, and what you can do is you can start this thing up right before you install a gem or right before you do a gem install and see what files it touches. The other thing with the Be Truthy gem was the exploit was triggered at gem install time.
00:37:59.520 So never gem install gems that you aren't very comfortable with or that seem non-malicious. Don't gem install from strangers! What you really want to be doing if you don't trust a gem is fetching it first. When you do a gem install, it downloads the gem and installs it immediately. If you do a gem fetch, it will download the gem, and it won't install it—it'll just sit there.
00:38:54.440 Then from that point, you can unpack it and see the contents, and that will allow you to see the malicious code before the install script runs. Remember, the rake file got removed, right? At that point, you're already screwed. Doing a gem install is basically like downloading something from the internet and just saying run it right away.
00:39:32.080 Or if you're not from Windows land—does this look familiar to anybody? Everybody's done this. I've done it. It's executing arbitrary code from the internet. Of course, in both these cases, we choose to trust RVM, Dojo, or FilePlanet.com or whoever else, but with gems, you're basically choosing to trust all of RubyGems.org, and anyone can put gems on RubyGems, including me.
00:40:40.280 If you want to trust the gems that are on RubyGems.org, you should really be installing them with 'DASP high security.' This requires signing of the gems, and in theory, what this gives us is the ability to track back the author. It doesn't necessarily mean that the gem isn't going to be malicious, but it should allow you to get back to that author, get back to a real person that wrote that gem.
00:41:34.560 Of course, that requires a high level of adoption to actually work. If you try to run this, if you try to install Rails with 'DASP high security,' you basically won't get anywhere because the support isn't signed. So what we really need to do is start building our gems with Sigs.
00:42:31.040 It's very simple. Use a basic public/private key: you sign your gem with your private key, distribute your public key, and anyone who has your public key can install and verify that you're the one who built that gem. Of course, you have to make sure that your private keys don't get compromised the same way I grab the gem cutter credentials, but that's another story.
00:43:21.760 There are a couple of projects around gem signing out there. They are attempting to beef up the whole system; they haven't got a whole lot of traction yet, but it is starting to move in a good direction. Another completely different option to all this is doing sandboxing. Create a sandbox, like a VM where you can install and test gems in an environment that can get trashed without having to worry about it.
00:44:21.680 Someone could also fork RubyGems.org and actually send notifications when new versions of your gems get pushed. That way, at least when someone does own your credentials, you'll get notifications that they are pushing new versions of your gem. We could write tools to detect malicious code, and this would be awesome if this was integrated into RubyGems because then every push could potentially flag a gem and say this might have malicious code.
00:45:02.560 This is a hard problem to solve, but still a possibility. Private gem repos are nice; you could literally go through every line of every dependency and vet it and make sure that there's nothing malicious. Throw it in a private repo and just install stuff from that private repo. I've heard that there are a couple of companies that are out there doing this now.
00:46:09.600 There’s potentially an opportunity for a service to do this where you pay someone to go through and manually vet all your gems. Of course, all this stuff is really hard. There's no silver bullet here; there's nothing that we can do that will protect us in all cases.
00:47:00.480 But the one thing I'd like everyone to do is: don't try this at home. The Ruby community is very, very kind and supportive, and let's keep it that way. But maybe take away from the talk the following few things: don't install gems you don't need to favor writing code yourself rather than installing some gem that got suggested on some Stack Overflow post.
00:47:52.600 If you're just trying to fix a small issue, just write it yourself. Pay attention to what your gems do. If you have some RSpec matcher gem, it shouldn't have C extensions for example. Monitor your system; Little Snitch and FS Eventor were the couple that I mentioned to do this.
00:49:07.680 Of course, read the source because that's where the rubber meets the road. There's one more gem, and this one's not malicious, I promise. You can choose to install it or not: Coal Mine Canary detects potentially a dangerous environment. It kind of does the things that I showed examples of—like modifying your bash profile, trying to grab your gem cutter credentials, or post out to a web service.
00:50:05.520 Basically, it tells you when you install it, 'It's doing some pentesting,' and here's the results, and it tells you what it was able to do. So in my case, it was able to post to a web service, write to my bash profile, grab my gem cutter credentials, and read my SSH keys and known hosts, and I killed all of my canaries. So try that out; it's kind of fun!
00:51:03.880 The last thing I'd like to say is that conferences always inspire me. I come to conferences; I learn about new things. I hope that this talk didn't inspire you to write malicious gems, but I hope it was at least entertaining and eye-opening. Thank you.
Ben Smith
@binji
Explore all talks recorded at Ancient City Ruby 2013
+1