00:00:04.960
You have to wear shoes up here. Can I go barefoot just to increase the hippie stereotype? Is that cool? Yeah? All right.
00:00:11.679
At my office, there are two rules: you can't drink beer before 4:30 PM, and you have to wear shoes. I try to take advantage of that whenever I can, plus it's cold in Colorado right now. Might as well.
00:00:24.160
So, I'm here to talk about hacking with gems. It's not hacking on gems or hacking like you typically think about at hackfests or anything. This is real hacking. This is about getting credit card numbers or passwords. A lot of people confuse that.
00:00:37.280
I was thinking maybe I should just change the title of my talk to 'How to Get Rich Quick and Maybe Not Go to Jail.' I don’t know, listen to the talk and tell me what you think. Maybe give me some feedback. You'll see why later.
00:00:48.960
A little bit about who I am: Like Josh said, my name is Ben Smith. I usually just go by Ben, although Ben Smith is the most common name on the internet. I try to make it a little bit unique by going by Benjamin. I've been doing Rails since version 1.16, which is a while now.
00:01:06.079
I work at Pivotal Labs, a Rails consultancy in the United States. We have offices in San Francisco, Boulder, Denver, and New York. We used to have an office in Singapore, but unfortunately, we sold that one. However, we are opening new offices in LA, Boston, and London. Hopefully, eventually, we'll have one down here.
00:01:22.640
So, if you guys keep having Ruby conferences and support startups and stuff, you might see us down here. I would love to come and work in Australia and get to surf every day; that'd be great. As I mentioned, I'm in the Boulder office, right in the middle of the United States near the Rocky Mountains, where it's just dumping new snow.
00:01:40.000
Every time I go out of town, this happens, and my friends get to enjoy this crazy powder. But that's fine; I'm actually enjoying it here. Here's a picture of me from last year. If you enjoy skiing, come to Colorado—it is just the best.
00:01:58.640
But yeah, it's great to be here. It's nice and warm, and I get to wear shorts for the first time in the last four months. Now, a little bit about what I'm not: I'm actually not a security expert. I have a fascination with it—I go to DEF CON every year—but 90 percent of the content goes way over my head.
00:02:22.560
I'm just a Rails developer; I'm a noob. The reason why I'm telling you this is not because I want you to get up and leave, thinking this guy doesn't know what he's doing. It's because when I show you this code and these gems, keep in mind this was all written by a complete noob. Anyone can do this.
00:02:45.599
That being said, please don't try this at home. It would actually be better if you forgot everything I'm about to tell you, except for the last few slides. And this image doesn't really do a good job of explaining it; it actually looks like a horrible idea. Who would want to do that?
00:03:03.120
But when you see some of these examples, they're actually kind of cool. You might think, 'Wow, that's kind of neat; I should do that.' But don’t do that. So, you know, with this knowledge, use it wisely.
00:03:20.080
Let me talk about how I got started on this topic. A couple of years ago, a client asked me, 'What's in my Rails app? What dependencies do I have?' He was a bit worried about having too many dependencies that he didn't know about and wanted to have some sort of handle on that.
00:03:37.360
So, I showed him the Gemfile.lock. That basically satisfied his curiosity, but it sparked my curiosity, and it got me wondering: What's the worst possible thing that could happen? What if there was a dependency in here that was malicious? What could it do?
00:03:56.799
If you generate the dependency diagram of all the gems in a standard Rails app, here’s what it looks like: there are 39 gem dependencies. This is a visual representation of it. So, what if one of those 39 dependencies had something malicious? What’s the worst possible thing you could imagine could happen?
00:04:15.599
I thought, well, maybe this client, who has customers, could have their private data compromised. The customers might lose faith in the product, and then it falls over, goes downhill, and we lose that client. We don’t get paid, and all this other bad stuff.
00:04:39.360
So, compromising data sounds pretty bad. I thought, how hard would it be to write a gem that did that? So I decided to try, but I realized that no one would just install a gem that is openly malicious and meant to hack you.
00:04:51.199
Therefore, I had to come up with something that was not malicious—something maybe useful. So, I wrote this awesome Rails flash messages gem. It's arguable whether or not this is actually useful, but at least it doesn’t just hack you.
00:05:05.840
If you take a Rails app and you have some flash message, like 'Beer was successfully created,' and you add this gem, it changes it to 'BEER WAS SUCCESSFULLY CREATED!!!1111!'. So much better—completely useful! But it also has some side effects.
00:05:30.720
If you poke around the code, you might notice some odd things, like this line of code. If this line evaluates to true, then some more code runs. It writes your params to something called 'development.log' that's located under your public directory.
00:05:56.479
In addition, for good measure, it also writes your params and posts them out to a web service somewhere—not sure where, but that's interesting. This line of code is basically saying that if your params contain 'password,' any time the Rails app receives a request with that param, it writes it to that development.log file and sends it to some web service.
00:06:19.280
So in your development.log file, you could end up with clear text usernames, emails, and passwords. Since it's under the public directory, all I have to do is go to your app's /development.log, and I can just view this. Of course, I don’t actually need to do that because I’m posting to a server that I control as well, so I can see all emails and passwords there in plain text.
00:06:49.280
They're not filtered, and they're not hashed passwords in the database—these are what the user entered in their browser. That's kind of interesting, right? There's this meme on the internet that you guys might be aware of: it goes like this: Step one, do something. Step two, do something else. Step three, is always question marks (I don’t know why that is), and step four is always profit.
00:07:05.680
So, if I were going to fill this in with some concrete actions, it might look like: Write a gem that does something. Check. Add code to harvest emails and passwords. Check. Instead of question marks, we can actually use those emails and passwords to go to banking websites and try to transfer funds. You can probably safely assume that people are reusing their passwords, so you might get some traction on this.
00:07:32.960
And then, of course, step four is profit. But after step four, I think you actually need step five here, which is flee the country. I don’t know what the laws are here, but in the States, if you did this, you’d probably go to jail. You might want to leave. I also don’t know a ton about extradition laws, but I do know that there are 54 countries with which the United States does not have extradition treaties.
00:08:01.680
So, if you want to visit me, I’m probably going to be in one of those countries. I was actually Googling to get this list of countries the other day and wondering who would actually put into Google, 'Which countries do not extradite to the United States?' Well, hopefully, I’m not on some watch list somewhere, because the only people who would Google that are obviously criminals.
00:08:20.320
Anyways, enough dreaming of moving to some small island off of India. Let's get back to the Ruby code. I wrote that first malicious gem and thought, ‘Well, that was easy. What else can I do?’ So I wrote another gem. This one is designed to detect my previous hack.
00:08:39.760
This gem detects calls to Net::HTTP and logs them. If I was using that awesome rails flash message gem, in standard out or in the development log or production.log, you’d see something like this, basically telling me that my params got posted out to some web service. This would alert me that something was fishy.
00:08:57.600
If you take a look, the code defines `post_form`, which logs out whatever is going on, and also defines `valid_post_form`—just in case you wanted to make a call out and didn’t want it to log the fact that you’re using Net::HTTP. But it does one more thing.
00:09:20.480
One of the lines in one of the files of this gem, at the very bottom, just in one line, is going and grabbing something from the internet and then evaluating it. It’s actually using the `valid` get there so it doesn’t log it. What do you think that’s actually doing?
00:09:38.560
Well, if we look at the code, this is what it’s doing: it’s adding a little before filter; it looks for a param called `db_console`. If it sees it, it executes some sort of ActiveRecord command. So if we took a standard Rails route, it could be anything in your app, and you added the `db_console` param to it, you would end up with a nice little interface to your database.
00:09:56.639
Now, as an outsider on your website, I could do things such as 'show me all the users', or I could make myself an admin. Why not? I could create a database admin. So the moral of this story is just be careful of wolves in sheep's clothing. Don’t trust something just because it says it does something.
00:10:17.200
If you actually wanted to do network monitoring, ‘Little Snitch’ is a good tool. There are many tools out there for this, but this one's telling us that something is running my Ruby process, and my Ruby process made a call out to some Heroku app somewhere.
00:10:38.560
So you should use something like this if you want to monitor your system. But anyways, if you had used that gem, you would have given me database access. So what could I do with database access? Well, let's see: write a gem? All right.
00:10:56.960
Get database access? Yup. Apply for a loan to buy a boat.
00:11:05.040
Again, in the States, it’s super easy to get loans. You basically need a small amount of personal information, and then you can apply for car loans, boat loans, credit cards, or whatever you want. So, I would probably apply for a boat loan and sail off in the boat. Profit! My favorite step! And then flee the country.
00:11:20.000
Out of those 54 countries, 39 of them have beachfront property. If you have a boat, you’ve got to sail there, right? I’m probably going to end up in one of these countries. But before I do that, I thought, ‘Well, that was pretty easy. What else can I do?’ So I wrote another gem: `better_date_2s`.
00:11:43.839
What it claims to do is strip extra white space from the two state formatters that you get with Rails. This is kind of an annoyance for me, and I wrote this thinking, ‘That’s kind of nice!' Of course, that’s not the only thing it does.
00:12:05.760
If you’re detecting a theme here, the theme is that none of these gems only do what they say they do. They actually do something very bad. This one's no exception. It’s all a little bit of a misdirection. I’m not a magician like Keith, but I wish I was.
00:12:24.400
What does it actually do? It calls this `set_date_formats_for` method and passes in the Rails environment and the Rails route. That seems strange; you wouldn’t think you’d need that for date formats, but okay, let’s see what that method does.
00:12:41.840
No one can tell me what that does because it's a compiled C extension. This compiled C extension was packaged without the source code, so you have no way of seeing what it does. It only contains the compiled code. If you want to see what it actually does, you’d have to come and ask me, and since I'm here, I’ll show you.
00:13:04.800
Does anyone want to tell me what that does? If you're running this in production, you’d end up with this file: `assets.tar.gz` under your public directory. Since this is under your public directory, I could download it. If I did and extracted it, I’d see something like this:.
00:13:23.199
If that doesn't look familiar, that's basically your app's source tree—it's all your source code. To be honest, this gem doesn’t quite work. I could get a torque, but I’m lazy. This is known as a fat gem. It’s a gem pre-compiled for certain platforms. When you do that, you don't have to distribute it with source code because it doesn’t need to compile wherever it's installing.
00:13:43.840
There are tools that allow you to do this. I spent a couple of hours trying to do it, but it’s kind of hard. And on top of that, really, what are you going to do with someone’s source? Sell it to their competitors? Maybe, I guess, but I feel like that's a sting operation waiting to happen.
00:14:04.960
'Hey, here’s the source code,' and then handcuffs go on. Can I sell it to China? Do the Russians still buy secrets anymore? I don’t know. So I just gave up on that; it’s too hard. I wanted to find something easier.
00:14:25.600
So, I wrote another gem: `be_truthy`. This one’s my favorite! Not much truth in this gem, but here’s what it claims to do: I don’t know how many of you use RSpec, but one pet peeve I have about RSpec is that 'should be true' and 'should be false' don’t assert against true and false. They assert against truthy and falsey.
00:14:44.160
So if you say `user.new`, that should be true. That says, 'Yep, that’s green; that passes.' But `user.new` is not actually true; it should be true. This gem fixes that. But what does it actually do?
00:15:05.680
It actually doesn’t do anything. There’s no code—there’s zero code, nothing to see. Of course, I did this mostly because I was lazy. I had this idea, and thought, ‘Well, I could write the code,’ but I didn’t write it. I just added a hack, and that was it.
00:15:29.200
But, if there's no code, then where's the actual hack? This is the question, right? This is the confounding thing. If you look at the source tree of this gem, everything looks pretty fine, pretty normal. So if you look at `b_truthy.rb`, you’ll see there’s no code there: literally, there’s no code—nothing.
00:15:48.640
So what's the catch? When you install this gem—whether it’s via `gem install`, like I did here, or if you use Bundler—it will tell you that it’s building native extensions. This could take a while. Usually, that means it's going to compile some C extension.
00:16:12.800
But if we look back at the source tree, there’s no C code—there's nothing at all. You probably don’t need C code for RSpec matchers, so that should spark your interest and make you dig some more.
00:16:32.480
If you look at the gem spec, this line right here, `gem.extensions`, is usually used for compiling C extensions, but in this case, it’s saying 'at install time, run the Rake file.' Keep that in mind because that’s at install time.
00:16:55.679
This Rake file runs right away; it doesn’t wait for you to require the gem or execute the code. It runs right away when you install the gem. So, where’s the Rake file? Well, there is no Rake file. Wow, that’s kind of tough!
00:17:12.960
How many people read the source of the gems that they’re worried about? Maybe they try using a new gem and look at the source to see if there’s anything bad going on. Does anybody do that? A couple of people, maybe not very many. All right, well, does anybody use `gem fetch` plus `gem unpack`? Does anyone know what that is? A couple of people? All right.
00:17:42.880
When you do a `gem install`, it downloads that gem from RubyGems and installs it immediately. `Gem fetch` allows you to do that first step—the download step—without the install. Once you download the gem, you can unpack it and see what’s in that gem before the installation happens.
00:18:10.000
So in this case, we do a `gem fetch` of the gem, and then unpack it. We could see what the source is. Notice that it looks a little bit different than before. The installation process changes what’s going on—you'll see the Rake file, which is what we’re looking for, and there’s also this `temp.rb` file.
00:18:30.560
So what does the Rake file do? This Rake file runs at install time. What does it do? It copies that `temp.rb` file into your home directory and names it `hijackers.temp`. Okay, that doesn’t sound too bad yet.
00:18:50.040
Then, it adds an alias for sudo to your bash profile and points it to that `temp` file. Now, when you run sudo, it’s going to run something completely different—that could be bad. Finally, it actually removes itself; this removes the file.
00:19:10.240
So if you did a `gem install`, that file is just gone. You won’t see it anymore. The only evidence left behind was that one line in the gem install command or your Bundler command, saying 'compiling native extensions,' and that one line in your gem spec that said 'run the Rake file at install time.'
00:19:33.040
There are also some tools that could help you catch something like this. `FSEventer` is a nice tool to monitor file system changes. What I like to do is start this up before I install a gem. It will show you everything that it touches. In this case, you can see it touched `profile.temp`, which raises a red flag.
00:19:54.640
But back to our gem: at install time, it aliases commands.
00:20:11.440
This is the code that runs when you run sudo. It behaves like sudo usually does; it prints out a warning, grabs your password, and runs whatever command you wanted to run. That seems fine, but one thing it does is grab your password.
00:20:39.040
Now, I can do other things with your password. It does also enable SSH, which creates a user. Now I can SSH into your box. So, what’s the takeaway here?
00:20:55.760
I presented some of these gems for the first time over a year ago at a Mountain West Ruby conference and this was the biggest takeaway for people. Ryan big came up to me afterward and said, 'Ben, I'm never going to install any of your gems,' and to this day, people just don’t trust me.
00:21:16.560
I wrote a gem yesterday and needed someone to test it, so I tweeted, 'Hey, can someone please install this? I promise it’s not a malicious gem.' The only place I would install your gem is in a VM—this sounds like a challenge to me!
00:21:38.960
The other takeaway I heard a year ago was: should I just use Windows? None of your gems work on Windows. Well, I guess you’re right. Renee said yesterday that Windows is so much easier to set up Ruby on.
00:22:01.520
Combined with better security—maybe we should just switch to Windows, right? The one thing that really stuck with me was, like, 'Don’t install my gems.' That’s fair enough.
00:22:22.480
I had an idea: How many people here are gem authors? How many have pushed at least a gem to RubyGems? There’s a good number of you; that’s awesome! My question is, how can I get you guys to install my gems?
00:22:43.760
You’re all smart—you’re getting smarter and won’t trust my gems, but you do trust some gems, right? Which gems are trustworthy? Probably Rails, RSpec, and Sinatra.
00:23:04.120
I'm sure you think the big gems are safe to run `gem install` on. I wonder how I can add my code to some of those trusted gems. I don’t think this will be as easy as just submitting pull requests, but there’s gotta be a way.
00:23:26.960
With this thought, I went back to my `be_truthy` gem and added a little bit more code. This runs at install time too—grabbing your gem cutter credentials and your list of installed gems, then posting them out to a web service.
00:23:50.400
Now I own your gems! If you uninstall this gem, I am able to clone your repo, add some of my code to your gem, build it, and push your gem up to RubyGems.org, meaning the next person who installs your gems is running my malicious code.
00:24:11.920
There’s no notification to you. I could do this, and you wouldn't know unless you tried to update your gem or checked RubyGems.org to see the version of your gem.
00:24:29.680
Do people trust your gems? It’s not even about me anymore; it’s about you guys. Do people trust your gems? Do people who install your gems have trustworthy gems? Because once they install your gems, I’m going to own their gems too. It kind of snowballs and becomes viral, right?
00:24:49.440
You start wondering, ‘How many steps does it take before I hit one of the dependencies of Rails? Once I get that, then it’s game over.' But there’s still one problem: I need a good way to bootstrap how I get you guys to install some of my gems.
00:25:11.040
I’ve tried asking my friends to install my gems, but everyone who knows me says, ‘Nope, I’m not going to do that again.’ I could go to my friends and say, ‘Hey, you can ride on my boat, let’s go!’ But I don’t think my boat’s big enough.
00:25:39.199
So, what else can I try? I tried being popular. I wrote a social experiment gem and a script to automatically download it over and over again to keep it as the most popular gem of the day. But I only got one download; it was such a disappointment. This leaderboard doesn’t exist on RubyGems anymore, so that doesn’t work.
00:26:00.000
Conferences are a good place; people always talk about gems at conferences. Even here, I’ve heard about various gems. Charles mentioned the W bench library they wrote, and Constantine talked about many of his own gems.
00:26:21.440
That would work; you could talk about a gem at a conference and get people to install it. People do that all the time. But it still doesn’t help me. It would be great for those guys, but if I’m up here saying, ‘Hey, install this gem,’ no one is going to do it.
00:26:41.360
So, I came up with an idea: I wrote a gem for Aloha RubyConf in October and called it Aloha RubyConf. It didn’t do anything malicious; it just grabbed usernames from whatever box it ran on. I printed out cards and left them around the conference, by the food, by the swag, and on side tables.
00:27:03.680
Throughout the course of the conference, I got 5% of the people to install it, which ended up being eight people, but that’s decent. If I did this here and got the same level of adoption, I’d have about 19 boxes, but don't worry; I didn’t do that here.
00:27:24.640
The funny thing was, Aloha RubyConf was similar to RubyConf Australia—it was a dual track. While I was in one track giving my talk, I started talking about this gem, and people in the other room started tweeting about it, saying, 'Hey, Ben social engineered us to install this gem.' As they were tweeting, people started installing it.
00:27:46.240
So I was trying to be sneaky, but maybe you just need to find the right group of people and say, 'Hey, install this!' There were actually some surprisingly big names from our community who installed this gem from some very big companies.
00:28:07.680
This is one way to bootstrap: You could go to a conference and either talk about a gem or leave cards all over the place. So what happens now? Let’s say someone did this for real—what would happen?
00:28:27.680
Well, RubyGems would probably go down temporarily, and then it would come back up, allowing you to install gems from it, which doesn’t make a whole lot of sense—especially since they wouldn't know if gems were compromised. Heroku deploys would get restricted, and you’d be left wondering if the gems you’re about to download are malicious or not.
00:28:48.240
That’s kind of the situation that happened last month when the YAML vulnerability was exploited on RubyGems.org. I think if someone did this, that's what would happen. If I did this, I would probably end up on a beach somewhere.
00:29:08.960
The interesting thing about the YAML exploit from last month was that it had a very specific time when the attack occurred. Luckily, they could determine when the attack occurred and compare good gems against questionable ones to decide what was okay.
00:29:30.960
In this case, I’m stealing gem cutter credentials and pushing gems as if I were the author. It becomes much harder to say when the attack occurred, as it could take weeks, months, or days to implement.
00:29:52.960
Getting back to a state where all the gems on RubyGems are good would be hard if this happened. How do we keep this from happening? There are some options I don’t really love, but we can start by signing all our gems and hope that our private keys don’t get compromised like gem cutter credentials were.
00:30:14.160
We could start installing all of our gems using high security; this requires that all our gems are signed. Has anyone tried this? You basically don’t get anywhere, as ActiveSupport is unsigned.
00:30:36.720
That requires a pretty high-level adoption to actually work. We could start sandboxing, where you install a new gem or version in a safe sandbox where it can’t potentially trash the entire box.
00:30:50.400
You could fork RubyGems, adding notifications that say every time you publish a new version of your gems you get a notification. This would help when someone steals your gem cutter credentials.
00:31:13.920
We could write tools to detect malicious code. Ideally, maybe this would get rolled into the gem publishing process, but this gets really hard quickly, especially with eval code.
00:31:33.040
We could start creating our own private gem repos, so if you know some gems are good, you could put them there and only install from that repo. I've heard of companies that are starting to do this now. They vet their dependencies, check every line of code, and put them into their own private gem repo.
00:31:54.480
I think there's potential for a whole service where a company vets every version of these gems and says, 'If you point your source at our private gem repo, you’re guaranteed to get something safe.'
00:32:14.000
We could do all of this, and it would be helpful but also very hard. One thing I’d like everyone to do is to just not try this at home. All this stuff is easy and fun, but please don’t do it.
00:32:35.440
But maybe from this talk, you could take away a few things: Don’t install gems you don’t need. Favor writing code yourself, especially if you’re on Stack Overflow and you see the answer is, 'Just install this gem to fix a small problem.' Write it yourself.
00:32:54.720
Pay attention to what your gems do. If you have a gem that has RSpec matchers, expect it not to use C extensions. Tools like ‘Little Snitch’ and ‘FS Eventer’ are good for monitoring. Finally, always read the source, because that’s where you’ll find the nitty-gritty details.
00:33:09.840
One last thing: you probably won’t install this, but I wrote a gem called `coleman_canary`, named after canaries used by miners. They would take canaries down into the mine because they’re more susceptible to things like carbon monoxide.
00:33:30.080
So, I wrote this gem, which basically checks your environment and sees how dangerous it is. It tries to modify your bash profile, post to a web service, and fetch your gem cutter credentials. It tries to grab your private SSH keys and known hosts, though I didn’t do that.
00:33:55.439
It tries to do all those things. This is a work in progress—actually the gem I wrote yesterday, and I couldn’t get anyone to install it. If you install it, you’ll see something saying 'At install time, here are the results.' It tells me that my box is completely open to everything.
00:34:14.600
I successfully posted to a web service, it wrote to my bash profile, it stole my gem cutter credentials, and it also stole my SSH key. I killed all four canaries, so check it out.
00:34:32.239
Thanks for coming to my talk! Conferences always inspire me. I love coming to conferences because I learn about new gems and new ways of doing things. I always want to go home and try out the new patterns or use new gems.
00:34:51.439
I hope this talk didn’t inspire you—at least not to write any malicious gems. I hope it was at least entertaining and eye-opening. Thank you!
00:35:06.000
I don't know if I have time for questions, but we’ve got a few minutes, so if you’ve got any questions, I’d love to hear them. One of my favorite things is to hear how you think I should hack people. This usually generates a lot of talk, and it’s kind of fun to think about.
00:35:22.720
Are there any questions?
00:35:39.200
The question on everyone's mind is: how do we prevent malicious gems, especially if a good gem is later PC as malicious? Are there any guidelines?
00:36:01.120
So the questions are aligning with how RubyGems could potentially play well with other packaging tools. I think some of those packaging systems might be good guidelines.
00:36:19.840
The Apache Foundation is pretty trustworthy. I think it would be great if we as a Ruby community decided to adopt one of those practices.
00:36:38.480
If you could suggest what we could do... that would be awesome. Others might have some insights as well.
00:36:54.400
Other questions?
00:37:06.400
You’ve discussed safety in gem development. What about reverse engineering a gem after installing?
00:37:19.680
The question is whether it is safe to clone a repository and build a Ruby gem from it?
00:37:32.480
If something has good documentation, good reviews, and community trust, then yes, that would be safe.
00:37:42.720
Time for one more question.
00:37:54.720
Okay, thank you very much!
