Ruby Video
Aloha RubyConf 2012
Hacking with Gems
Ben Smith
@binji
#hacking
#security
#rubygems
#dependency-management
#ruby-on-rails
#open-source
Summarized using AI

Hacking with Gems

by Ben Smith

In the talk titled "Hacking with Gems" presented by Ben Smith at Aloha RubyConf 2012, the focus is on the potential dangers associated with using Ruby gems. Smith, a Rails developer, emphasizes that many in the Ruby community often trust gems without proper vetting, which can lead to serious security vulnerabilities.

Key points discussed in the presentation include:

  • Introduction to the Speaker: Ben Smith shares his background as a Rails developer and expresses his passion for security, though he admits he lacks formal training in the field.
  • Understanding Dependencies: He recounts a client's inquiry about harmful dependencies, prompting a discussion on the risks associated with malicious gems.
  • Example of a Malicious Gem: Smith created a hypothetical gem called "Awesome Rails Flash Messages" that appears to provide useful functionality but actually logs sensitive information and sends it to a remote server. This serves as an example of how seemingly harmless gems can contain harmful code.
  • Demonstration of Exploits: He explains how he developed another gem, "Net HTTP Detector," to reveal unauthorized POST requests, showcasing how easy it is to capture sensitive data such as emails and passwords.
  • Further Exploitation: Smith describes creating a series of deceptive gems with escalating levels of malicious intent, including one that manipulates database access through a hidden interface, potentially allowing a hacker to gain significant control.
  • How Trust Can Be Misplaced: He warns that trusted gems may also harbor malicious code, using the notion that larger gems can have dependencies that are unverified.
  • Community Consideration: The talk takes a turn towards community practices, advocating for gem signing and private repositories as potential solutions to these security threats.
  • Conclusion and Takeaways: Smith concludes with an important reminder: developers should not blindly trust gems. They should vet code, use tools to monitor for malicious activity, and consider writing their code when feasible.

In summary, this talk underscores the significance of vigilance and due diligence in software dependency management within the Ruby community. Smith encourages developers to be proactive in understanding and verifying the gems they use while highlighting tools that can assist in monitoring for potential vulnerabilities. His key message is a call for better practices, urging attendees to think critically about their code dependencies.

00:00:14.790 Good morning, everybody! My name is Ben Smith, and I'm here to talk to you about hacking with gems. This should be an entertaining talk; I hope you all enjoy it.
00:00:20.470 But first, I'd like to talk a little bit about who I am. I'm a Rails developer and have been doing it since Rails version 1.1.6. Professionally, I work for Pibil Labs.
00:00:32.860 I would like to thank them for sending me here and giving me a full ride; it’s awesome, and I couldn’t do it without them.
00:00:39.129 Currently, I'm in Boulder, Colorado, where it was actually snowing on Saturday when I left. So, it’s nice to enjoy some warm weather before I have to embrace the winter for good.
00:00:51.850 It’s also nice to be back here because I was actually born just next door, and I haven't been back since I was a child, so this feels a little bit like a reunion for me.
00:01:10.330 Now, I'd also like to clarify what I'm not. I'm not a security expert; I have a fascination with security, but no formal training or background in it.
00:01:16.720 I enjoy going to DEF CON every year, but 90% of the content there goes straight over my head.
00:01:23.590 Keep that in mind during this talk because I'm just a Rails developer sharing some ideas that I've discovered, and it turns out those ideas were really easy for me to come up with.
00:01:36.280 So, anyone can do this stuff! That being said, please don’t try this at home.
00:01:45.100 In fact, it would be better if you forgot everything I'm about to tell you, except maybe the last four or five slides.
00:01:57.399 This talk is a little bit conflicted; on one hand, I am telling you how to avoid being hacked and things you should look out for, and on the other, I’m showing you exactly what to do if you want to compromise other people’s systems.
00:02:03.380 This picture doesn’t really convey that very well. If you look at this, you might think it looks like a horrible idea; why would anyone ever do that? The truth is, a lot of these things seem fun, and they make you think, "Wow, that’s neat! I should try that." Please don't.
00:02:20.480 About a year ago, I had a client who asked, 'What are the dependencies in my app?' He wanted to have an understanding of what was going on, and he was a bit worried about having too many dependencies that might be shady or odd.
00:02:34.250 So, I showed him the Gemfile.lock of his Rails app, which satisfied his curiosity, but it got me wondering: what’s the worst that could happen if a malicious gem was introduced?
00:02:58.900 What could a malicious gem do, and what’s the worst possible thing that could happen to this client? I thought about it and realized that this client has customers, and if their private data were to leak or get compromised, everyone might lose faith in the product, and that could lead to total failure.
00:03:15.769 So, I thought, how hard would it be to write a gem that wouldn’t do that? I wrote this gem called Awesome Rails Flash Messages.
00:03:20.930 I decided that no one would actually install a gem that would openly hack them. It has to be something that appears to do something else—something useful—but behind the scenes is actually doing something malicious.
00:03:40.720 It could be argued that this gem is useful; it takes Rails flash messages and enhances them with all capital letters and random sets of exclamation points, making them look better.
00:03:49.640 But this gem has some dangerous side effects. If you poke around the code, you might notice some odd things, like a line that checks if parameter 2 matches a base64-decoded string.
00:04:06.500 If that evaluates to true, it writes the parameters to something called development.log in your Rails 'public' directory and, just for good measure, it also sends the parameters to some web service.
00:04:25.310 Going back to that if statement, if parameter 2 matches something that actually translates to if it matches 'password,' then anytime the Rails app receives a request that contains 'password'—like from a login page or a registration page—it writes to that development log and sends it to a web service.
00:04:43.850 That development log file, located in the public directory, ends up containing clear-text emails and passwords; they are not filtered like those in the real development log. They are in clear text and, of course, end up somewhere else on the internet.
00:05:06.500 I could start throwing those credentials at banking websites, but that wouldn’t work, right? Because nobody uses their passwords here, and that would just annoy my client.
00:05:12.500 However, I wondered just how easy it would be to do this. So, I wrote another gem called Net HTTP Detector, which I created to detect the hack that I had done previously.
00:05:26.359 This gem logs calls from Net::HTTP and shows you what’s going on when you start using that Awesome Rails Flash Messages gem. If something happens, you’ll see in your standard output or Rails log file something like this: it logs POST requests with parameters, including the email and password.
00:05:43.760 While it’s not pretty, it demonstrates what’s happening. This works by overriding the POST form to log what’s happening and defining a valid POST form so that if you have HTTP calls in your app, they can be executed without logging.
00:06:05.310 But it does one more thing. In one of the last lines of the code, there's a snippet that reaches out to a web service and grabs something. In the adventurous spirit of Ruby, I decided to grab some code that adds a before filter and looks for a parameter named 'db_console'.
00:06:28.710 If it finds it, it manipulates the database using Active Record. This means that if you took a normal Rails route and added the 'db_console' parameter, you’d end up with a nice little interface to the database.
00:06:43.320 This would allow you to execute queries such as, 'Show me all the users' or even to promote yourself to a database admin.
00:06:59.440 The moral of this story is to be careful of wolves in sheep’s clothing.
00:07:06.360 If you actually want to do network monitoring, there are tools out there like Little Snitch.
00:07:12.570 This tool shows your network traffic from all your processes. You can see something like 'Ruby made a request to this Web app.'
00:07:18.210 I was able to access emails and passwords; my client would have been horrified if he’d discovered this was happening.
00:07:26.969 So, what more could I do? I don’t actually own a cat, but after all the talks here, I felt like I needed cat pictures. Last night, I started searching for cat images.
00:07:43.919 That was easy! Now that I had database access with usernames and passwords, I wondered what else I could accomplish. I wrote another gem called Better Date to S, which claims to take the date format from Rails and strips out extraneous whitespace.
00:08:07.560 The bottom line had an extra whitespace, and it was annoying, so it gets rid of that. You might be catching on at this point: none of these gems do what they claim to do. It’s all just misdirection.
00:08:25.660 What my gem actually does is it calls a method for setting date formats in Rails, but this method is compiled in C.
00:08:41.150 The rub is that it was packaged without the source code; it only contains the compiled code, so there’s no way to see what it actually does.
00:08:51.580 Since I wrote it, I know what it does. This is the source code before it was compiled into the gem. If I ran this in production—with the intention of hiding it—you would end up with an assets file in your public directory.
00:09:02.670 If I were to come along and download that tarball and extract it, I'd find something that holds your Rails source code. So now, I have the full source code of your app.
00:09:22.229 But to be honest, this gem doesn’t actually work. It could, but I’m somewhat lazy. It’s what’s called a fat gem—gems that are precompiled for specific platforms.
00:09:36.920 There are tools available to do all of this work, but in the time I tried to make it work, I just gave up. I decided that there had to be easier ways of achieving my goals.
00:09:58.900 So, I wrote another gem called New Treaty. How much deception do you think this will effect? RSpec is great, but its matchers are somewhat deceptive. When you say something should be true, it may not actually be true.
00:10:11.600 So, I decided to write this RSpec matcher, so when you say 'true.should be true', it passes, but with 'user.should be true', it shouldn’t pass.
00:10:25.500 It actually doesn’t do anything at all. There’s no functionality; I just haven’t gotten to implementing it yet. It’s because I grew tired of writing useful gems and decided to go straight for the hack.
00:10:57.580 So where’s the trick in this? When you install this gem, it tells you it’s building native extensions, which usually means compiling C extensions locally.
00:11:23.490 However, looking at the source tree, you’ll see there are no C files. This could be a red flag, so let’s take a closer look.
00:11:36.630 This is the gemspec for this gem. The line about gem extensions is usually for compiling C extensions, but in this case, it’s designed to run the Rake file at install time.
00:11:49.900 This is crucial—this happens at install time, not when you require the gem. So, if you look back in the source tree again, there’s no Rake file present.
00:12:07.170 So where does it go? Out of curiosity, how many people here read the source code of any gems they think might be shady? Some people do...
00:12:38.650 Okay, so how many of you before you ever do a 'gem install', execute 'gem fetch' or 'gem unpack', examine the source code, and then, if it looks okay, proceed to install?
00:12:57.890 Not many, huh? When you use 'gem install,' or bundle your application, it immediately downloads and installs the gem, which is the default behavior.
00:13:11.490 If you don’t want to install the gem yet, you have to use 'gem fetch.' It will then download it for you to inspect its contents.
00:13:31.750 Then you can use 'gem unpack,' which will expand out that gem so you can see what’s inside.
00:13:48.090 So if I unpack this V3T gem, what do we find? There are a couple of extra files here, including that missing Rake file and a file called 'temp.rb'.
00:14:03.890 What does that Rake file do? Well, when we installed the gem, it ran this Rake file. It copies 'temp.rb' into your home directory and creates a hidden file.
00:14:19.420 Then it adds an alias for 'sudo' in your bash profile pointing at that new file, and finally, it removes itself.
00:14:31.540 It basically states that any file executing should remove itself, which is why we didn’t see it during the 'gem install.' The only evidence that it ever existed was the alert about compiling native extensions.
00:14:43.920 There are actually tools available that can help you catch something like this—FSO Venter is one of those. You can activate it to watch for file system changes.
00:15:00.890 So, back to the gem: the Rake file aliases 'sudo' to run something. This means it appears to work as usual while secretly grabbing your password.
00:15:17.780 Once it has your password, it can do a lot of damage, such as enable SSH, create a new user, and set their password, and then notify a web service.
00:15:31.630 Now the box is compromised, and the service says, ‘Hey, there’s a box out here, and it's ready to be SSHed into.’
00:16:00.520 I presented some of these gems at a lightning talk at Mountain West, and the biggest takeaway from that was, 'Never install Ben’s gems!'
00:16:19.900 That's a valid sentiment! It’s probably a good thing I didn’t have to convince anyone to install them.
00:16:42.860 So my next question is, how can I get you to install my other gems? You all seem smart enough to know not to install gems that you don't trust.
00:17:02.560 What gems do you consider trustworthy? Rails? That seems safe. RSpec? Sinatra? They look safe, too. But you also have to trust all their dependencies.
00:17:22.300 The chances are high that you can trust the bigger gems. So, how could I sneak my code into those trusted gems?
00:17:40.880 Take, for example, the gem that I've just described. This would grab your gem cutter credentials and a list of installed gems and send that information to a web service.
00:18:00.820 Now I can own your gems! I hope you all write trustworthy gems that are useful.
00:18:12.680 I could then clone your repo, add my own code, build your gem, and push it. This wouldn’t create a new gem—it would push a new version of your existing gem.
00:18:29.800 What’s worse is that you would never know this happened. There are no notifications when a new version is pushed.
00:18:47.750 Unless you check rubygems.org for version updates, or you execute 'gem update,' you might remain oblivious to my changes.
00:19:01.440 This all happens just by grabbing that little gem API key and your gem cutter credentials.
00:19:26.960 So, do people trust your gems? And do people who install your gems have trustworthy gems themselves?
00:19:43.610 I could hack your gems, and then your friends would install those gems, and I could then hack their gems, spreading to larger and larger networks.
00:20:00.690 This leads to the question: how many steps would it take for me to get from my gems to one of the dependencies of Rails?
00:20:17.420 One last problem remains: how do I bootstrap this? How do I get the first group of people to install these gems?
00:20:36.850 To be popular! This is a screenshot from rubygems.org. I wrote a gem and decided to keep it at the top of the downloaded list by running a script.
00:20:50.440 I kept it in the top 5 all day, but I only got one installation!
00:21:07.440 Being popular kind of sucks, although I realized there might have been a bug affecting the script; it might only catch installations from root.
00:21:22.860 So, maybe being popular isn’t so bad. But how else can I get people to install my gem?
00:21:39.710 Presenting at conferences is a great way to spread the word about gems and libraries. I've noticed that many talks, even at this conference, share insights on various gems.
00:21:55.610 What if I just handed out business cards with the gem name on it and left them around the conference? How many would install it?
00:22:14.240 That’s right, a few people would. I was super nervous about this, but I tried.
00:22:30.140 The previous morning, I came down to the conference early. I had all these cards in my pocket and was looking for places to leave them.
00:22:47.530 Even after talking to the conference organizer, who assured me it was fine, I was so nervous about 'sneaking' around.
00:23:05.070 I managed to leave my hotel room key somewhere among the cards. So my major takeaway was that I should never be a thief because I would just look so guilty.
00:23:21.760 However, I did it, and some people picked it up. Some even installed it. Here’s who installed it: six people! I was hoping for a little more adoption.
00:23:37.310 I did get a pull request from Evan here, and I may incorporate that into my next gem at the next conference.
00:23:54.480 If you installed this gem, and if you have your own gems, I could potentially have owned those gems—but luckily for you, this gem doesn't do anything.
00:24:10.890 It only posts your username to a server; that’s all. If you looked at the GitHub account, the code would look perfectly fine.
00:24:34.320 But there’s nothing saying that what’s on GitHub matches what’s on RubyGems; it could point to a completely different repo.
00:24:48.490 I just didn’t push the RubyGems for the last week while I was adding the hack.
00:25:04.150 Ultimately, I got 3.8% adoption, which isn't too bad. At this conference, there were 155 participants; if I took this idea to RailsConf or RubyConf, I might get a better response.
00:25:21.649 So, what now? How do we keep this from happening? There are a couple of things we can do.
00:25:37.649 We can start by signing all our gems and hoping that our private keys don’t get compromised like how I stole those gem cutter credentials.
00:25:54.940 Of course, these steps require high adoption; you want all gems you install to be signed. It takes significant effort to implement this.
00:26:12.330 Someone could fork RubyGems and add code to send notifications when gems are compromised. This ensures you know when there’s a new, potentially harmful version of your gem.
00:26:28.750 We could also create private gem repositories, manually vetting every line of every gem you wish to install, keeping only secured gems.
00:26:43.180 This is a lot of work, but I’ve heard of some companies starting to do this. There's likely a business opportunity in providing private repos.
00:27:00.740 While that’s a lot of work and probably a good practice, I want everyone to remember: do not try this at home.
00:27:14.560 The Ruby community is wonderful and supportive; we have conferences like this where everyone is friendly. Let’s keep it that way.
00:27:36.240 From this talk, I hope you take away this key idea: don’t just install gems without caution. It’s crucial to favor writing code yourself.
00:27:51.780 I don’t know how many times I've gone on Stack Overflow, and the answer is often to install a gem that someone made for a problem that can be solved with a little more effort.
00:28:07.740 Pay attention to what your gems do—if you’re installing an RSpec matcher, ensure it’s not using C extensions that might affect your system.
00:28:22.710 Tools like Little Snitch and FSO Venter are useful for monitoring network requests. Read the actual source of these gems.
00:28:36.150 Not just the one on GitHub… perform a gem fetch, and a gem unpack; inspect that source. If you do that, you might catch vulnerabilities.
00:28:50.830 One additional gem—'Coal Mine Canary'—will bundle all the hacks I've presented today. It will help you perform basic pen testing.
00:29:04.480 When you install this, it will notify you if any of your credentials were compromised, grabbing SSH keys or Gem cutter credentials.
00:29:18.700 I hope to push the first version of this sometime next week.
00:29:32.020 Finally, I want you to know that conferences tend to inspire me. I often go home with a bunch of ideas for new projects and gems.
00:29:50.910 I hope this talk inspires you, but not in the way of writing malicious gems. I do hope it was entertaining, though!
Edit
Explore all talks recorded at Aloha RubyConf 2012
+13