Search
RailsConf 2013
Security is hard, but we can't go shopping
Edit
#ruby
#ruby-on-rails
#security-vulnerability
#bundler
#rubygems
#web-security
Summarized using AI

Security is hard, but we can't go shopping

André Arko • June 25, 2013 • Portland, OR • Talk

The video titled "Security is hard, but we can't go shopping" by André Arko, delivered at Rails Conf 2013, discusses the increasing frequency of security vulnerabilities in Ruby and Rails ecosystems. Arko addresses the importance of understanding and managing security issues as Ruby libraries become more popular, thus attracting the attention of security researchers. He highlights several key points:

  • Increased Frequency of Vulnerabilities: Arko notes that the Ruby community has seen an unusual spike in security vulnerabilities, with over 12 to 13 CVE (Common Vulnerabilities and Exposures) announcements in just a few months, compared to far fewer in years prior. This has drawn the scrutiny of both white hat and black hat security researchers.

  • Responsible Disclosure: The video emphasizes the practice of responsible disclosure in the security community, where security issues are reported privately to the developers before being disclosed publicly, allowing time for a fix to be made.

  • Handling Security Issues in Own Gems: Arko urges developers to consider the security implications of their own libraries. With the proliferation of gems, maintaining security becomes challenging as multiple dependencies could introduce vulnerabilities.

  • Importance of Updates: Security updates are framed as essential insurance to mitigate risks associated with vulnerabilities. Arko explains that neglecting timely updates could lead to significant losses and even legal repercussions if a breach occurs.

  • Responding to Security Researchers: Developers are encouraged to handle communications with security researchers carefully and to respect their findings. A collaborative approach is necessary to resolve issues effectively.

  • Community Learning: Arko suggests that learning from past mistakes of the Ruby and Rails communities regarding security can lead to a more robust handling of security issues in the future.

  • Ruby Ecosystem Initiatives: He mentions the establishment of a mailing list for announcing Ruby ecosystem security releases to encourage transparency and community awareness.

In conclusion, Arko stresses the need for Ruby developers to take security seriously, engage proactively with security researchers, and maintain their libraries to prevent vulnerabilities from impacting users. The video ultimately encourages a cultural shift towards better security practices within the Ruby community, allowing developers to focus on creating rather than worrying about security breaches.

Security is hard, but we can't go shopping
André Arko • June 25, 2013 • Portland, OR • Talk

The last few months have been pretty brutal for anyone who depends on Ruby libraries in production. Ruby is really popular now, and that's exciting! But it also means that we are now square in the crosshairs of security researchers, whether whitehat, blackhat, or some other hat. Only the Ruby and Rails core teams have meaningful experience with vulnerabilites so far. It won't last. Vulnerabilities are everywhere, and handling security issues responsibly is critical if we want Ruby (and Rubyists) to stay in high demand.
Using Bundler's first CVE as a case study, I'll discuss responsible disclosure, as well as repsonsible ownership of your own code. How do you know if a bug is a security issue, and how do you report it without tipping off someone malicious? As a Rubyist, you probably have at least one library of your own. How do you handle security issues, and fix them without compromising apps running on the old code? Don't let your site get hacked, or worse yet, let your project allow someone else's site to get hacked! Learn from the hard-won wisdom of the security community so that we won't repeat the mistakes of others.

RailsConf 2013

00:00:17.199 Let's get going. My name's André Arko. I am involved in basically everything that exists at Cloud City Development, where I do Rails consulting.
00:00:22.320 I'm here to talk to you about how security is hard and, unfortunately, going shopping is not an option.
00:00:30.000 To start things off, let's talk about Ruby itself. I don't know how long all of you have been Rubyists, but historically, Ruby security issues have come up about once or twice a year.
00:00:36.239 Ruby has not regularly had major security issues pop up on a consistent basis. However, in the last few months, that has not been the case.
00:00:46.960 We've had releases 327, 385, and 392, all of which have included major security issues. This is really not normal. Since November, we have experienced six months with three releases, which is akin to 18 months or two years' worth of normal releases all in a short span.
00:01:07.119 In fact, it’s so unusual that the Ruby core team itself has been apologizing for how frequently they are releasing updates. Compounding this is the fact that Rails also has security releases, and since it’s RailsConf, if you’re here, you probably care about Rails security releases.
00:01:20.320 Rails has had an interesting number of security releases just since the beginning of the year.
00:01:27.040 In fact, I couldn't fit the entire list of security releases that have occurred this year onto this slide. That's not normal either. Rails has already had around 12 or 13 CVE security announcements and patches this year.
00:01:36.240 This is already more than there were in the entirety of either 2012 or 2011, and we are only in the middle of April.
00:01:49.600 The situation in 2012 and 2011 was such that there just weren’t many security issues; we only had a single page worth of them.
00:01:55.439 So, now that I've informed you that Rails and Ruby have both been experiencing a plethora of CVE security vulnerabilities recently, this prompts a common question from many people during conversations, who stop and ask, 'Wait, what’s a CVE?' I know that it's bad and related to security issues, but I’m not sure what it means.
00:02:14.800 CVE stands for Common Vulnerabilities and Exposures. When it gets down to it, CVEs are just numbers assigned by a company called MITRE, and many organizations—including corporations and governments—have agreed to use these CVE numbers as a single, coordinated point of reference for security issues.
00:02:27.200 MITRE assigns blocks of CVE numbers to companies on the list of numbering authorities, which mainly includes large software companies like Apple, Adobe, Cisco, Red Hat, and Microsoft.
00:02:39.680 Each of these companies, as they work with their own software or frequently used software, assigns numbers from their block to new security issues that arise.
00:02:52.560 The Ruby community has primarily been supported in this effort by Red Hat, which has an open-source security task force. The main interface has been Carl Seyfried, who has provided standardized advice and CVEs from Red Hat's repository of open-source CVEs when we’ve faced high-stakes issues.
00:03:07.360 Once you have a CVE number, both MITRE and NIST, which is the government’s National Institute of Standards and Technology, host websites where you can look at the list of all CVEs that have ever been assigned, their names, the software they affect, and links to workarounds or patches.
00:03:20.400 So now that we understand what CVEs are, historically, the Rails and Ruby communities have shared a pervasive mindset of being nice, and as a result, we haven’t encountered many problems.
00:03:37.360 However, the flurry of Ruby and Rails security activity that we’ve seen in recent months has sparked increased interest among security researchers in the Ruby ecosystem, bringing scrutiny to vulnerabilities.
00:03:51.440 This attention means that the Ruby community is now under intense observation from individuals whose jobs or hobbies focus on identifying problems with programming languages and their implementations.
00:04:01.759 Fortunately so far, all the significant security vulnerabilities I mentioned in Ruby and Rails have originated from security researchers on the white or at least white-ish side of the white hat-black hat divide.
00:04:15.120 Luckily, these researchers have allowed us to promptly provide fixes simultaneously with public announcements of these issues. But it is important to recognize that just because we've been lucky does not guarantee that it will continue.
00:04:29.440 That's why it's essential to start discussing this topic now, as security researchers are honing in on us, before we experience misfortune.
00:04:46.000 Moving on from a very pleasant matter—DHH, the creator of Rails, is not as nice as Mats.
00:05:02.080 While that's true, we can definitely learn from Rails and DHH about how they have dealt with security vulnerabilities.
00:05:15.440 Over the years, Rails has accounted for somewhere between 80 and 100 of the security issues that the entire Ruby community has faced.
00:05:29.040 This is because historically, Rails represented the entire attack surface of the Ruby programming language to the internet; if there were issues not involving Rails, there wasn’t much to attack.
00:05:45.840 If you had a problem with Rails, it effectively meant that you could attack nearly everything involving Ruby at that point.
00:05:56.800 What that leads us to is that the Rails security team has had to develop and refine a lot of the processes I’m going to explain.
00:06:04.800 Something really cool happened with Rails over the last year or so: we now have gems for basically everything. Rails 3 marked the great modularization of Rails—it's no longer a gigantic monolithic system.
00:06:18.560 You can pick and choose what components to use, and swap out some things for others. It’s a great time for bundler, which allows you to plug in any fork from GitHub with one line of code.
00:06:31.200 This functionality allows you to use a variant of the released and well-known code, which may or may not have security issues associated with it. However, having so many gems also implies that now there are potentially dozens or even hundreds of gems providing security vulnerabilities to your application.
00:06:50.400 For instance, in the first eight weeks of this year, there were security vulnerabilities fixed across several releases of Ruby gems, including bundler, json, rexml, rack, aural, active record, action pack, active support, and rdoc—even the latter had an XSS bug.
00:07:06.640 Even if you think to yourself, “Oh, I only use the standard library; I don’t use any of those Rails gems,” it’s very likely that you've utilized something on that list, which poses a significant problem.
00:07:21.919 So, what do we do when everything we could possibly use is facing simultaneous security issues? Updating is a huge pain.
00:07:34.400 Updating distracts you from feature development, and it’s tough to sell to a boss or manager who has deadlines for delivering what's deemed "business value."
00:07:47.040 Handling security updates is akin to purchasing insurance; your boss doesn't say, "Oh, we could have had two more weeks of runway if we didn't pay for insurance on our office." If your office burned down with everything inside it, it would be catastrophic.
00:08:05.760 Because of the small risk combined with large potential losses, it's worth investing that small amount in security updates. Security updates are essentially exploitation insurance.
00:08:24.560 There’s a famous analogy that says you don’t need to be faster than the bear; you only need to be faster than the guy behind you. If you can apply security patches before someone else, then it's likely that it will be someone else who suffers the consequences for not having patched.
00:08:36.560 Without updates, failures can be catastrophic. Just cleaning up from a single security breach can lead to significant amounts of lost time, effort, and lost customer goodwill.
00:08:53.039 Moreover, there are disclosure laws in various states that now require companies to inform every individual whose data may have been compromised. This could mean sending thousands of emails and facing potential civil and criminal penalties.
00:09:08.640 The key takeaway is to invest the time to do updates as patches are released, which greatly reduces the risk that you'll have to tackle these issues.
00:09:17.680 In conclusion, I understand that updating is hard work. It’s not enjoyable or cool. You won't get to present talks about how you successfully updated your Rails app from version 3.2.12 to 3.2.13.
00:09:29.680 But the reality is that if you do the updates, you can rest easy at night knowing that the chances of facing security incidents and their repercussions are significantly minimized.
00:09:46.480 As a developer, it’s likely that you use other people's code. So let’s talk about reporting security issues. Given that Ruby and Rails haven’t had to face many security issues before, the Ruby community doesn't have established cultural norms around security issues yet.
00:10:01.199 When you find a bug in a gem within your application, your first instinct might be to rush to GitHub and say, "Hey, your thing doesn’t work." But I urge you to stop and think for a moment before taking that action.
00:10:14.559 There are various reasons why it’s crucial to approach this carefully. The field of security research has established a well-known policy called responsible disclosure.
00:10:27.439 Responsible disclosure can be regarded as the best possible scenario except for all the other situations that are worse. It provides some level of assurance that no one ends up severely affected.
00:10:39.680 The first part of responsible disclosure involves informing necessary parties about the vulnerability so they can address it. Companies typically dislike this because they would prefer to keep quiet about losing sensitive customer information.
00:10:50.800 The other side of responsible disclosure is about being responsible. Clever individuals who discover vulnerabilities often feel the urge to publicize their findings to gain recognition for their cleverness.
00:11:04.480 Responsible disclosure means that instead of making the vulnerability public, the person who found the problem discusses it privately with the company, urging them to fix it before it becomes widely known.
00:11:24.480 Security researchers typically agree to this process, as it helps protect themselves from potential legal actions stemming from their discoveries.
00:11:38.240 When properly carried out, responsible disclosure is meant to foster a collaborative approach. Companies need to address vulnerabilities while researchers can communicate their findings without fear.
00:11:53.440 Some companies are trying to make this process more encouraging by rewarding security researchers. For instance, Facebook has an extensive responsible disclosure page that outlines how to report vulnerabilities along with a minimum reward for every valid finding.
00:12:09.040 On the other hand, GitHub incentivizes researchers with the possibility of receiving a t-shirt, although they don’t have a stated reward policy.
00:12:24.320 Engine Yard, however, has clarified that they don’t offer rewards. Unsurprisingly, being explicit about not providing rewards can sometimes lead to adverse reactions.
00:12:38.240 Now that you understand the overview, let's revisit the situation where you find bugs in other people's code.
00:12:50.480 Congratulations on responsibly discovering a bug; this is how software improves! However, before running to the issue tracker, please consider two questions regarding the bug.
00:13:01.679 First, can you access something that you, as the owner of an account on that website, shouldn’t be able to access? For example, can you see someone else's credit card number?
00:13:17.040 The second question is whether you can disable something for others. For instance, if you discover that refreshing a page multiple times makes the website unresponsive for everyone, that indicates a denial-of-service vulnerability.
00:13:31.520 If either answer is yes, you absolutely need to disclose the issue responsibly. Do not publicly post any sensitive information related to this vulnerability.
00:13:44.640 Contact the author of the library directly, as it’s essential to identify which specific library has the bug. Doing so makes it more likely to have the issue patched before it’s announced publicly.
00:13:56.000 This is particularly vital for larger projects like Rails, where timely patching can prevent issues from becoming public dangers.
00:14:06.240 If you're going to contact the author, ideally review their security policy, if they have one. Rails, for instance, has a security page and a private mailing list that is dedicated to security matters.
00:14:21.520 If no security information is available, check the gem specification file. Every gem contains an email address for the author in its spec.
00:14:37.440 As a last resort, you can find the maintainer's email on GitHub, which can help you initiate contact.
00:14:53.440 Most of the time, you might be faced with complications. I’ve seen situations range from not knowing whom to contact to having a maintainer who committed a fix but is unavailable to deploy it.
00:15:09.680 Please work together with the developers. They are also people who may not immediately respond because of their day jobs, but they do have the collective goal of ensuring safety.
00:15:24.000 Once you hear back from the author, offer to help. Sometimes they may appreciate the offer to resolve the security issue collaboratively.
00:15:36.160 After addressing the issue, ensure that the fix and the announcement occur simultaneously or even try to release the fix a bit earlier than the announcement.
00:15:51.200 It's crucial to avoid publicizing vulnerabilities—even if you’ve figured out how to solve them—until the fix is available in the current gem release.
00:16:09.440 If it comes to the point where you cannot contact anyone, take matters into your hands and fork the code.
00:16:17.680 If you encounter a security issue, it’s wise to fix it on your own, especially if you are already using that code.
00:16:30.560 However, only apply this strategy if the responsible party is unresponsive. Otherwise, working with them is ideal.
00:16:46.240 Finally, I'd like to touch on the topic of your own gems. Since many of you are Rubyists, I’d like to ask how many of you have written your own gem or library.
00:17:03.520 Ruby makes writing libraries straightforward, and a lot of developers have their own projects. However, every single one of those projects is a potential security vulnerability for someone else.
00:17:16.960 At this point, I want to discuss being on the other side of responsible disclosure—how do you responsibly receive reports of security issues, and how do you assist users who depend on your library?
00:17:30.280 Unless your code is perfect—a claim I'd urge you to question—there’s a good chance you might face this scenario at some point.
00:17:42.480 The simple scenario is where another Ruby developer politely emails you, stating, 'Hey, I found something that seems problematic. Can we collaborate on fixing this?'
00:17:55.680 In this situation, collaborate to fix it, review the change, release the fix, and announce the security issue to your user base.
00:18:09.440 The next challenge occurs when a bug is identified and is already publicly known. In this case, promptly addressing the issue and getting the fix to users is crucial.
00:18:22.560 If a vulnerability exists and has been publicly disclosed, you need to act quickly—the bad actors are likely already aware.
00:18:36.560 My rule of thumb is this: if additional publicity won’t significantly worsen the situation, aim to make users aware of the issue as soon as possible.
00:18:41.920 Finally, in the hardest scenario, a security researcher contacts you, possibly with information about remote shell vulnerabilities.
00:18:56.080 If you receive such an email, respond promptly and keep the lines of communication open. Regularly inform them about your efforts to fix the problem.
00:19:09.280 Be clear about your actions. If things go silent, researchers might feel inclined to go public with their findings.
00:19:23.760 As a gem author, ensure that you make it easy for people to report issues to you. Include your email address in the gem specs and on your GitHub profile.
00:19:37.600 If you're part of a library team, establish a security address or contact protocol. Creating a disclosure policy, even a brief one, encourages responsible reporting.
00:19:53.680 To support the Ruby ecosystem, I’ve established a mailing list for announcing Ruby ecosystem security releases. This space handles only security-related announcements, ensuring low traffic.
00:20:10.080 If we can follow the hard-won lessons learned by the security community, one day we can return to a state of calm and tranquility—and then we can go shopping.
00:20:26.080 Thank you for your attention! Do you have any questions? Unfortunately, we don't have time for questions, but thanks a lot for being here!
00:20:39.840 You.
André Arko
@indirect
Explore all talks recorded at RailsConf 2013
+93