00:00:17.199
Let's get going. My name's André Arko. I am involved in basically everything that exists at Cloud City Development, where I do Rails consulting.
00:00:22.320
I'm here to talk to you about how security is hard and, unfortunately, going shopping is not an option.
00:00:30.000
To start things off, let's talk about Ruby itself. I don't know how long all of you have been Rubyists, but historically, Ruby security issues have come up about once or twice a year.
00:00:36.239
Ruby has not regularly had major security issues pop up on a consistent basis. However, in the last few months, that has not been the case.
00:00:46.960
We've had releases 327, 385, and 392, all of which have included major security issues. This is really not normal. Since November, we have experienced six months with three releases, which is akin to 18 months or two years' worth of normal releases all in a short span.
00:01:07.119
In fact, it’s so unusual that the Ruby core team itself has been apologizing for how frequently they are releasing updates. Compounding this is the fact that Rails also has security releases, and since it’s RailsConf, if you’re here, you probably care about Rails security releases.
00:01:20.320
Rails has had an interesting number of security releases just since the beginning of the year.
00:01:27.040
In fact, I couldn't fit the entire list of security releases that have occurred this year onto this slide. That's not normal either. Rails has already had around 12 or 13 CVE security announcements and patches this year.
00:01:36.240
This is already more than there were in the entirety of either 2012 or 2011, and we are only in the middle of April.
00:01:49.600
The situation in 2012 and 2011 was such that there just weren’t many security issues; we only had a single page worth of them.
00:01:55.439
So, now that I've informed you that Rails and Ruby have both been experiencing a plethora of CVE security vulnerabilities recently, this prompts a common question from many people during conversations, who stop and ask, 'Wait, what’s a CVE?' I know that it's bad and related to security issues, but I’m not sure what it means.
00:02:14.800
CVE stands for Common Vulnerabilities and Exposures. When it gets down to it, CVEs are just numbers assigned by a company called MITRE, and many organizations—including corporations and governments—have agreed to use these CVE numbers as a single, coordinated point of reference for security issues.
00:02:27.200
MITRE assigns blocks of CVE numbers to companies on the list of numbering authorities, which mainly includes large software companies like Apple, Adobe, Cisco, Red Hat, and Microsoft.
00:02:39.680
Each of these companies, as they work with their own software or frequently used software, assigns numbers from their block to new security issues that arise.
00:02:52.560
The Ruby community has primarily been supported in this effort by Red Hat, which has an open-source security task force. The main interface has been Carl Seyfried, who has provided standardized advice and CVEs from Red Hat's repository of open-source CVEs when we’ve faced high-stakes issues.
00:03:07.360
Once you have a CVE number, both MITRE and NIST, which is the government’s National Institute of Standards and Technology, host websites where you can look at the list of all CVEs that have ever been assigned, their names, the software they affect, and links to workarounds or patches.
00:03:20.400
So now that we understand what CVEs are, historically, the Rails and Ruby communities have shared a pervasive mindset of being nice, and as a result, we haven’t encountered many problems.
00:03:37.360
However, the flurry of Ruby and Rails security activity that we’ve seen in recent months has sparked increased interest among security researchers in the Ruby ecosystem, bringing scrutiny to vulnerabilities.
00:03:51.440
This attention means that the Ruby community is now under intense observation from individuals whose jobs or hobbies focus on identifying problems with programming languages and their implementations.
00:04:01.759
Fortunately so far, all the significant security vulnerabilities I mentioned in Ruby and Rails have originated from security researchers on the white or at least white-ish side of the white hat-black hat divide.
00:04:15.120
Luckily, these researchers have allowed us to promptly provide fixes simultaneously with public announcements of these issues. But it is important to recognize that just because we've been lucky does not guarantee that it will continue.
00:04:29.440
That's why it's essential to start discussing this topic now, as security researchers are honing in on us, before we experience misfortune.
00:04:46.000
Moving on from a very pleasant matter—DHH, the creator of Rails, is not as nice as Mats.
00:05:02.080
While that's true, we can definitely learn from Rails and DHH about how they have dealt with security vulnerabilities.
00:05:15.440
Over the years, Rails has accounted for somewhere between 80 and 100 of the security issues that the entire Ruby community has faced.
00:05:29.040
This is because historically, Rails represented the entire attack surface of the Ruby programming language to the internet; if there were issues not involving Rails, there wasn’t much to attack.
00:05:45.840
If you had a problem with Rails, it effectively meant that you could attack nearly everything involving Ruby at that point.
00:05:56.800
What that leads us to is that the Rails security team has had to develop and refine a lot of the processes I’m going to explain.
00:06:04.800
Something really cool happened with Rails over the last year or so: we now have gems for basically everything. Rails 3 marked the great modularization of Rails—it's no longer a gigantic monolithic system.
00:06:18.560
You can pick and choose what components to use, and swap out some things for others. It’s a great time for bundler, which allows you to plug in any fork from GitHub with one line of code.
00:06:31.200
This functionality allows you to use a variant of the released and well-known code, which may or may not have security issues associated with it. However, having so many gems also implies that now there are potentially dozens or even hundreds of gems providing security vulnerabilities to your application.
00:06:50.400
For instance, in the first eight weeks of this year, there were security vulnerabilities fixed across several releases of Ruby gems, including bundler, json, rexml, rack, aural, active record, action pack, active support, and rdoc—even the latter had an XSS bug.
00:07:06.640
Even if you think to yourself, “Oh, I only use the standard library; I don’t use any of those Rails gems,” it’s very likely that you've utilized something on that list, which poses a significant problem.
00:07:21.919
So, what do we do when everything we could possibly use is facing simultaneous security issues? Updating is a huge pain.
00:07:34.400
Updating distracts you from feature development, and it’s tough to sell to a boss or manager who has deadlines for delivering what's deemed "business value."
00:07:47.040
Handling security updates is akin to purchasing insurance; your boss doesn't say, "Oh, we could have had two more weeks of runway if we didn't pay for insurance on our office." If your office burned down with everything inside it, it would be catastrophic.
00:08:05.760
Because of the small risk combined with large potential losses, it's worth investing that small amount in security updates. Security updates are essentially exploitation insurance.
00:08:24.560
There’s a famous analogy that says you don’t need to be faster than the bear; you only need to be faster than the guy behind you. If you can apply security patches before someone else, then it's likely that it will be someone else who suffers the consequences for not having patched.
00:08:36.560
Without updates, failures can be catastrophic. Just cleaning up from a single security breach can lead to significant amounts of lost time, effort, and lost customer goodwill.
00:08:53.039
Moreover, there are disclosure laws in various states that now require companies to inform every individual whose data may have been compromised. This could mean sending thousands of emails and facing potential civil and criminal penalties.
00:09:08.640
The key takeaway is to invest the time to do updates as patches are released, which greatly reduces the risk that you'll have to tackle these issues.
00:09:17.680
In conclusion, I understand that updating is hard work. It’s not enjoyable or cool. You won't get to present talks about how you successfully updated your Rails app from version 3.2.12 to 3.2.13.
00:09:29.680
But the reality is that if you do the updates, you can rest easy at night knowing that the chances of facing security incidents and their repercussions are significantly minimized.
00:09:46.480
As a developer, it’s likely that you use other people's code. So let’s talk about reporting security issues. Given that Ruby and Rails haven’t had to face many security issues before, the Ruby community doesn't have established cultural norms around security issues yet.
00:10:01.199
When you find a bug in a gem within your application, your first instinct might be to rush to GitHub and say, "Hey, your thing doesn’t work." But I urge you to stop and think for a moment before taking that action.
00:10:14.559
There are various reasons why it’s crucial to approach this carefully. The field of security research has established a well-known policy called responsible disclosure.
00:10:27.439
Responsible disclosure can be regarded as the best possible scenario except for all the other situations that are worse. It provides some level of assurance that no one ends up severely affected.
00:10:39.680
The first part of responsible disclosure involves informing necessary parties about the vulnerability so they can address it. Companies typically dislike this because they would prefer to keep quiet about losing sensitive customer information.
00:10:50.800
The other side of responsible disclosure is about being responsible. Clever individuals who discover vulnerabilities often feel the urge to publicize their findings to gain recognition for their cleverness.
00:11:04.480
Responsible disclosure means that instead of making the vulnerability public, the person who found the problem discusses it privately with the company, urging them to fix it before it becomes widely known.
00:11:24.480
Security researchers typically agree to this process, as it helps protect themselves from potential legal actions stemming from their discoveries.
00:11:38.240
When properly carried out, responsible disclosure is meant to foster a collaborative approach. Companies need to address vulnerabilities while researchers can communicate their findings without fear.
00:11:53.440
Some companies are trying to make this process more encouraging by rewarding security researchers. For instance, Facebook has an extensive responsible disclosure page that outlines how to report vulnerabilities along with a minimum reward for every valid finding.
00:12:09.040
On the other hand, GitHub incentivizes researchers with the possibility of receiving a t-shirt, although they don’t have a stated reward policy.
00:12:24.320
Engine Yard, however, has clarified that they don’t offer rewards. Unsurprisingly, being explicit about not providing rewards can sometimes lead to adverse reactions.
00:12:38.240
Now that you understand the overview, let's revisit the situation where you find bugs in other people's code.
00:12:50.480
Congratulations on responsibly discovering a bug; this is how software improves! However, before running to the issue tracker, please consider two questions regarding the bug.
00:13:01.679
First, can you access something that you, as the owner of an account on that website, shouldn’t be able to access? For example, can you see someone else's credit card number?
00:13:17.040
The second question is whether you can disable something for others. For instance, if you discover that refreshing a page multiple times makes the website unresponsive for everyone, that indicates a denial-of-service vulnerability.
00:13:31.520
If either answer is yes, you absolutely need to disclose the issue responsibly. Do not publicly post any sensitive information related to this vulnerability.
00:13:44.640
Contact the author of the library directly, as it’s essential to identify which specific library has the bug. Doing so makes it more likely to have the issue patched before it’s announced publicly.
00:13:56.000
This is particularly vital for larger projects like Rails, where timely patching can prevent issues from becoming public dangers.
00:14:06.240
If you're going to contact the author, ideally review their security policy, if they have one. Rails, for instance, has a security page and a private mailing list that is dedicated to security matters.
00:14:21.520
If no security information is available, check the gem specification file. Every gem contains an email address for the author in its spec.
00:14:37.440
As a last resort, you can find the maintainer's email on GitHub, which can help you initiate contact.
00:14:53.440
Most of the time, you might be faced with complications. I’ve seen situations range from not knowing whom to contact to having a maintainer who committed a fix but is unavailable to deploy it.
00:15:09.680
Please work together with the developers. They are also people who may not immediately respond because of their day jobs, but they do have the collective goal of ensuring safety.
00:15:24.000
Once you hear back from the author, offer to help. Sometimes they may appreciate the offer to resolve the security issue collaboratively.
00:15:36.160
After addressing the issue, ensure that the fix and the announcement occur simultaneously or even try to release the fix a bit earlier than the announcement.
00:15:51.200
It's crucial to avoid publicizing vulnerabilities—even if you’ve figured out how to solve them—until the fix is available in the current gem release.
00:16:09.440
If it comes to the point where you cannot contact anyone, take matters into your hands and fork the code.
00:16:17.680
If you encounter a security issue, it’s wise to fix it on your own, especially if you are already using that code.
00:16:30.560
However, only apply this strategy if the responsible party is unresponsive. Otherwise, working with them is ideal.
00:16:46.240
Finally, I'd like to touch on the topic of your own gems. Since many of you are Rubyists, I’d like to ask how many of you have written your own gem or library.
00:17:03.520
Ruby makes writing libraries straightforward, and a lot of developers have their own projects. However, every single one of those projects is a potential security vulnerability for someone else.
00:17:16.960
At this point, I want to discuss being on the other side of responsible disclosure—how do you responsibly receive reports of security issues, and how do you assist users who depend on your library?
00:17:30.280
Unless your code is perfect—a claim I'd urge you to question—there’s a good chance you might face this scenario at some point.
00:17:42.480
The simple scenario is where another Ruby developer politely emails you, stating, 'Hey, I found something that seems problematic. Can we collaborate on fixing this?'
00:17:55.680
In this situation, collaborate to fix it, review the change, release the fix, and announce the security issue to your user base.
00:18:09.440
The next challenge occurs when a bug is identified and is already publicly known. In this case, promptly addressing the issue and getting the fix to users is crucial.
00:18:22.560
If a vulnerability exists and has been publicly disclosed, you need to act quickly—the bad actors are likely already aware.
00:18:36.560
My rule of thumb is this: if additional publicity won’t significantly worsen the situation, aim to make users aware of the issue as soon as possible.
00:18:41.920
Finally, in the hardest scenario, a security researcher contacts you, possibly with information about remote shell vulnerabilities.
00:18:56.080
If you receive such an email, respond promptly and keep the lines of communication open. Regularly inform them about your efforts to fix the problem.
00:19:09.280
Be clear about your actions. If things go silent, researchers might feel inclined to go public with their findings.
00:19:23.760
As a gem author, ensure that you make it easy for people to report issues to you. Include your email address in the gem specs and on your GitHub profile.
00:19:37.600
If you're part of a library team, establish a security address or contact protocol. Creating a disclosure policy, even a brief one, encourages responsible reporting.
00:19:53.680
To support the Ruby ecosystem, I’ve established a mailing list for announcing Ruby ecosystem security releases. This space handles only security-related announcements, ensuring low traffic.
00:20:10.080
If we can follow the hard-won lessons learned by the security community, one day we can return to a state of calm and tranquility—and then we can go shopping.
00:20:26.080
Thank you for your attention! Do you have any questions? Unfortunately, we don't have time for questions, but thanks a lot for being here!
00:20:39.840
You.