Ruby & You

Talks

Ruby & You

Terence Lee

#developer-experience-dx

Ruby & You

by Terence Lee

The video titled "Ruby & You" features Terence Lee discussing his journey and experiences contributing to the Ruby programming language, particularly in context of addressing serious security vulnerabilities. The main focus of the talk is the significant security exploit, CVE-2013-4164, that was revealed in November 2013, affecting Ruby runtime versions 1.8.7 and 1.9.2. Heroku's initiative to provide support for legacy Ruby versions is a central theme, showcasing how companies can take proactive measures to protect their customers.

Key points discussed include:
- Ruby Hero Award: Terence starts with a light-hearted note about receiving a Ruby Hero Award and reflects on community interactions.

- Community Contributions: He emphasizes the importance of community help and collaboration, acknowledging his mentor Zzak and other key figures at Heroku and Ruby’s development team.

- Security Exploit Explained: The heap overflow exploit in floating point parsing is detailed, illustrating its potential impact on applications using Ruby, particularly in parsing data from user inputs.

- Heroku’s Response: When Ruby core provided patches for more recent versions only, Heroku decided to maintain and secure versions 1.8.7 and 1.9.2 for customers still using them, which involved backporting security fixes.

- Joining Ruby Core: Terence narrates the process of how he obtained commit access to Ruby core to help maintain these versions and the collaborative support he received through this journey.

- Encouragement to Contribute: He encourages developers to get involved, explaining how to contribute to Ruby core through patches, working on documentation, and enhancing the Ruby experience across platforms.

- Responsible Disclosure: A critical point discussed is the importance of responsibly disclosing security vulnerabilities to uphold community trust.

- Future Aspirations: Terence expresses desires for improved community engagement, transparency, and the consolidation of onboarding resources to assist new contributors in joining Ruby's development efforts.

The talk concludes with a call for increased participation within the Ruby community, aiming to make Ruby a better programming language for all. Additionally, attendees are reminded of the need to upgrade from outdated versions of Ruby to avoid potential security risks.

00:00:00.160 Good morning, everyone! So today's Friday, and we made it to Friday!

00:00:07.200 I received a Ruby Hero Award for being a Friday Hug superhero this past year.

00:00:12.800 I was going to do one now, but it seems like a lot of people are still sleeping. So, Johnny asked me to do it later in the day. Unfortunately, I won't be doing a Friday Hug during my talk.

00:00:24.880 Here's Aaron doing a Friday Hug with his cat.

00:00:32.399 I thought there was a game night tonight, so I actually brought these Lord of the Rings cards, which is a co-op card game that weighs about 10 pounds in my suitcase. However, Aaron told me it actually happened yesterday. I had to practice my talk, so I didn't get to play with anyone. If anyone wants to learn more about it later, feel free to come talk to me after my talk.

00:00:50.000 So, I'm Terence, and I come from Austin, Texas, the taco capital of the nation. I also found out that we have these really awesome hand-cleaning machines. This is at a local barbecue joint called Rudy's.

00:01:03.760 They have these hand jacuzzis where you literally just stick your hands in, and it washes your hands for you. They're super clean, and they even have awesome stickers you can wear afterward. So that's pretty cool. If you ever come to Austin and I'm in town, I will take you to get some tacos or to clean your hands.

00:01:20.960 Additionally, I wear blue hats, if you haven't noticed on Twitter or while I've been walking around. We've formed a Blue Hat League, and here are pictures of Chris Kelly from GitHub and Jessica from Ruby on Ales on Bundler Core.

00:01:38.720 I also work for Heroku, and like Richard was suggesting, we all have RTFM basically moniker acronyms. I'm the Rude Task Force Manager for the team. I appreciate Heroku a lot for allowing me to work extensively on open source and give back to the community. One of the things that is really amazing is that we sponsor three people to work full-time on Ruby and move that project forward.

00:02:11.760 Internally, they're called Matt's Ruby Team, and obviously the leading member of this team is Matt himself. He goes and directs that team, manages them daily, and deals with what projects they work on. They also have regular meetings to figure out these matters.

00:02:30.720 Among the team, there's also Koichi, who works full-time on performance improvements to the VM. If you were around during the 1.8 days when we moved to 1.9, he worked on YARV, which is the current VM implementation. More recently, in 2.1, he handled almost all the garbage collection improvements. He is deeply invested in Ruby's performance and spends all his time at Heroku working on it.

00:03:11.840 Lastly, there's Novo, who works full-time on fixing bugs in Ruby, which is an impressive contribution! The fact that Rails has worked well with the recent 2.1 release involved a lot of effort on his part in fixing reported bugs. He has the nickname "the Patch Monster" among the core team because if you look at the commits in Ruby since 2012, you'll find him at the top with over two thousand commits.

00:03:44.799 In fact, there's also an SVN bot that has 710 commits. I'll delve into what that bot actually does later in this talk. I'd like to highlight another figure here in this talk, Zzak, who is the fifth human committer. I'd like to propose that next year, Zzak should be recognized as a ruby hero.

00:04:09.600 He's been instrumental to my effectiveness on the Ruby core team. He has been a mentor to me and helped onboard me. He leads the C Ruby documentation efforts and cares greatly about it. He also assists with general open source matters, and whenever I need help, he is more than willing to help me get my tasks done for the betterment of the community. It's hard to find someone else who cares as deeply about the Ruby community and its success.

00:04:54.080 Now that I've covered that ceremony and those acknowledgments, let’s get into the actual talk. I’ll discuss my journey of how I got onto Ruby core and how it happened.

00:05:12.960 Next, I’ll explain how you can contribute to Ruby. Lastly, I want to conclude with some projects and ideas on my mind regarding Ruby and things you all can help with.

00:05:31.360 Let's rewind back to November of last year. There was a notable heap overflow when parsing floating points in Ruby, which was assigned the CVE identifier 2013-4164. If you review my slides later and click this link, it will take you to the news regarding Ruby. I want to explain the exploit to showcase the impact and severity it had.

00:06:06.479 Essentially, if you took any amount of untrusted data and converted it to floats, you were potentially vulnerable to this exploit. While that may not seem bad in isolation, if you're working with services where you handle a Marshall, JSON, or YAML, and you’re parsing anything with a float, it calls a 2f on it.

00:06:23.759 For instance, in a Rails app, if you're parsing a JSON body where a user can input a floating-point value, your application could be exposed to this bug. The Metasploit project allows you to test if your application is vulnerable to various known exploits, so I extracted some code from Metasploit specific to the CVE.

00:07:02.639 They provided a method to construct a malicious floating string that, when parsed by JSON, causes Ruby to segfault. In a Rails application, where you often deal with random JSON data, this could lead to a simple denial-of-service scenario by crafting specially designed floating-point values and crashing numerous Rails servers.

00:07:21.199 When Ruby posted about it and I reviewed the affected versions, I realized this impact affected almost everything running in production so anything beyond 1.8.6 was vulnerable.

00:07:38.720 This included the entire 1.8.7 series and the latest patch levels. For example, patch 484 was released the same day, and 353 preview 2 along with the trunk revision were also affected. If you were running any Ruby version at that point, you were effectively vulnerable. It wasn't just a theoretical problem; it had a significant practical impact, especially for Rails servers.

00:08:13.280 In their post, they outlined solutions on how to work around the issue and how to patch your Rails app running in production to protect against these vulnerabilities.

00:08:31.680 One notable detail was that they only released patches for 1.9.3, 2.0.0, and 2.1. As I reflected on this, I remembered that at Heroku, we had customers still using 1.8.7 and 1.9.2. 1.9.2 was part of the original Ruby on the Cedar stack.

00:09:02.640 In a subsequent news post, there was a quote indicating that the Ruby 1.8 series or any earlier releases were already considered obsolete, with no plans to release new fixed versions.

00:09:38.240 Simply put, if you were running 1.8, you wouldn't receive a patch. This post didn’t even mention 1.9.2, even though they had stated it was vulnerable. I inquired with people in Japan about Ruby 1.9.2, and Ayumu tweeted that Ruby 1.9.2 was ‘dead.’

00:09:56.000 This was something done at Ruby Kagi last year, and while I was present, I don’t recall seeing this. Yuki, who was the maintainer for 1.9.2, confirmed this sentiment, making it clear Ruby 1.9.2 was essentially dead.

00:10:07.760 At Heroku, we were dealing with a few issues. We had several customers on those versions, which led to a couple of options. We could choose to patch all versions and then be done with it, or go the extra mile to maintain 1.8.7 and 1.9.2 for our customers.

00:10:30.000 Another option was to do nothing and inform customers that Ruby core no longer supported those versions, leaving them on their own. We weighed the pros and cons extensively, especially as customer applications were vulnerable.

00:10:56.240 On one hand, forking and maintaining Ruby versions not supported by core was not our specialty. On the flip side, telling people on the day of the incident that we wouldn't patch the vulnerabilities was the worst possible response.

00:11:27.920 We decided to release two Ruby versions, distinct from vanilla Ruby, that were not maintained by Ruby core. We reviewed the fix, backported it, and confirmed it resolved the vulnerabilities.

00:11:49.680 We took the latest releases of each affected version, incremented the patch level, and applied the fix. For instance, we incremented 1.9.3 by one and released 1.9.3.0 and 1.8.7 by one to release 1.8.7.0.

00:12:11.760 We created a GitHub fork where all this work was available, allowing users to download the source and build the specific Ruby versions we were instructing customers to use locally.

00:12:48.240 Eventually, Ruby Build also began supporting those versions we built.

00:13:00.000 We notified Ruby core of our plans: ‘Hey, is it okay if we’re creating patch levels that are not part of the Ruby version?’ They were supportive, and we aimed to help commit these fixes upstream.

00:13:24.240 After lengthy discussions, it was suggested I join Ruby core to help maintain these versions. Zzak, who sponsored me for this, was the first to advocate for my commitment to Ruby.

00:13:54.000 He guided me through the process, where I had to send an email to Matt, the final decision-maker, to express my desire to maintain 1.8.7 and 1.9.2 and provide examples of my previous work.

00:14:25.560 If chosen, I would then receive input from core team members regarding my commitment, and Matt would ultimately approve or deny it.

00:14:38.080 Fortunately for me, I got on Ruby core, and Zzak tweeted about my achievement. Together, we drafted a statement extending the maintenance of 1.8.7 and 1.9.2.

00:15:09.280 This decision made sense, as if I'm going to maintain these versions in the context of Heroku, there’s no point in doing it in isolation. It’s beneficial for the broader community who still rely on these versions.

00:15:42.000 Despite our freedom to commit to the actual repositories and branches on Subversion, we could not cut a new release as a standalone entity. We could only push commits upstream.

00:16:13.600 So there's a lot of ways like a community can get involved with Ruby core. You can send enough patches to the point where Ruby core gets tired of you submitting them, prompting them to grant you commit access.

00:16:51.520 Another valuable contribution is working on non-POSIX platforms, like enhancing the Windows Ruby experience, which is quite beneficial for users.

00:17:19.280 If one creates their own library and it gets integrated into the standard library, they automatically become a part of Ruby core, similar to how Eric Hollow has maintained RubyGems.

00:17:53.920 However, this path for joining as a new core member is less common now as community members strive to keep the standard library manageable, especially with the gemification trend.

00:18:22.920 Currently, I highly encourage anyone interested in contributing to Ruby core to maintain security back-ported versions.

00:18:40.000 Let’s briefly check in on those still on older versions of Ruby. How many people are in the room yet still on 1.8.7? (Pauses for group response) Wow, nobody! What about 1.9.2?

00:19:10.000 (Audience response: a few hands raised) We’re ending support for that in June 2014, so I recommend you upgrade if you find yourself stuck on Ruby 1.9.2.

00:19:40.000 As for 1.9.3, we have a fair number of people still using it, but the bug fixes are restricted to security only. The end of this support is due in February next year, which is right around the corner.

00:20:05.000 It's imperative that you transition off of it soon unless you want to handle your own security maintenance, which I can contest from experience is not an enjoyable task.

00:20:41.000 Now, let’s discuss how you can get involved in Ruby itself.

00:20:57.000 To get started, obtain the source code of Ruby, primarily available via Subversion. It’s recommended to use Subversion for this purpose, and you can check out the trunk repository.

00:21:24.000 The trunk serves essentially the same purpose as the 'master' branch in Git, hosting all active development. The upcoming Ruby 2.2 is slated for release this Christmas, with all ongoing development occurring in the trunk.

00:22:00.000 Core developers talk primarily about trunk, so it's beneficial to familiarize yourself with it. Meanwhile, you might want to check out specific release branches using Subversion.

00:22:36.000 I went through the experience of using git-svn to interface with the Ruby repositories. Just a heads up: expect it to take quite some time to clone the repository, as I once took four days to complete that process.

00:23:05.000 On that note, if you lose network connectivity during the process, you can simply run git-svn rebase, allowing you to continue without starting from scratch.

00:23:40.000 Along the way, I learned a few things about working with patch levels in Ruby. For example, if you don’t increment the version.h file for each commit, your changes won’t go through.

00:24:09.000 When I first submitted a patch, I failed to conform to Japan Standard Time for dates, which earned me a nice commit message from the SVN bot, reminding me to comply with the date requirements!

00:24:49.760 Once you have the code, you’ll want to run tests to ensure it’s functional. It's advisable to build Ruby outside of your working copy to avoid generating unwanted artifacts.

00:25:09.000 Creating a separate directory for building can help keep things organized. You can set the prefix during installation to manage where Ruby ends up and this allows for better organization.

00:25:40.000 It's also essential to link libraries that aren't readily available. If you're using Homebrew, ensure you link to required dependencies like OpenSSL, readline, zlib, etc.

00:26:06.000 Ruby's testing suite is entirely written in Ruby, using a few units and checks to verify functionality, allowing for maximum compatibility.

00:26:52.000 If you're working with the Redmine issue tracker for Ruby, at times you'll need to create patches and attach them to the reported issues or feature requests. Additionally, small bugs can be directed through GitHub pull requests.

00:27:19.000 For larger discussions or bugs, it may be best to use Redmine directly, as it helps maintain track of relevant conversations and submissions.

00:27:41.000 Each time you submit an issue on Redmine, attach useful information, such as details on how to reproduce bugs or the relevance of a feature. This should include a proper description of what you are reporting, making it easier for others to respond.

00:28:29.480 I also suggest keeping track of who maintains what branches and the responsibilities each maintainer holds. This will guide you in assigning your requests to the correct individuals.

00:29:07.000 When I submitted proposals for policy changes, I initially overcomplicated them in lengthy formats, only to realize they should be concise and straightforward. Always consider language proficiency when communicating in diverse teams.

00:29:55.200 A better approach is to write brief points, linking to previous threads and allowing discussions to unfold thereafter.

00:30:07.000 Successful proposals can lead to documented processes on platforms like Redmine or contributions towards creating policies that benefit the community.

00:30:54.400 Ruby boasts a dedicated security team alongside a mailing list for discussions. Membership is open to C Ruby committers and developers working on Ruby implementations, such as JRuby.

00:31:23.760 When identifying security vulnerabilities, responsible disclosure is vital for maintaining the integrity of the community. Risking exposure can lead to exploitation and unwanted scrutiny of the development team.

00:32:02.000 When you report a security vulnerability, including a reproducible case is crucial, as it helps maintainer branches assess if their specific version is affected.

00:32:30.000 As part of my role maintaining 1.8.7 and 1.9.2, it’s critical to analyze these requests carefully before taking action.

00:33:00.000 Just this past month, a significant discussion emerged regarding OpenSSL's use of insecure ciphers by default in Ruby. We need to clarify which party is responsible for maintaining security standards on this front.

00:33:35.840 In conclusion, the balance of maintaining a secure cipher list will shift to Ruby as OpenSSL updates its standards.

00:34:09.760 While we want to uphold security through various patches, we must ensure the changes remain backward compatible and do not disrupt existing applications.

00:34:43.760 For developers wanting to join Ruby Core meetings, be sure to draft an agenda and seek approval from lead contributors to ensure a productive session.

00:35:24.000 Regarding versioning, while Ruby’s versioning system aims to mirror semantic standards, there's a slight deviation from classic semantic rules.

00:35:58.880 Major version changes signify significant updates, while daily versions are expected to roll out every year. The maintenance of patch versions finds a rolling schedule for fixes organized every two to three months.

00:36:38.800 Just recently, I received a message requesting participants for upcoming developer meetings to discuss urgent topics, including a major OpenSSL issue requiring attention.

00:37:11.040 To summarize my experience working with Ruby Core, it’s crucial for one to communicate concisely, avoiding unnecessary complexity in proposals.

00:37:56.160 Code submissions are appreciated because they allow quick evaluation by fellow Ruby core members, easing future discussions and developments.

00:38:34.000 I’d like to wrap up by sharing my aspirations for improving Ruby's community involving transparency, trust, and long-term support for legacy versions.

00:39:32.800 I am particularly interested in enhancing onboarding resources, striving to consolidate useful information to guide new contributors towards participation.

00:40:12.480 My concluding wish is to witness increased involvement in Ruby core and to collaborate with you all in elevating Ruby into an even better programming language.

00:40:36.000 Thank you for being a fantastic audience!

00:48:18.000 (Applause)

Terence Lee

@hone

Ancient City Ruby 2014