00:00:00.299
Hello, everyone! I hope you're all having a fantastic Wednesday so far. Thank you for joining and listening. Hopefully, you're excited to learn a little bit about RubyGems MFA and its past, present, and future. I'm Jenny, and I'm a developer at Shopify on the dependency security team, where we help secure Ruby's supply chain. I also recently became a maintainer on RubyGems.org, which is quite exciting!
00:00:20.580
What's even more exciting is that this is my first ever time giving a talk at a conference, and I'm so glad that RubyConf Mini is the place where I'm doing it. Recently, some of you may have seen that in June, RubyGems announced that they are requiring owners of popular gems to have MFA enabled. Some folks from Shopify and I helped create this change, so today I'm going to tell the story of how this policy came about. This includes discussing why MFA is important and why we need it for RubyGems accounts, how we implemented the policy, and lastly, what’s coming next.
00:01:04.320
To start, let me ask a random question: raise your hand if you've ever heard of the term 'goiter' and know what it is. Oh, okay, we got a couple of hands! For those who don't know, a goiter is a condition that makes your thyroid gland abnormally large, causing your neck to appear swollen. The cause of this condition is iodine deficiency. I found out yesterday that I’ve been pronouncing 'iodine' wrong—some people say 'iodine' while I say 'I didn’t.' So, bear with me if I mispronounce the word! Iodine is essential for thyroid hormone production, and goiter can result from an insufficient production of this hormone.
00:02:10.640
Up until the past century, iodine deficiency was a global public health issue. In ancient times, people treated goiter by consuming seaweed or sea sponge. It wasn't until the 19th century that iodine was discovered and determined to be the solution. Then, it was discovered that iodizing salt was an effective way to add iodine to people's diets as a simple preventative measure. Since then, 120 countries have mandated salt iodization, and areas that enforced this, like Canada, have removed the need to monitor iodine levels in the population because it mitigates a significant risk.
00:03:32.640
So, why have I been talking about goiter in the last few minutes? This is RubyConf and not GoiterConf! However, there are loads of similarities between goiters and supply chain attacks. There has been an exponential increase in software supply chain attacks over the past few years. Sonatype reported an average increase of 742 percent year-over-year in the past three years. For those who don't know, the supply chain consists of all the components that help create a piece of software. A supply chain attack occurs when a malicious actor targets specific components within the supply chain.
00:04:08.400
Examples of supply chain attacks include an undercover employee pushing malicious code directly to main, bypassing any code reviews, or obtaining access to the CI or deployment infrastructure to insert malicious code. More relevantly, one of the attacks we're most interested in is when your piece of software uses a package or gem containing malicious code. The second most common attack is account takeovers, while the first is typo-squatting. If you're interested in learning more about these topics, I highly recommend Ashley Pierce and Betty Lee's RailsConf talk, "Gem Install: What Can Go Wrong" and how to take over a Ruby gem.
00:05:02.759
Back to account takeovers—if a malicious actor gains access to your RubyGems.org account by cracking a reused password, they can sign in and generate a key that allows them to publish gems. Additionally, if the attacker has access to your key through other means, such as your Git history, they can set an environment variable—'GEM_HOST_API_KEY'—to authenticate as you. Thereafter, they can release a new version of the gem with malicious code by downloading the gem contents, inserting the malicious code into a file of their choice, bumping the gem version, building the gem, and then pushing that gem up to RubyGems.
00:05:54.720
Anyone installing this gem will be affected. An example of this was the rest-client gem. In 2019, someone took over the job maintainer's account and inserted code that executed commands from a Pastebin URL. In that Pastebin, dangerous actions were performed: it sent the URL of the affected host to the hacker's defined endpoint, transmitted all of your secrets to the hacker, and if your app had clients, it would send user emails and passwords by overloading the 'authenticate' method in identity. Worst of all, if the attacker sent a cookie that matched the regex defined, the contents of that cookie could be executed, allowing the hacker to do whatever they wanted.
00:07:21.240
However, the rest-client gem isn't the only gem affected—strong-password was taken over and suffered a similar injection by evaluating a Pastebin file. This highlights just a few instances within Ruby, and these incidents also occur in other package ecosystems like npm and PyPi.
00:08:00.780
Fortunately, there is a simple preventative measure to combat account takeovers, and I believe you might already know what it is—it's multi-factor authentication, or MFA! Adding another factor during authentication significantly complicates the process for someone attempting to log in as you. Following the incidents I mentioned, gem maintainers rotated their passwords and enabled MFA.
00:08:26.019
I mentioned there are many similarities between goiters and account takeovers, so what are they? Both represent widespread problems: iodine deficiency was a global health issue, while supply chain attacks are becoming increasingly common in software development. Both problems can be addressed with effective and simple preventative measures, namely iodization for goiters and multi-factor authentication for account security.
00:09:05.680
However, implementing these countermeasures requires changes from specific groups. For iodization, producers of salt must invest in equipment, hire staff, and obtain materials, while for MFA, package maintainers may need to invest time in publishing and maintaining their gems, which can be burdensome.
00:09:44.280
So why have laws and policies been created in these cases? The answer is simple: the cost of preventing these issues is significantly lower than the costs incurred when they happen. When people fall ill due to iodine deficiency, they incur medical expenses which are costly for both the individuals and the healthcare system. Similarly, when malicious packages are released, time and effort must be expended to remove the gem, draft advisories, and communicate the incident. This can have a mentally taxing effect on users of the gem—they may need to ensure their applications remain safe by rotating their secrets, and if they have clients, inform them of potential breaches and possibly downgrade the gem.
00:10:38.280
I hope I’ve convinced you of the importance of creating an MFA policy for RubyGems accounts. Now, in the next portion, I will discuss how a policy like this is made. But while we're talking about MFA, let's take a moment to review the state of MFA in RubyGems before any policies were enforced.
00:11:09.420
Currently, RubyGems has MFA in the form of time-based one-time passcodes, or TOTP, which you can install via an authenticator app like Google Authenticator or 1Password. Once enabled, there are three levels of MFA requirement. For UI-only scenarios, you need to enter a verification code upon logging into your account. For UI and gem sign-ins, you will enter an OTP code when publishing your gem. Lastly, for UI and API roles, in addition to UI and gem sign-ins, you will need to input a one-time passcode for all privileged actions like gem pushes.
00:11:57.000
You may wonder why there are so many levels—it can be confusing. The main reason is that continuous integration and deployment systems complicate MFA implementation. At Shopify, we use the Ship It engine for gem deployments. After clicking the deploy button, a script triggers gem builds, tagging, and publishing to RubyGems. Because of this automated process, it is impossible to enter an OTP code in the workflow.
00:12:39.000
Additionally, many maintainers utilize GitHub Actions. Another notable feature of RubyGems is that gem maintainers can enforce MFA by setting the 'ruby_gems_mfa_required' field to true in their gem specification, ensuring that all gem owners have MFA enabled.
00:13:16.100
Now that you know how MFA works in RubyGems, let's discuss how this policy came into being. There are several factors to consider when crafting such a policy. In hindsight, I would categorize the key aspects into three main areas: how many users should be required to enable MFA; what would MFA enforcement look like; and how can we make this change seamless for gem maintainers.
00:13:41.420
To ensure that the decisions made are right for the community, we sought input from the community. As with many open-source projects, RubyGems has a request for comments (RFC) process in place for significant changes. Their RFCs reside within a GitHub repository. Before logging a formal proposal, we typically discuss ideas openly to gauge community interest.
00:14:05.040
My colleague, Jacques, opened an issue with questions to solicit initial thoughts, and we received overwhelmingly positive feedback. This validated the views of our team and provided valuable suggestions for the proposal's first draft. Now, let's revisit the three areas I mentioned earlier and how the community feedback influenced each of them.
00:14:50.820
Firstly, how many users should be required to enable MFA? Ideally, we want all users to enable MFA, but it may not be best to enforce this immediately. An iterative approach could be more effective—learning from each iteration. The better question becomes: how many users should we enforce MFA for first? It was agreed that gem maintainers with the most significant impact when compromised should be prioritized. A good measure for this in RubyGems is the total gem download count. Luckily, RubyGems offers a daily dump of gem information for public use. From the previous year, there were a staggering 10.6 billion downloads in RubyGems' history.
00:15:47.160
I queried the total downloads for the top X number of gems, and I discovered that after the top 100 gems, the increase in download numbers significantly declines, covering more than a third of all total downloads. Out of over 180,000 gems, this information was quite eye-opening. I revisited the data to gather more points and plotted it on a graph—you'll see that, after the top 10,000 gems, they account for the majority of RubyGems' total downloads.
00:16:34.380
Next, let's address what MFA enforcement would look like for users. On the surface, it seems simple: just mandate that users enable MFA. However, we need to determine the level of MFA that should be required and which actions should be protected if users don't have it enabled. For starters, we agreed that UI and API authors should have MFA enforced—users will need MFA to create an API key necessary for gem pushes. Additionally, they will need a code to sign in to the UI and create a key.
00:17:13.740
However, if someone already has access to a key, they can publish a gem without needing MFA for UI and gem sign-ins. This enforcement was chosen due to the CI integration case mentioned earlier. Based on community discussions, we received several suggestions to hone API key scoping. For example, enabling MFA on specific keys could help if some keys are used for automation, allowing for more flexibility without compromising security.
00:18:02.460
Another suggestion was implementing key scoping so that actions would be limited to specific gems, allowing maintainers working on one gem to avoid needing access to all gems. Additionally, we discussed adding expirations to access keys, removing them from use after a certain period. Next, we focused on which actions to protect if a user doesn’t have MFA enabled.
00:19:02.340
After deliberation, we decided to protect all authenticated pages on the website. If significant changes need to occur on an account, we want that to be noted clearly—for example, when users sign in, they would be unable to access those pages without MFA. For the command line, we determined that it would be best to protect actions requiring an OTP code, particularly for actions that modify accounts since these are the most critical for maintaining security.
00:19:56.820
Finally, we aim to make the requirement for MFA the most seamless transition possible. Like the deprecation of features, it's good practice to give users time to adjust before enforcing any changes. Not everyone has a perfect habit of updating their systems. Hence, during this onboarding time, gem maintainers will receive warnings on their first sign-ins and for specific actions on the command line if they haven't enabled MFA.
00:20:45.660
After considering all these factors, we drafted an RFC outlining our discussions for community feedback. There was invaluable feedback with suggestions that refined this policy. For instance, it was noted that the number of top gems could shift over time, so if a gem moves from rank 100 to 101, they wouldn’t be required to have MFA enabled. Conversely, if a gem becomes more popular, like moving from rank 101 to 100, then it must enforce MFA.
00:21:38.460
Another suggestion was to use download thresholds as a metric, whereby once a gem surpasses a certain threshold of downloads, MFA would be required. When they approached this threshold, we can notify maintainers that they will soon need to enable MFA. Another vital point raised concerned communication—some users may not frequent the platform during transitions, so we should take steps to inform them of the need to enable MFA.
00:22:26.580
For this, it was recommended to post a blog, send emails, and share the information on social media to maximize awareness. Along with logistical feedback, underlying hesitance existed regarding the pressure to push forward this initiative. Concerns were raised about gem maintainers' volunteer nature—they faithfully devote their free time to nurture gems and may feel burdened by additional requirements.
00:23:27.480
Listening to these concerns is essential. We can address them and help people understand the bigger picture. While it may seem time-consuming for maintainers to enable MFA now, when an incident occurs, addressing the fallout requires considerably more time, effort, and energy—not just for maintainers but also for users affected by compromised gems. Another crucial perspective is that other package managers are also adapting these policies.
00:24:05.300
For instance, npm announced their policy requiring MFA for the top 100 gem maintainers, and PyPi is following suit, while GitHub plans to enforce MFA for all users by the end of 2023. Collaboratively working toward an industry standard would foster acceptance of such measures, reducing the hesitance to advance these initiatives. Moreover, support from trusted community members adds significant value in assuring maintainers.
00:25:05.220
With these strategies, our RFC was merged in March, and in June we publicized the policy. We informed users owning gems with over 165 million downloads about the importance of enabling MFA and encouraged them through prompts on the website and command line. In August, we began requiring MFA for users whose gems reached at least 180 million downloads.
00:26:02.760
We reiterated this change and began protecting actions for users that did not have MFA enabled. Furthermore, we dedicated time to implement features for API keys based on community feedback—users can now scope an API key to a specific gem and select whether or not that key requires MFA.
00:26:50.160
Feedback from the community has been overwhelmingly positive—most, if not all, users see the necessity for this change and appreciate that this initiative is one of the first steps toward securing the supply chain. As stated, for the top 100 gem maintainers, MFA is required, but we’re just getting started with rolling it out further to prevent account takeovers on RubyGems.
00:27:52.680
One point of feedback raised during the rollout was the demand for security key support. Users expressed interest in using biometric devices, as they are simpler than using TOTP—it’s simply a touch rather than a frantic race to enter a six-digit code. They also noted that security keys are harder to phish, further emphasizing their security benefits.
00:29:02.660
In order to extend the MFA policy rollout to more users, we want to ensure enabling MFA is as seamless as possible. We are actively working on adding that support. Additionally, we need to investigate how CI and automation can safely perform actions on behalf of users—this effort is crucial to elevate security for all privileged actions.
00:29:59.440
Lastly, we recognize that gem maintainers are human and can lose their second factor. Therefore, a sustainable process for MFA resets is necessary before expanding MFA requirements to potentially all users. RubyGems currently addresses these requests on a voluntary basis.
00:30:53.820
To support and fund this work, Ruby Central recently announced a partnership with Shopify, with Shopify contributing $1 million over four years to fund security initiatives for RubyGems.
00:31:03.900
In closing, we explored the past of RubyGems MFA and discussed its importance, the present landscape of MFA after policy implementation, and what the future holds. Initially, at this talk, we noted that not many knew what goiter was. This rarity highlights the possibility that, one day, account takeovers and supply chain attacks could become equally uncommon.
00:31:25.020
If there's anything you take away from this talk, please enable MFA, regardless of how many gem downloads your own gem has. It's the best practice for security. And for those who aren't gem maintainers, enable MFA on all your accounts.
00:31:57.660
As a token of appreciation for those who have enabled MFA, I have some stickers designed by my colleague Betty. They're quite adorable! Feel free to grab one after this talk. Thank you for listening, and I would like to thank Ashley, Betty, and Jacques from my team, and the RubyGems maintainers, especially Joseph, for their help in rolling this out. If you have any questions or would like to chat, I’d love to connect with you afterward!
