Rack::Attack: Protect your app with this one weird gem!

Talks

Aaron Suggs

#developer-experience

#open-source

Rack::Attack: Protect your app with this one weird gem!

by Aaron Suggs

In the video titled 'Rack::Attack: Protect your app with this one weird gem!', Aaron Suggs, an Operations Engineer at Kickstarter, discusses how Rack::Attack, a Ruby middleware, can be utilized to handle abusive requests that often threaten the performance and reliability of web applications. This middleware helps developers focus on primary tasks by reducing the overhead created by malicious requests, naive scrapers, and other unwanted traffic.

Key Points Discussed:

- Introduction to Rack::Attack: Rack::Attack was developed at Kickstarter to block and throttle abusive requests, thereby improving overall site performance and availability.

- Abusive Requests Defined: Examples of abusive requests include login attempts from malicious actors, scrapers, and other forms of automated requests that can overload site resources.

- Origin Story: The necessity for Rack::Attack arose from a specific incident in 2012 when Kickstarter faced an attack aimed at cracking user accounts. This prompted the need for a robust solution to manage request loads.

- Middleware Functionality: Suggs explained how Rack middleware works by wrapping around applications, managing incoming requests and filtering them based on pre-defined rules using an elegant Domain Specific Language (DSL).

- Throttling and Blocking Requests: The presentation included practical examples demonstrating how to set up throttles for specific actions such as login attempts per IP address, and other custom configurations that can enhance security.

- Integration with Existing Tools: Rack::Attack complements other security measures such as hardware firewalls and CDN services, providing Ruby developers with a tailored and efficient way to optimize request handling.

- Community and Open Source Contribution: Suggs emphasized the importance of open source contributions from developers worldwide, highlighting how Rack::Attack has evolved through community support.

Conclusions: Rack::Attack serves as a key tool for maintaining web application performance amidst the inevitable challenges of the online environment. It allows developers to mitigate bad traffic efficiently enabling them to concentrate on feature development and user satisfaction. This gem not only secures apps but brings developer happiness by simplifying the complexity associated with abusive requests.

00:00:16.260 All right. Can people hear me okay? I'll go ahead and get started. This talk is about Rack::Attack and how to protect your app with this one weird gem.

00:00:20.320 So where does Rack::Attack come from? We built it at Kickstarter. If you haven't heard of Kickstarter, it is a funding platform for creative projects. If someone has an idea for a film, a comic book, an open-source project, or a gadget, they can put their project up on our site. They can offer rewards for various pledge levels, and their friends, family, and strangers on the internet can come and support them.

00:00:30.369 At the end of the deadline, if they’ve reached their funding goal, we process the transactions, and the creators get the funds they need to pursue their project. To give you a sense of scale for what we do, we recently crossed over a billion dollars pledged to the site, which averages over a million dollars a day across more than 60,000 creative projects.

00:00:45.580 A quick introduction: my name is Aaron Suggs. I go by K Theory on social media. I love dancing in my bear outfit, and I'm the Operations Engineer at Kickstarter. We have a very DevOps-style workflow, which means I end up writing a lot of Ruby code, and I love writing Ruby code.

00:01:05.740 Rack::Attack is a tool I wrote; it’s Rack middleware for blocking and throttling abusive requests. What do we mean by abusive requests? These can be malicious attackers trying to take down your site by attempting to crack user accounts or gain sensitive information, or they can be naively written scrapers who are just acting like people on the internet.

00:01:13.209 While this behavior is often harmless, the sheer volume of traffic can strain your app and consume substantial resources. Rack::Attack provides a very elegant DSL to address these issues and help constrain that behavior so your website stays up.

00:01:24.489 Rack::Attack is on GitHub at 'github.com/kickstarter/rack-attack'. It’s an open-source Ruby gem, and you can find a README there that outlines everything you’d expect from it.

00:01:35.320 The big wins that Kickstarter has experienced from using Rack::Attack are primarily based on performance. We developed it because we wanted to increase site performance. We had issues with abusive requests making our websites slow, as they consumed too much app server CPU or too many database resources. By constraining these requests, we were able to speed up the website for more important activities, like users wanting to watch videos or pledge money, rather than for those scrapers trying to download content.

00:01:58.540 We also improved our site's availability. Sometimes, the volume of abusive requests was so high that our site would go down, or we’d experience other issues resulting in downtime, which negatively impacted our availability. However, the biggest win we achieved was in developer happiness. Dealing with these bad actors on the internet, especially when it affects the site's stability, can derail developer productivity and the product roadmap. With Rack::Attack, we could focus more on building cool features rather than managing abusive requests.

00:02:30.160 Now, let me share the origin story of Rack::Attack: what happened at Kickstarter that made us realize we needed this tool. Let’s rewind to the summer of 2012. One Saturday afternoon, we experienced an unusual spike in failed login requests. The blue line in this graph shows our regular successful logins fluctuating throughout the day. Suddenly, we observed a surge of bad login requests.

00:02:52.570 For a while, we were confused: did we deploy a feature that broke login? No, someone was trying to crack our user accounts by guessing emails and passwords at speed from different IP addresses. As the operations guy, this issue fell in my lap, and I needed to stop it because it was detrimental to the site.

00:03:12.560 In response, I wrote a basic before-filter for our login action to keep a counter in Memcached. If it reached a limit, we'd display an error page. This was far from ideal since it altered a critical feature of our site under the pressure of urgency, and I had to apologize for the poorly tested pull request.

00:03:30.400 Reflecting on this later, I recognized we required a more elegant solution to prevent bad requests, as the login attack was just one of many types of problems we may encounter in the future. We always assumed it would be essential to throttle login requests; however, it continually lingered as a low-priority item in our ticketing system. In light of this incident, we realized we now needed a generic tool to tackle these bad requests.

00:03:56.960 Now we transition into the coding portion of the talk. Here’s an example of a basic Rack middleware. Middleware effectively 'wraps' around your application, whether it's a Rails app or a Sinatra app, allowing you to manage incoming requests from clients. Every request calls a method that passes in the environment, containing information such as the requested page and client cookies.

00:04:21.869 Rack middleware allows you to manipulate requests, log them, cache them, or potentially block them, which is the function of Rack::Attack. This is a simplified version of the Rack::Attack call method. It checks whether the request should be allowed; if so, it proceeds to your application, which does the intensive work of querying the database and rendering views.

00:04:37.370 If the request is deemed abusive, however, it quickly returns an 'Access Denied' response, avoiding the overhead of executing potentially heavy application processes. Rack::Attack can process several hundred access denied requests per thread, making it highly efficient.

00:05:02.990 The important part is configuring this 'allow' method, which you define based on what you want to throttle. Here's a generic throttle configuration example from an initializer to set up Rack::Attack. The key is that we call the throttle class method on Rack::Attack, giving it a unique name (for example, 'throttle by IP') and defining how the throttling is managed.

00:05:30.150 We specify the time period for throttle requests and the limit, allowing a certain number of requests during that time. For instance, you might allow 10 requests every 5 seconds, which provides flexibility for burst traffic while maintaining an average limit over time.

00:05:54.290 Next, we pass our request through to evaluate if it matches the criteria we’ve set. Inside the block, we determine the identifier for throttling. For example, if we want to throttle by IP, the block would return the client's IP address. If we want to throttle requests for a specific resource or API token, we would configure it accordingly. Returning nil or a false value allows the request to go through without throttling.

00:06:30.090 Next, we tackle the challenge of storing throttle state. Our Rails application can use its cache to store counters for each IP address. Using caching systems like Redis or Memcached efficiently allows us to keep track of these counters while incrementing them atomically whenever a request is made. The construct for every request includes incrementing the count per IP address.

00:07:16.709 For each request, we can build a unique key by combining the request type and IP address along with the current time divided by the period for tracking purposes. This approach maintains efficiency, as it allows us to store minimal data while being able to rotate keys with minimal churn on our cache.

00:07:38.370 The Rack middleware incrementally manages key access requests, allowing us to determine when a user exceeds their limit and trigger an 'Access Denied' response based on those requests. This setup provides us with a global throttle per IP address and improves site performance. Over time, we developed more features.

00:08:15.650 About a year after our initial success, we faced a new challenge that put Rack::Attack to the test. In the summer of 2013, a script called kick_sniper.py highlighted a problematic behavior on Kickstarter we termed 'reward sniping'. As I described earlier, Kickstarter users often set limited rewards for their projects.

00:08:40.860 In this scenario, a popular video game project offered desirable reward tiers that sold out quickly. Some enterprising Python developer created a script to refresh frequently and capture the available lower-tier rewards when someone updated their pledge. This led to a surge in requests aimed at changing pledges, resulting in excessive strain on our database.

00:09:10.260 As a result, the master database CPU usage spiked, leading to alerts for our entire dev team that our database was consuming too much CPU. When we investigated, it became clear that the high request volume was tied to this script that users were running to refresh their pledges.

00:09:29.420 Recognizing that this was a misuse of resources, we considered throttling users attempting to change their pledges. By limiting the number of requests a user may make, we optimized the pledge change process while balancing it against user experience. With only one line of code in Rack::Attack, we restricted the number of requests that users could perform, which brought the CPU load back to normal.

00:10:03.670 Implementing this change was both simple and effective, allowing us to prioritize performance without radically altering our existing functionalities. Rack::Attack proved its worth as a solution to enhance our application's resilience for high-stress scenarios.

00:10:25.680 Now I’ll share some general pro tips to help you maximize Rack::Attack for your application. We previously discussed throttling access based on IP. However, we want this approach to be even more granular for login requests, which typically require stricter controls.

00:10:49.090 To implement this, we configured a new throttle called 'logins per IP' to limit the number of login attempts an IP address could make. We ensure that this control applies only to post requests going to the login URL.

00:11:06.740 Additionally, we needed to address the potential threat of attackers using various IP addresses to crack passwords for a specific email address. Hence, we modified our block to return the email parameter instead of the IP address, thus allowing us to limit login attempts across all IP addresses for that specific email.

00:11:30.380 This kind of throttling mechanism should ideally be incorporated into most applications to prevent common abuses before they escalate. Another feature we find beneficial in Rack::Attack is the blacklist.

00:12:01.780 With blacklisting, we can outright deny requests from known harmful sources without trying to throttle them first. Implementing a blacklist allows important sections of your app, like admin areas, to deny access based on certain IP addresses, improving overall security.

00:12:31.610 For instance, in Kickstarter, we restrict access to our admin section to specific IP addresses using this feature. If a request is made from an unauthorized IP address, it is immediately denied, preventing any malicious access.

00:13:02.160 Another handy feature included within Rack::Attack is that it supports Active Support notifications. As a Rails app, these notifications get triggered every time a request is blocked or throttled, allowing developers to navigate and graph request patterns effectively.

00:13:26.820 As we wrap up, I want to stress that while Rack::Attack is an excellent tool for improving security, it should be part of a comprehensive strategy that includes using firewalls and other protective layers like IP tables, limit con, CDNs, or web application firewalls.

00:13:56.800 Rack::Attack shines particularly when embedded within your Ruby application because you can leverage your application's logic for security. It’s also easier to test and deploy, as everyone on our team is familiar with Ruby language.

00:14:20.780 In closing, I’d like to thank the many contributors on GitHub who have helped shape Rack::Attack by adding features and improving documentation. Their input has made a significant impact on the community, and I appreciate how globally distributed this collaborative effort has become.

00:14:48.100 We live in an era where we will face unpredictable challenges online, and it's reassuring to have tools like Rack::Attack ready to maintain website operations. It's essential for the web to remain a place of innovation while ensuring our applications remain robust.

00:15:00.000 That's all I had on Rack::Attack at Kickstarter. If you have any questions, I’d love to address them now. If you're more comfortable, please don't hesitate to find me on Twitter or after the talk.