Rails Against the Machine

Brittany Martin

Brittany Martin

Rails Against the Machine

by Brittany Martin

In the video presentation titled 'Rails Against the Machine' by Brittany Martin at Ruby on Ice 2019, the speaker addresses the challenge of managing malicious users within online platforms, particularly in the context of ticket sales via a Ruby on Rails application. Martin, the lead web developer for the Pittsburgh Cultural Trust, provides insights on how to effectively identify and deal with 'bad users'—individuals who exploit systems for personal gain while masquerading as legitimate customers. Key points include:

Identifying Bad Users: Martin emphasizes the need for businesses to recognize patterns of negative behavior among users. Through collaboration with customer support and review of user personas, companies can better understand the profiles of bad actors, who may engage in practices such as ticket reselling, returning items fraudulently, or manipulating online voting systems.
Case Examples: Specific examples illustrate the issue, such as ticket resellers acquiring large quantities of tickets for resale at inflated prices and serial returners who exploit return policies by substituting product labels for profit.
Mitigating Bad User Impact: Martin outlines two potential strategies—actively banning bad users or quietly degrading their experience. She argues that temporarily allowing bad users to continue their activities while monitoring them often results in better long-term outcomes for legitimate customers.
Automation and Tools: Effective management of bad users can be achieved through automation, harnessing the capabilities of Ruby on Rails and AWS technologies. Martin shares her team’s approach to monitoring and flagging bad users via a CRM system, using background processing with Sidekiq, and sending alerts through Slack for efficient communication.
Utilizing AWS Features: She discusses the importance of using features like Amazon’s Web Application Firewall (WAF) to block known malicious traffic and prevent abuse. Through the application of WAF, her organization has automated the process of identifying and restricting access from problematic referral sites.
Incentivizing Legitimate Users: Martin highlights the significance of understanding user experiences and preferences, recommending that businesses incentivize legitimate users, such as offering exclusive benefits for early logins to encourage account creation.

In conclusion, Martin urges Ruby on Rails developers to leverage available tools and automation to proactively defend against bad users, ensuring a better experience for their genuine patrons and maintaining the integrity of their online applications. The Q&A session following the talk allows attendees to engage further with her insights, where she encourages dialogue about user engagement and protection strategies.

00:00:12.010 The next speaker unfortunately couldn't be here with us, so we arranged her talk over video. It's going to last roughly twenty minutes, and there will be time for Q&A. Bust is going to be here on stage, translating your questions to Brittany over the interwebs.

00:00:22.249 Brittany works as the lead web developer for the Pittsburgh Cultural Trust, a nonprofit organization. She is the host of the 5x5 Ruby on Rails podcast, and her alter ego, Norma Skates, plays and referees roller derby. Although she couldn't be with us in person, we are glad she was able to join us virtually. Let’s give her a round of applause as we enjoy her talk.

00:00:58.940 Today, Brittany's talk is titled 'Rails Against the Machine.' She will tell us how we can use Rails and AWS to manage bad actors and protect our online businesses. Enjoy Ruby on Ice! My name is Brittany Martin, and thank you for tuning in as I kick off my talk, 'Rails Against the Machine.'

00:01:23.240 First, I want to thank the organizers for making it possible for me to participate in this conference, even though I had a scheduling conflict. This conference features wonderful speakers, snowy activities, and I imagine a great group of attendees. I am super jealous and hope you’re having an absolutely wonderful time! Fortunately, the organizers have worked with me to host a live Q&A after this talk.

00:01:55.250 Throughout the talk, you'll see my Twitter handle at the bottom footer, @brittjmartin. If you have any questions or comments about the talk, please tweet at me, as I am eagerly reading those tweets back in the States. So, who am I? I'm the lead web developer for the Pittsburgh Cultural Trust, an organization that supports the arts in Pittsburgh. We aim to enhance the cultural and economic development of the Cultural District, a 14-square-block area of downtown Pittsburgh. I work on a Ruby on Rails e-commerce site, which serves as the ticketing platform for ten different arts organizations, including our very own ballet, opera, and symphony.

00:02:40.750 The ideas around this talk are a culmination of all the experiences I've had, which span various roles including working with a hosting provider and supporting a call center. As I mentioned earlier, I also play roller derby at night. I am Norma Skates, a play on Psycho's famous mother, Norma Bates. I play for the Little Steel Derby Girls in Youngstown, Ohio, and the Pittsburgh East Roller Villains in Greensburg, Pennsylvania. I started playing about two years ago, and I didn't know that derby would become one of the most significant and enjoyable things I would do in my life.

00:03:18.200 Between programming and derby, these activities have become pillars of my self-confidence. If you have ever had an interest in roller derby, I highly recommend reaching out to your local league to see how you can get involved. It's a wonderful sport that teaches you about athleticism and teamwork. Lastly, I’m the host of the 5x5 Ruby on Rails podcast, which you can subscribe to through various platforms, including iTunes. I have enjoyed chatting one-on-one with many of my Ruby heroes, including several of the speakers at this very conference. Please talk to me about it, and remember to tweet me if you have feedback or guest suggestions.

00:04:01.400 But enough about me; let’s move on to why I’m appearing before you today. In an ideal world, our customers would visit our websites and apps and happily purchase the products we create for them. However, in reality, that is not always the case. There are villains who visit your website. Who are these villains? They are the people accessing your website with the intent to game the system, to cheat, and to manipulate your business for their benefit.

00:04:54.810 Throughout this talk, I will refer to these users as 'bad users,' because they are indeed using your site, even if you want to ignore their existence. These users cheat, harass, and seek to manipulate you and your business, aiming to claim what they want. For the purposes of this talk, I will focus on bad users who disguise themselves as regular customers, a crucial distinction. I am not addressing BOTS or hackers who attack via security vulnerabilities on your website; rather, I'm focusing on the behavioral bad users. I do want to emphasize that those who engage in theft are generally not doing so to intentionally cause harm; their actions often stem from underlying needs.

00:05:37.920 As an example, imagine if I threw an American football reference into this talk! We're not focusing on a streaker wandering onto a field with the intent to disrupt. Instead, we are concentrating on learning the tactics to deal with players who've been rostered onto the team but commit penalties to gain an advantage. The first step in protecting yourself is identifying which users on your site may be attempting to break the rules. Luckily, we have a variety of avenues to achieve this.

00:06:46.460 Ask questions, review your user personas with your design team, and consult with your customer support team to identify negative patterns. Your customer support and sales teams generally have the most considerable one-on-one interaction with your customers. Lastly, review your application logs to find patterns of malicious users—not necessarily users trying to hack your system, but users trying to exploit coupons or seek excessively favorable deals.

00:07:04.350 So, what do these bad users look like? I'd like to present some examples of bad users I've encountered in the wild. One main type of bad user I deal with daily is ticket resellers. They buy tickets in large quantities and resell them for profit. Many ticket resellers use URLs similar to legitimate box office websites and often spend significant amounts on Google AdWords to promote their sites above legitimate ones. They imply that they are the official ticket source without clearly stating the real prices of the tickets they're selling compared to what they paid for them.

00:08:06.650 Another example is serial returners, who might switch the UPC code on a $600 product for a lower-cost code that rings up at $50. They buy the expensive item, replace the fake UPC tag with the original higher-priced code, and return the item without a receipt to receive store credit for $600, which they can later sell online. Additionally, I want to address upvote rings, a practice where users form rings to upvote items on community platforms like Hacker News, avoiding the need to pay for advertising to promote their content.

00:08:56.240 What are the consequences of these bad users? When they come between you and your ideal customers, it can lead to lost profits, damaged relationships, and inflated prices you need to charge. It is up to your company to decide on a strategy: you can either openly kick the bad customers off your site or quietly degrade their experience. In my experience, I find that quietly degrading the experience is more effective. Word spreads quickly among bad users; if you establish strict rules, they will work harder to circumvent them.

00:09:53.480 In this talk, I will show you the tools at your disposal to execute the second strategy. Regarding identifying and executing a strategy against bad users, I recommend automating the process as much as possible to reduce resource demands and maintain fairness in identifying bad users. As developers, we are fortunate to have a wealth of tools ready for use, particularly in the Ruby on Rails community, which is why we are all here today.

00:10:40.960 Once you have identified bad users, create a model that categorizes these users throughout their journey. For instance, our model is called 'reseller.' We’ve determined that users purchasing tickets within our tri-state area are considered safe. When assessing whether a user is permitted to print their tickets at home, we perform a check on their billing address against a list of safe states.

00:11:09.780 We share a CRM between our box office, call center, and website, which is critical for ensuring we have accurate customer records. This allows us to track when user interactions suggest a potential reseller. When we identify a user as a reseller, we mark them in our CRM, which we use as the definitive source for making future decisions about that user.

00:12:01.250 As users navigate the purchasing process, we check their session to determine if they are a reseller. We conduct these checks as they perform typical actions on our site, such as logging in and creating accounts. These checks incur costs since they require calling our CRM API. Therefore, we utilize background processing to manage this workflow, specifically using Sidekiq, a popular library in the Ruby community.

00:12:48.350 Whenever we detect a new ticket reseller logging onto the site, we push a new worker task into the queue. As we chose the quieter degradation strategy, we allow them to continue making purchases for now while we track their activity. When we find a reseller on our site, we don't block them immediately. Instead, we trigger another worker that sends a request to a Slack channel, allowing us to monitor how many resellers are being flagged.

00:13:35.860 In our Slack notifications, we create a JSON string that includes the user’s information, indicated by a skull emoji to denote a reseller. This alert system allows us to monitor reseller activity while keeping the identification details minimal. We can look up their ID in our CRM if further investigation is necessary without sharing personal details in our Slack channel.

00:14:54.280 As we implement these systems, we rely on Rails' cache fetch method to efficiently manage low-level caching. Each user is assigned a unique session when they log in, and we only need to check if the user is a reseller every 30 minutes. To optimize performance, we cache the results of these checks, improving our application's efficiency.

00:15:56.080 We should assess how often we need to gather user information to streamline our processes. The more data we have, the better. Knowing which user is currently accessing your site is essential. I recommend encouraging all users to log in as early as possible. For our site, we offer exclusive discounts to subscribers who log in, which incentivizes early registration.

00:17:04.090 As for anonymous users, they are still considered innocent until proven guilty. They cannot buy from us without logging in. With the future of third-party cookies becoming uncertain due to rising privacy concerns, we use cookies as a secondary set of data. When a known ticket reseller logs into our site and creates a new account, we insert a small cookie into their session. This information allows us to tag that account as belonging to a ticket reseller in our CRM.

00:18:15.520 Once we've identified a bad user on the site, it's time to execute your plan. Collaborate with your team to determine which features of your website users love and which ones might be used to exploit the system. For us, we found that three out of four attendees want to choose their own seats for events, a feature we allow for regular users but remove for ticket resellers.

00:19:23.540 Additionally, we offer print-at-home tickets, which are convenient for legitimate users but not available to ticket resellers. When they attempt to purchase tickets, they are unable to select the print-at-home option, effectively degrading their experience. We have witnessed this system work in practice—during a pre-sale event, the non-legitimate users did not secure any tickets, while genuine customers had access to the seats they desired.

00:20:03.620 As you develop features for your site, consider what the experience is like for both your ideal patrons and any ticket resellers. For instance, I can simulate different user experiences while developing features to understand how they interact with the site. This insight helps in building a more user-centric application.

00:20:57.769 Your hosting provider can also assist you in protecting your application. As a certified AWS developer, I've utilized their tools to enhance our automation process. Of all hosting providers, Amazon boasts the largest market share and offers a comprehensive suite of features for applications like these. One such tool is the Web Application Firewall (WAF), which protects your applications from common exploits that could compromise security or hinder performance.

00:21:22.319 With WAF, you can tailor your traffic rules to allow or block specific requests. In our recent analysis, we identified referral sites where ticket resellers congregate, and we now use WAF to automatically block them from accessing our site, effectively making it seem that tickets are sold out. Additionally, Amazon GuardDuty continuously monitors for malicious activity, such as unusual API calls, and provides another level of insight into potential threats.

00:22:30.230 To wrap up this talk, I want you to remember all the tools you have available. We are fortunate to have a wealth of resources at our disposal within the Ruby and Rails community. Fight tirelessly for your users, as they will certainly appreciate your efforts. Thank you so much for listening to my talk today. You can follow me on Twitter @brittjmartin or on GitHub @Wonderwoman13. I couldn't have done this without the wonderful organizers of Ruby on Ice, for which I am forever grateful.

00:23:26.210 Now, after today's talk, we have arranged a special live Q&A. I will be joining you shortly. Thank you again for joining me.

Ruby on Ice 2019