Team Building
From 0.10 to 5.2. The story of a 13 year old Rails app

Summarized using AI

From 0.10 to 5.2. The story of a 13 year old Rails app

Braulio Carreno • April 03, 2019 • Minneapolis, MN

The video titled "From 0.10 to 5.2. The story of a 13 year old Rails app" by Braulio Carreno presents the evolution of ActBlue, a nonprofit platform that facilitates small-dollar donations for political campaigns and nonprofit organizations. Since its founding in 2004, ActBlue has raised over $3 billion for 30,000 organizations, demonstrating the power of grassroots funding.

In this presentation, Carreno shares important insights from their experience over 13 years with the platform, particularly focusing on how they scaled their monolithic Ruby on Rails application to handle significant traffic spikes and maintain reliability while managing a large codebase.

Key points discussed include:

- Technical Debt Management: Carreno emphasizes treating technical debt as a feature rather than ignoring it, allowing teams to prioritize improvements alongside new features. Clear identification and prioritization are critical to effectively managing technical debt over time.
- Obsolescence and Upgrading: He notes that software obsolescence, such as outdated versions of Rails or gems, can introduce vulnerabilities. The strategy to handle this includes maintaining a solid test suite and carefully following the Rails upgrade guides to avoid skipping important minor versions and to address deprecation warnings.
- Scaling a Growing Application: Carreno discusses challenges faced due to the increasing size of the codebase, emphasizing the significance of well-structured code, good commit messages, and comprehensive documentation in maintaining code clarity, especially as team size grows.
- Handling Traffic Spikes: He illustrates how ActBlue managed high-traffic events, such as during debates or elections, by preparing infrastructure in advance and utilizing asynchronous job processing, caching, and load balancing to ensure contribution processing without significant downtime.
- Choosing the Right Tools: Carreno explains how the team opted to separate complex forms into different applications using frameworks like React to handle JavaScript more efficiently, while the main application handled core functionalities.

The high volume of contributions during significant events illustrates the urgency and importance of robust software design, enabling the infrastructure to handle sudden spikes in traffic. Carreno concludes that leveraging the right frameworks and investing in well-designed architecture is essential for long-term software success and adaptability.

Overall, the session is rich with practical lessons for developers and managers interested in building resilient long-running systems that can evolve with changing requirements and increased operational demand.

From 0.10 to 5.2. The story of a 13 year old Rails app
Braulio Carreno • April 03, 2019 • Minneapolis, MN

RailsConf 2019 - From 0.10 to 5.2. The story of a 13 year old Rails app by Braulio Carreno
_______________________________________________________________________________________________
Cloud 66 - Pain Free Rails Deployments
Cloud 66 for Rails acts like your in-house DevOps team to build, deploy and maintain your Rails applications on any cloud or server.

Get $100 Cloud 66 Free Credits with the code: RailsConf-19
($100 Cloud 66 Free Credits, for the new user only, valid till 31st December 2019)

Link to the website: https://cloud66.com/rails?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
Link to sign up: https://app.cloud66.com/users/sign_in?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
_______________________________________________________________________________________________
Small-dollar donations were critical in the last election. The first version of the software that powers ActBlue, the main player in the space, was written in Rails 0.10.

During 13 years, 30K organizations have used the platform to raise $3 billion.

Some lessons we'll be sharing on this presentation: how to scale a monolith to process 1.5M RPM and 250 payments/sec, how to be productive with a 110K line code base, how to minimize the pain of Ruby/Rails upgrades and technical debt.

Intended for beginner to intermediate developers, managers, and anyone interested in building a lasting system.

RailsConf 2019

00:00:19.260 I would like to start with a question: how many of you have used a platform to make a donation? One, two, three... I don't see very well how many of you tipped. Fewer hands. How many of you tipped $10 or more? I want to see those hands higher. Who has the highest tip?
00:00:40.650 At some point, you tipped $5. You're the winner! I did this at our local Boston Ruby meeting, and someone donated $100. Our app had its birthday after I sent the proposal, which is why I marked the date as the 14th. My name is Braulio, but you can call me Brad if that's easier. I have been with ActBlue for six years. I didn't know ActBlue was an early adopter until I prepared this presentation.
00:01:10.860 I'm originally from Chile, in case you're curious about my accent. For those who don't know, ActBlue provides a platform that empowers people to fundraise for political candidates and campaigns. We all know we need money to win elections, but you shouldn't have to rely on the wealthy. Founded in 2004 as a nonprofit, ActBlue is independent of political organizations like the DCCC.
00:01:40.470 We help nonprofits as well. Today, all Democratic candidates for president are using ActBlue to fundraise. In fact, there is a new rule stating that to qualify for the debates, a candidate can either have 1% in three national polls or can fundraise with 65,000 unique donors.
00:02:06.660 Now, how does the contribution form look? Here’s an example featuring a nonprofit that saves animals facing extinction. Since I donated, my name is on it, and they also have my card number. It's as easy as clicking one of the red buttons, and the donation will be made. If you're a first-time visitor, you'll need to provide more information such as your name, address, and credit card number. But the concept remains the same.
00:02:31.599 ActBlue is not just about processing payments; we also help with compliance. We have a team of lawyers who handle filings with the Federal Election Commission and state and local agencies, ensuring that the person fundraising doesn't have to worry about anything. We provide tools and A/B tests to maximize the conversion rates of the forms. Statistics and reports for their campaigns are also available, and we send notifications when a contribution is made.
00:03:05.620 In the last election cycle, we processed 41 million contributions amounting to $1.6 billion, with an average contribution of $39. We helped 50,000 organizations in the last two years. Looking at the lifetime volume since the beginning, the total contributions have reached billions. It took us 12 years to raise the first billion, 2 years for the second, and just one year for the third. I also noted when I joined ActBlue in this graph, so you can see when things really picked up.
00:04:23.849 Our application is called Indigo. All the logos look the same to me, which is why I had to put the hex code for clarity. This is a piece of archaeology: the first commit in Rails! I want to prove this statement; at the bottom is the version of Rails 0.10 from 2005. You need to realize that 14 years is a long time. I bet some of you were programming in Java or .NET, and some of you might not have been programming at all. I see some young people, maybe you were in high school.
00:05:16.820 The author of this commit is Matt DeBergalis, one of the founders along with Benjamin Thomas. So, why did Matt make the first commit? We create software to address business needs. In our case, we're a nonprofit, so we refer to ourselves as an organization. However, in most cases, it will be a business. But the business landscape is not static; it constantly changes, and we must adapt.
00:06:14.270 The challenge with long-running software is that it becomes increasingly difficult to adapt due to the growing complexity and larger codebase. Therefore, our goal in this presentation is to explore how we can navigate these changes efficiently and with low costs while maintaining reliability, ensuring donations keep coming, and ensuring the security of sensitive information, like credit card data.
00:06:47.700 I have two examples related to this. Recently, we had to implement Apple Pay. We were able to do this quickly because we leveraged existing code we had for processing credit card transactions. While the two are different, the existing code was good enough for adaptation. This allowed us to keep up with our improvement schedule.
00:07:10.949 Another example is preparing for upcoming elections. We know that traffic will surge, so we need to manage it effectively. The challenges associated with long-running applications can be categorized into two groups: issues that arise from aging code and issues that arise from an increasing codebase. These challenges are not unique to Rails; they are common across all software projects.
00:08:01.360 I want to share some lessons learned from dealing with these challenges, specifically regarding technical debt. Technical debt occurs when we choose an easier solution instead of a more effective approach that would take longer to implement. If this becomes a pattern in a long-running project, it results in a significant amount of technical debt, which can create major problems.
00:08:49.930 Our approach is to treat technical debt as a feature. However, this doesn't mean that we ignore the ugly parts of the code. Addressing technical debt involves making a case for its resolution, explaining the benefits such as improved maintainability and ability to adapt to changes. This requires discussions with the rest of the organization since everyone has their priorities.
00:09:19.830 We establish both short-term and long-term plans for addressing technical debt during quieter periods, like December after elections or at the end of a quarter. I have two small stories that highlight our experience with this. One involved implementing Apple Pay, where I saw code that had complicated structures, and yet we haven’t modified it in six years because there simply hasn't been a need. The complexity of that code hasn't justified the effort.
00:10:14.520 In another case, we introduced a feature for recurring contributions based on a suggestion. Initially, we doubted its usefulness, thinking nobody would utilize it. We implemented it anyway, and it turned out to be highly popular. Later, when we added a weekly contribution option, we were skeptical again but were once again proven wrong. Thus, today, the recurring contributions feature plays a crucial role, processing about 30,000 contributions every day.
00:11:12.400 In summary, addressing technical debt should be treated as a feature rather than a problem. The distinction between technical debt being a feature was not always clear, but we’ve learned through experience.
00:11:51.600 Next, let's address obsolescence, which is not just about aging code but also includes outdated versions of gems and frameworks that are no longer maintained. The challenge is that vulnerabilities are discovered frequently, and if you don't apply patches for years, you risk exposing the application to serious issues. I understand that some might still be running Rails 3.2, and it’s crucial to stay updated.
00:12:46.200 To mitigate this, you need a robust test suite. It's imperative to have good tests when considering upgrades; skipping minor version updates can lead to major complications. The Rails guides offer excellent information, so it's essential to read them, particularly regarding version upgrades.
00:13:54.700 The Rails guides provide clear instructions for upgrading from one version to another. Engage with the release notes, as they contain critical information that might directly affect your application. Once the tests pass, it's equally important to fix any deprecation warnings, since they might not always appear on standard output.
00:14:49.780 If a new feature has been introduced into a release that resembles functionality from a gem, it is advisable to drop the gem and use the native feature. Transitioning existing features before the actual upgrade is beneficial since it minimizes issues during the upgrade process.
00:15:40.880 Be proactive in forking and fixing any gems that are no longer maintained, contributing those fixes back to the open-source community. In our experience with Ruby and Rails, upgrading the gems has presented more challenges than upgrading the frameworks, with the notable exception of transitioning from Ruby 1.8.7 to 1.9.
00:16:27.850 There are workshops dedicated to upgrading Rails; I recommend attending if you haven't done an upgrade yet. Over the years, our codebase has grown significantly, currently boasting 35,000 commits, 110,000 lines of source code, and a range of classes and modules.
00:17:11.420 Working on legacy code can be intimidating. I always say that in software development, you learn to write before you learn to read. This mentality can lead to people gravitating towards new applications rather than legacy ones. When I joined ActBlue, I encountered an application that was old but not a mess.
00:18:09.560 To maintain a healthy code base, good commit messages are vital. There are many resources online dedicated to this. I recommend finding and following best practices for writing commit messages to ensure clarity and inform future developers.
00:19:18.620 While having no comments in code can be damaging, excessive comments can also lead to confusion. It’s helpful to document complex algorithms or logic to make it easier for others who will come after you. Additionally, maintaining documentation for the data model can assist developers in understanding the project's logic.
00:20:04.020 As our team grew from just two founders to 120 people, including 34 in technical roles, communication became a significant challenge. As the team expanded, effective coordination became crucial.
00:20:59.140 To facilitate smoother communication, hiring top developers is key. While they may be more expensive, higher quality talent significantly enhances productivity. Ensuring developers feel empowered through autonomy and purpose also fosters a positive culture that encourages retention.
00:21:57.850 We conduct code reviews for all pull requests, ensuring at least one person reviews it, and for more complex changes, at least two developers. Discussions during the pull request process can lead to deeper clarity.
00:22:32.170 Regarding our code reviews, even senior developers are not exempt from having mentors, ensuring knowledge transfer within the team. Let me illustrate a point about managing complexity. I shared an example from 2015 regarding a contribution form for a nonprofit.
00:23:44.420 After a user completed a contribution form, if they abandoned the browser, the contribution would not be saved. This led to the realization that we needed to improve the form's functionality.
00:24:19.670 We decided to structure a new application to handle these processes more effectively. This new 'Form App,' built with React, allowed us to optimize the user experience significantly.
00:25:45.510 Building the Form App not only enhanced functionality but allowed us to minimize the problematic hand code in our JavaScript, resulting in more maintainable code.
00:26:40.890 In recent years, we've experienced surges in traffic during high-stakes political moments. Let me illustrate this with some examples.
00:27:26.830 The first spike was during a session in Congress in 2009 when President Obama spoke about the Affordable Care Act. The second spike occurred later that same day due to news coverage, while another large spike followed the next day.
00:28:35.900 In 2016, another event occurred when Bernie Sanders delivered a victory speech and urged supporters to contribute online. At one point, we registered over 2,650 contributions per minute.
00:29:10.520 Each surge represents valuable lessons regarding how we handle spikes in traffic. Analyzing our responsiveness to load revealed processes that coped efficiently and others that didn’t.
00:30:31.720 These spikes serve as timely reminders of the importance of preparation during key moments of increased activity to ensure that we reliably serve our contributors.
00:31:49.170 Scalability can often hinge on architectural decisions, like how many servers we have or how powerful they are. Ensuring the code can operate in parallel is essential to successfully managing a load increase.
00:32:22.880 Emphasizing asynchronous job processing, we queue slow actions, like sending emails, rather than involving the web server directly. Utilizing caching along with a Content Delivery Network (CDN) has proven beneficial as it ensures requests from different regions are handled efficiently.
00:33:09.020 Choosing the right framework and language greatly simplifies the management of legacy software. We are thankful for Ruby on Rails and the supportive community behind it.
00:34:00.330 Remember to read the commit message guidelines. They have significant value!
00:34:24.490 We have two minutes for questions. Is there anyone?
00:34:40.610 Okay, so the question is why we decided to use React instead of Angular. I wish I could answer that, but it wasn't my call. The backend is Indigo, our monolith handling contributions and processing. The main thing with scaling is preparing for momentous traffic spikes when you have prior knowledge
Explore all talks recorded at RailsConf 2019
+102