Observability

More Engineers, More Problems: Solutions for Big Teams

More Engineers, More Problems: Solutions for Big Teams

by Noel Rappin, David Trejo, Brian Lesperance, and Chris Dwan

The video session titled "More Engineers, More Problems: Solutions for Big Teams" features speakers from Chime discussing the challenges and solutions associated with scaling engineering teams in a rapidly growing environment.

Key Points Discussed:
- Team Growth and Communication: The presentation starts by highlighting the exponential increase in potential communication paths as a team grows. With just three people, there are three potential communication lines; however, with 14 people, that jumps to 91 lines. This underscores the reality of having to manage increased communication as teams expand, particularly in fast-growing organizations like Chime, which tripled its engineering team in 18 months.
- Overview of Chime: Chime is presented as a financial technology company committed to making banking services helpful, easy, and affordable. With about 600 engineers, the team works mainly with Ruby and React technologies, focusing on a macro services architecture.
- Proactive Security Culture: David Trejo, a security engineer at Chime, shares insights into scaling their security practices. A Rails app was built to provide security grades for repositories, empowering engineers to actively manage security vulnerabilities and improve the overall security posture of the organization. Trejo emphasizes the importance of having the right tools and understanding of best practices to safeguard customer data.
- Observability and Multi-faceted Systems: Brian Lesperance discusses the significance of observability in complex systems, detailing how it prevents teams from reinventing the wheel and aids in maintaining efficient operations within a large engineering team.
- Onboarding New Developers: Chris Dwan will address onboarding processes that facilitate new engineers’ acclimatization to Chime's culture, specifically emphasizing their Ruby culture. This aims to ensure that new hires seamlessly integrate, fostering a collaborative and supportive work environment.

Conclusion: The session encapsulates how Chime effectively manages the challenges posed by rapid growth in their engineering team. The discussion presents onboard strategies and tools necessary for maintaining efficient communication, securing customer data, and fostering a strong team culture—a vital aspect for any expanding organization.

00:00:14.059 Hello everyone! This is the Chime sponsor session titled "More Engineers, More Problems: Solutions for Big Teams." Thank you for spending part of your conference time with us at Chime. My name is Noel Rappin, and I am a staff engineer at Chime. My team’s name is literally "I Heart Ruby," and I'm responsible for some of the Ruby practice and the internal Ruby community building, as well as other infrastructure within Chime.
00:00:30.599 We will hear from three Chime engineers today. I’d like to acknowledge all of the engineers in the room, so if you are a Chime engineer, could you please stand up? We have many more Chime engineers here today wearing Chime shirts. If you have questions about Chime or Chime engineering and see any of us in the room, please feel free to approach us.
00:01:09.840 I would like to start with what I believe to be one of the most terrifying graphs in software engineering. This isn't a latency graph, a mean time to failure graph, or even a list of errors. Instead, it's a very abstract network graph. For our purposes, you can think of the vertices as people or systems. This graph shows that the potential communication paths increase super-linearly as your team grows. For example, with just three people, there are three potential lines of communication. But by the time you reach 14 people, that jumps to 91 potential communication lines. This illustrates a truth that you’ve likely all experienced: as teams grow—especially quickly—the amount of communication within your organization increases, and it often feels like you're perpetually playing catch-up.
00:02:29.220 Speaking of rapid growth, Chime engineering more than tripled in size within 18 months, from November 2020 to April 2022. This surge brought its own set of challenges, which we will discuss today. First, let me provide a brief overview of Chime and Chime engineering. Chime is a financial technology company founded on the premise that basic banking services should be helpful, easy, and free. It has been gratifying to see attendees at this conference, including Chime members and staff, visit our booth and share how much they appreciate and enjoy our services.
00:05:10.380 Chime members receive a transaction debit card that allows them early access to their paychecks. They have accounts with no monthly fees and fee-free overdrafts up to $200. Additionally, we provide a secured credit card that helps users build credit without leading them into debt. Our goal is to help our members achieve financial peace of mind through simple, low-cost, and human-centered financial products. It is very important to us that we profit when our members succeed—not at their expense. Chime is a mission-driven organization in that respect.
00:06:49.440 Chime engineering consists of about 600 engineers, with our main offices located in San Francisco, Chicago, and Vancouver. We also have several full-time remote employees, including myself. Our backend primarily uses Ruby, supplemented with a bit of Go, while our frontend is built with React and React Native. While I wouldn't call it a microservices architecture, we operate in a macro services architecture with many internal services. Most backend services communicate through APIs. For more information about our career openings, you can visit careers.chime.com.
00:08:09.840 Chime is truly a wonderful place to work. We pride ourselves on being a great workplace, and I encourage you to follow up with us. Now, I want to introduce three of my colleagues who will discuss some of the challenges associated with working in a large engineering team. First, we have David Trejo, who will discuss how Chime fosters a proactive security and engineering culture in the face of rapid growth.
00:08:29.640 Next, we have Brian Lesperance, who will speak about observability and its importance within complex, multi-faceted systems, as well as how Active Support can prevent teams from reinventing the wheel. Lastly, Chris Dwan will share insights on our onboarding processes for new developers and how we help them acclimate to Chime's culture, particularly our Ruby culture. Thank you all for being here, and I’ll now turn it over to David.
00:14:59.880 Good morning, everyone. I’m David Trejo, a security engineer on the security engineering team at Chime. I've been with Chime for nearly a year and a half, and I’ve been thoroughly enjoying my time here. Today, I will discuss a Rails app we’ve been building to help us scale our security processes across all our production services.
00:15:05.700 As a growing security team, we've encountered numerous challenges. One common scenario is receiving a message indicating that your service's security score is not where it should be. When you see a message like this, it can be quite emotional. You might wonder if you’re in trouble or question why nobody warned you about not doing something. It’s important to improve our security posture as we continue to expand and add new services to our offering.
00:27:57.600 The risks are significant because people trust us with their banking. Losing any customer data would be disastrous. As we look to improve our security processes, we also need to minimize the complexity caused by the numerous tools we've implemented. It can quickly become overwhelming for engineers trying to improve the security of their production services. Compliance checks can also be burdensome without an effective system in place. To address these concerns, we built an internal Rails app that provides security grades for our repositories. This system has allowed us to track security score factors and reduce the workload on our compliance team while empowering engineers to improve their security scores effectively.
00:29:09.560 In summary, we want to ensure our engineers understand best practices, are equipped with the right tools, and are able to address any vulnerabilities quickly. Our efforts have led to measurable improvements in the security grade of our repositories, ensuring that customer data remains safeguarded. I hope this overview helps illustrate the steps we’re taking to create a stronger security culture at Chime.
00:29:18.560 Next, I will turn the presentation over to Brian Lesperance. Thank you.