Refactoring Your Team For Fun and Profit

by Aaron Harpole

In his talk titled "Refactoring Your Team For Fun and Profit," Aaron Harpole, the Director of Happiness at Fullscreen, discusses the challenges and solutions surrounding the growth of their engineering team as they scaled from a startup to a significant player in the media industry. Fullscreen, a company primarily known for its vast YouTube network, experienced rapid growth, leading to operational challenges within their engineering team. During the talk, Harpole highlights the following key points:

Company Background: Fullscreen is one of the largest YouTube networks, with over 30,000 channels and 300 million subscribers, having grown significantly since its inception three years prior to the talk.
Growth Challenges: The rapid increase in staff size pressured their existing systems, particularly their product dashboard, leading to operational inefficiencies and general errors.
Dashboard Inefficacy: The dashboard became overloaded with unrelated tasks, prompting the need for a comprehensive rewrite. Harpole describes it as a 'Yamaha app' due to its multiple roles.
Need for Refactoring: After discussions about their issues and the mistakes made, including the use of MongoDB for inappropriate tasks, the team determined that a rewrite of the dashboard was necessary.
Service-Oriented Architecture: He emphasizes the transition to a service-oriented architecture, breaking the dashboard into smaller, more manageable services to streamline operations. However, initial progress was slow due to ongoing demands and a lack of coherent planning.
Implementation Strategies: Harpole shares the team's approach to refactoring, expressing that while they recognized the need for change, actual implementation was hindered by existing duties and a lack of clear direction.
Challenges of Scaling: As the engineering team grew, shipping new updates became increasingly difficult, leading to a situation where they needed to expedite changes while managing the pressure from the business side for new features.

In conclusion, Harpole's talk emphasizes the importance of adapting software development practices to team organization and operations, allowing them to improve efficiency despite challenges. The core takeaway is that structured approaches like service-oriented architecture can significantly enhance productivity within growing teams, enabling them to deliver more effectively and with less stress.

00:00:25.560 All right, hey guys! My name is Aaron. I work for Fullscreen, which is based in Southern California. I have a really cool job; I’m the Director of Happiness. That sounds like a really cushy job, and it kind of is, but it's also really challenging too. My job is to optimize the engineering team at Fullscreen, ensuring that everyone is as happy as possible and that Fullscreen is an awesome place to work.

00:00:28.960 If you've never heard of Fullscreen before, you're not alone. We're kind of a quiet company, but we actually have a pretty big presence. We are a media company, primarily on YouTube, and we are actually one of the largest YouTube networks, representing tens of thousands of channels. The last time I checked, we had over 30,000 channels in our network. If you don’t recognize any of these channels and that makes you feel old, you're not alone—I feel really old too! Try going to VidCon sometime; everyone there seems to be around 14.

00:00:52.399 Our network is enormous; we have over 300 million subscribers across all our channels. It blows my mind to think about that. We accrue over 3 billion monthly views every month. However, we were a startup; we started from humble beginnings just like any other startup. This is our CEO, George. I think this photo was taken about three years ago. We just celebrated our third birthday a couple of weeks ago.

00:01:01.680 So, how do you get from this to what we have now, which is almost 200 people? We had to grow using technology, and George saw that immediately. He hired Drew, my roommate and our founding engineer. As we began to grow, the engineering team expanded pretty quickly. This is a headcount chart showing our rapid growth. It's especially notable when you factor in that we're all sharing a single bathroom in the office.

00:01:17.159 We started to feel the pains of growing. I know those are first-world problems to have, but growing is really hard, especially in an engineering team. Our product dashboard looked like this—it served as command central for Fullscreen. It was responsible for a lot of important tasks, such as ensuring that money coming in and going out was correct, making sure everyone got paid correctly, and informing everyone about splits when working with tens of thousands of channels.

00:01:22.320 That’s a lot of work and a lot of contracts to manage. I was looking around at all the different roles that this app had to play, and it's a lot. I call the dashboard the Yamaha app because it does so many different and sometimes completely unrelated tasks. From early on, we were really scrappy. When the business demanded something, we delivered. We added many features to the dashboard very quickly, and that caught up with us.

00:01:35.800 As you might know if you've ever experienced something similar, we started encountering general errors because we were struggling to scale. It began to cause panic. We knew that we had to change something. Our team got together in 2012, which seems like an eternity ago, and we started talking about our issues with the dashboard. We aired our grievances and discussed the mistakes we made using MongoDB as our main data store for things for which it wasn't appropriate.

00:01:50.880 We talked about what we did wrong, what we wanted from the dashboard, and came to the conclusion that we needed to do a big rewrite. We knew the existing solution wasn’t working, but we thought that the rewrite might work for us because we believed we had the right approach. We discussed what we wanted to do, but we didn’t necessarily know how we were going to do it. After that meeting, we felt optimistic—but then nothing happened for several months.

00:02:04.760 This was partly because we were still busy putting out fires constantly. Our team didn’t exactly have a game plan for how to architect this new product, and so we were still working in a somewhat dysfunctional way. The leaders of the engineering team came together again, and we started discussing our approach. We decided on a service-oriented architecture, which is a great methodology. I figured if that’s what Amazon does, they probably know what they’re doing.

00:02:20.280 So, we had an idea of how this was going to work: we would break the dashboard up into a series of small services, each responsible for a specific function. However, we didn’t exactly know how to transition away from the existing setup or move the existing app to these new services. While we knew what we wanted to build, it took time to outline the new architecture.

00:02:36.640 So, we continued supporting our existing applications while also starting to build new ones. A few months went by, and it still didn’t look like we were going to hit the timeline needed for Dashboard 2.0. Bets were being placed with the accounting team on whether we’d meet our deadlines, and we weren’t winning those bets. I was becoming alarmed because our team was larger than ever before, yet we were still shipping things pretty slowly.

00:02:53.680 We needed to implement some changes, and those changes came in the form of both a blessing and a curse. We were receiving increased demands from the business. Instead of just giving a thumbs-up like, "Sure, we can do this,\