Sweat the Small Stuff

by Aaron Harpole

In the talk titled "Sweat the Small Stuff" presented at RubyConf 2018, Aaron Harpole explores the challenges and changes team leaders face when their companies undergo rapid growth. He emphasizes that while excitement accompanies a growing team, the transition from small to large can diminish productivity and complicate workflows. Harpole identifies key characteristics of small teams, such as their ability to ship features quickly, maintain shared understanding, and operate with fewer overheads. He argues that scaling teams must focus on improving productivity rather than merely hiring more staff. Harpole suggests delaying new hires and instead investing in existing team members' efficiency, recommending solutions such as acquiring faster hardware and optimizing Continuous Integration (CI) setups.

He discusses the importance of crafting a thoughtful recruiting process, emphasizing the need for leaders to take an active role in attracting talent, building genuine relationships with candidates, and communicating the company's unique value proposition during interviews.

As teams expand, he notes, the complexities of processes, documentation, and meeting culture increase. Leading larger teams requires an adjustment in management styles, emphasizing the necessity of onboarding processes and the importance of creating a positive meeting culture. Harpole illustrates the concepts by discussing the potential pitfalls of organizational metrics-over-ethics, citing real-world examples and emphasizing a leadership approach that supports human growth and team dynamics. Ultimately, he concludes that successful leadership involves embracing both technology and the human elements that underpin effective team collaboration.

00:00:15.560 My name is Aaron, and I'm going to cover a lot of material in the next 40 minutes or so. I've actually posted notes and some other materials on my website that you can check out. It includes some overflow material that I couldn't fit into this talk, so it's pretty good. If you want to connect with me online, you can find me on Twitter, and I'm also active on Mastodon more recently, as I’m kind of a fan of decentralizing the web a little bit more.

00:00:28.019 My website is at the bottom of the slide. I work here in LA at a company called Carbon Five. If you've never heard of us, we're a digital consultancy that helps companies, both small and large, create digital products. We have offices in LA, San Francisco, Seattle, Chattanooga, and New York. So if your company wants to work with us, or if you would like to work with us as an employee, please come talk to me. All right, let's get started.

00:00:52.620 It feels really exciting to work at a company that's growing; there’s a certain energy that things have when there are tons of new hires every week. It seems like there's always some new milestone that’s been hit, but between the celebrations, you're starting to notice that things feel a little different. When you're growing, you might have days or even weeks where it feels like your team didn't get anything done. You have resources that you once only dreamed of, but you're underperforming compared to your past experiences when you were scrappy.

00:01:18.100 If there's something that makes it hard to get things done in a big team, then it stands to reason that there must be something that makes it easier to do tasks in a small team. So, I'd like to explore some characteristics of working on smaller teams to see if we can learn from that and maybe apply these insights to working in larger groups.

00:01:36.350 When you're in a small team, it feels like you get more done. You might not be doing more work in a small team, but the output feels higher. In a small team, you are personally responsible for shipping major features of your product, and because there aren’t as many people writing code, you have less code overall. You're probably also more familiar with a larger portion of that codebase.

00:01:54.800 When you have a small codebase in a small team, everyone tends to have a better shared understanding of how it works. This makes for quick and productive technical discussions. Everybody on the team is a jack-of-all-trades; your team isn't big enough yet to have specialists. There’s often no marketing department or even a marketing person, but everyone picks up the slack to help out with marketing tasks. You don't need a PR person, but when you ship something new and exciting, you'll excitedly share it with the tech blogs.

00:02:11.850 If you're browsing Reddit late one night, as you sometimes do, and someone complains about your company because they encountered a bug, you're going to directly message that person to find out more about the issue. The next day, you'll come in and fix that bug, probably pushing it within the same day. It feels incredibly rewarding to be empowered to spot a problem and take care of it yourself.

00:02:30.390 In a small team, you don't have meetings; you have conversations instead. Your code architecture is simple. Your deployment procedure might just be as straightforward as a git push to Heroku Master. If something goes wrong, it's just as easy to roll it back or make a quick fix and push it out.

00:02:41.900 Deployments are cheap and risk-free, so you do them frequently. When someone writes a line of code, it's not long before it’s live in production, and a customer is using that code. This allows you to learn very quickly from what you've shipped, creating a tight feedback loop that is incredibly healthy for your development process.

00:02:55.450 No one is writing documentation, and that's okay; your team itself is your documentation. If you need to figure out something you’re unsure about, you can just ask someone nearby. Writing docs isn’t worth the trouble at this stage. When faced with a generic problem that needs solving, you tend to pick an off-the-shelf solution and run with it.

00:03:09.960 It's kind of shocking that tools like continuous integration (CI) and even source control used to be considered luxuries that only the biggest companies could afford. Now, they are commodities. With all these great tools, you don’t have to write much that isn't pertinent to your company's specific problem, and this tends to bring a lot of business value.

00:03:27.510 There’s no dedicated customer support team, so everyone pitches in. You know what your customers are struggling with, and you're incentivized to fix those issues. As a result, you no longer get interrupted by customers calling for help, and they notice how quickly their problems are resolved, which makes them feel valued. It’s a virtuous cycle, and a small team can accomplish a lot.

00:03:47.010 Being part of a small team gives you significant structural advantages, but if you want to maintain that small-team dynamic, you must be selective about what you commit to doing. This can be challenging because it’s tough to know you can do something better yet still say no, recognizing the need to focus on your core features. Small teams have less overhead per person, so your productivity per capita is at its peak.

00:04:08.560 Every extra hire you bring on is one more person you need to keep in the loop. With each additional team member, it becomes increasingly difficult to preserve those small-team dynamics. Therefore, I recommend delaying additional hires as much as possible. I, too, am a natural procrastinator in that regard.

00:04:25.600 Instead of attempting to grow your team, invest in your team's productivity first. When you make engineers more productive, the returns are significant, as they can spend more time creating value for customers. So look for inefficiencies or opportunities to make processes work a little better— even if it’s just a small improvement, because every minute saved adds up.

00:04:44.370 Consider getting more CI servers so that you're not waiting as long for builds. Pay for faster internet. If there’s an engineer who has been asking for a faster computer, buy it for them, as saving just a few minutes a day can justify the cost. These issues are literally some of the easiest kinds of problems you can solve; you can just throw money at them, and they will resolve quickly.

00:05:02.280 Address these issues right away and don’t try to compensate for being a small team by working longer hours. I know that occasionally you’ll need to hit deadlines, and you may have late nights, but if that becomes a habit, it’s going to hurt your productivity. I've searched extensively and have never found a study on knowledge worker productivity that suggests any long-term gains from working more than 40 hours a week. Even if you could get more work done by extending hours, that's not a sustainable way to live and work.

00:05:19.490 It's not a good way to treat your team, so I refuse to subscribe to a culture of workaholism. I don't want to inadvertently worsen the issue by creating a workplace environment where that is accepted. Eventually, if you maintain these practices, you'll pick all the low-hanging fruit and reach a point where the demands on your team surpass what your current set of individuals can manage.

00:05:36.170 Now it's time to scale the team, and scaling begins with the recruiting process, which can be quite challenging. The typical recruiting process is fundamentally flawed, and if you’ve ever received an email from a recruiter, you know exactly what I’m talking about. The recruiting process is often a terrible experience for engineering candidates, however, if you can recruit better, you can distinguish yourself from other companies.

00:06:00.820 I'd advise that if you want to excel at one thing as the leader of engineers, let it be building a great team. You can be mediocre at everything else, but if you have an eye for talent and can persuade those talented individuals to join your team, you’ll be all right. So don't delegate the task of reaching out to potential candidates to a recruiter; instead, take ownership of building your team yourself.

00:06:15.030 If you want to do it well, you need to be actively involved because a recruiter can't convey the opportunity in the same way you can. This doesn’t mean recruiters are not valuable or that you should fire them; they can be useful for identifying candidates and assisting with negotiations and other elements of the hiring process. But when it comes to initially contacting engineers and fostering relationships, that’s your responsibility.

00:06:35.620 When you look at your inbox, you might see a generic-looking form letter from a recruiter sitting next to a personally crafted email from an engineering manager who is genuinely interested in building their team. Which one are you more likely to respond to, even if you're not currently looking for a job? I suggest that you meet candidates for coffee, learn about their career ambitions, and find out what projects excite them.

00:06:52.520 Discover which programming languages they're passionate about. A recruiter can do a good job at building a network of connections, but when you meet engineers face to face, you can build meaningful relationships. If you can reach a point where engineers are eager to leave their current roles to work with you, you're setting yourself up for long-term success.

00:07:10.270 Additionally, it’s a good time to think about how you interview people. During periods of rapid growth, every hire you make is significant. Each person you hire holds incremental importance, yet every candidate is selecting a company to work for that they'll be committed to for a while. Hence, it’s crucial to keep that perspective during interviews.

00:07:27.350 Help the candidate visualize themselves working at your company. When you invite them for an on-site interview, ensure everyone has lunch together to foster conversation. At Carbon Five, we often have silly discussions about trivial things, like what exactly constitutes a 'salad.' It’s crucial to spend time pairing with candidates on tasks that reflect their day-to-day work.

00:07:42.970 This approach is beneficial for you to assess their fit, and for them to understand what they’re getting themselves into. Highlight the aspects that make working at your company enjoyable, ensuring the candidate has opportunities to experience those aspects during their visit.

00:08:07.300 As your team grows, the things you seek in candidates will evolve along with your team's composition and technical staff. Your technical questions will need to change, but be mindful—the non-technical aspects of a job become significantly more important as you scale. For instance, writing good documentation isn’t crucial in a three-person engineering team, but it becomes key when multiple teams are creating APIs for one another.

00:08:17.610 Gathering requirements from non-technical stakeholders might be an uncommon task when it’s just a few engineers, but with the hiring of non-technical employees who oversee other areas, that skill becomes critical. Additionally, be attentive to personality traits that may indicate a potential bad fit for your company.

00:08:33.260 It's easy to think you can overlook some negative personality traits if someone is a talented programmer because they produce good work. However, in a larger team, you can’t afford that. A significant part of the job is interacting with others, and if someone creates friction in their collaborations, that disqualifies them from the role, regardless of programming skill.

00:08:53.760 Interviewing in itself could warrant an entire talk. It's one of the biggest opportunities for you as a leader to shape company culture. Changing the culture can be quite challenging with an existing team, but if you're looking to shape the future, the interviewing process is your tool.

00:09:13.170 Also, don’t overlook onboarding. I've had those moments when I'm relaxing at home on a Sunday, thinking about the week ahead, only to remember that I need to prepare for a new hire starting the next day. A new hire's first week is critical— it sets the tone for their expectations of your company.

00:09:29.000 If you do onboarding poorly, it could leave a bad impression, and if it's really badly executed, they may leave in that first week for a better offer. However, the upside of onboarding is that it can be standardized fairly well, allowing for multiple repeatable steps for new hires.

00:09:47.150 If you can figure out those steps, I encourage you to document them. Create a template in your to-do list app, and when a candidate accepts an offer, you'll have those reminders ready to ensure you don’t forget anything. Make sure their desk is ready with a computer and all the equipment they requested and have plenty of company swag on hand, ensuring the sizes are appropriate.

00:10:07.550 You're going to introduce the new hire to the team, but also set up a series of one-on-ones. These meetings enable the new hire to pair with different people throughout the company to learn the inner workings. For example, pairing them with a site reliability engineer can give them insights into deployments and production support. Similarly, pairing with a designer can teach them about collaboration between designers and developers.

00:10:24.080 Before you know it, your team will grow substantially. At that point, you will need to adjust your work habits to align with this new scale. I wish I had some fancy tricks to share, but I haven’t found any. You’ll have to focus on the basics, but I do have a few recommendations I believe will help you maintain productivity as a larger team.

00:10:43.700 Let's consider an example. Suppose we establish an Internet of Things (IoT) company that creates Wi-Fi-connected cameras for refrigerators, allowing users to view their fridge contents from afar. We’ll call it Chill Flicks. Obviously, this is a fantastic idea that will attract numerous investors, and soon enough, those cameras will be flying off the shelves.

00:10:56.600 You embark on a hiring spree, and before long, your once-small, close-knit team has expanded significantly. Now, your software stack includes multiple components. First, the devices require firmware that communicates with a back-end API. The cameras will also need video streaming services, mobile applications, and a web application—all of which need a back-end API to support them. You’ll also need an internal management app for your support team to assist them in troubleshooting.

00:11:17.150 As your team grows, a common initial instinct is to assemble separate teams around each codebase. This approach does carry some advantages, like granting ownership over specific code. However, it also assumes that for each feature you intend to ship, you can accomplish it using just one team, which is often not the case.

00:11:38.550 If you have a team maintaining your mobile application and a different team handling the back-end API that powers it, sooner or later, a new feature will require collaboration across both teams, resulting in interdependencies. This can delay your ability to ship products, as communication breaks down when teams have to pass requirements back and forth.

00:12:01.750 One effective alternative to alleviate these issues is to develop cross-functional teams. Members of these teams should be capable of working on multiple codebases. Then you can assign focus areas based on your company's high-level business goals. For instance, Chill Flicks might want to introduce new features for users, such as motion alerts to notify them when someone opens the fridge.

00:12:23.010 Your customers might face challenges with device setup and connecting to their networks, so a team focused on making it easy to install devices could help tackle that entire process—from the mobile app to the firmware and everything in between. At some point, this issue may get resolved, allowing the team to disband and redirect their efforts elsewhere, but for now, it’s significant enough to justify having a dedicated team.

00:12:44.060 Finally, you might have another team that collaborates closely with customer support, addressing issues customers encounter. When customer support struggles to resolve a problem, this team can provide assistance. Because they’re working directly with the support staff, this team is also naturally inclined to add new features to the internal support application when required.

00:13:03.860 By establishing these teams, each can maintain an independent backlog of work focused on their particular challenges, and each team is empowered to solve these problems in whatever way they find most effective. Essentially, we’ve created miniature startups within the main startup structure—each with its distinct mission because we centered them around specific business goals.

00:13:23.520 As your organization gets larger, it's natural to discuss the importance of meetings. With more teams, you will encounter an increase in meetings. Therefore, it's crucial to reflect on how your meeting culture functions. Meetings often receive a lot of criticism, and I've spent time considering the various aspects that people dislike about them. I've narrowed it down to two main reasons for the disdain toward meetings.

00:13:39.280 First, people resent being pulled away from their ongoing work. You may have been absorbed in solving a complex issue, right on the verge of a breakthrough, only to be ushered into a meeting scheduled by someone else. The second reason meetings are disliked is that they usually don't yield sufficient value considering the time invested. If you were given four uninterrupted hours to work, you might accomplish great things, but one hour with four attendees may not yield the same productivity.

00:13:54.390 One company I've looked at with a healthy meeting culture is Amazon. Although I've never worked there, I’ve read significantly about their meeting practices and confirmed their methodologies with people I know who work there. Their rule of thumb is that meetings shouldn't be too large; the size should allow everyone to be fed with two pizzas.

00:14:06.800 Also, Amazon doesn’t use slides for presentations during meetings. Instead, someone prepares a structured memo—often around six pages long—where no bullet points are included, only cohesive text. Considerable effort goes into this document; it’s not prepared the hour before the meeting but is revised repeatedly, receiving feedback from coworkers over a span of days.

00:14:23.290 For the first 30 minutes of the meeting, everyone reads the memo in silence, which I think is an innovative way to conduct a meeting. It reflects Amazon's culture, stemming from their origins as a bookstore, emphasizing the value of reading. This requirement for a detailed document ensures meetings are purpose-driven and well thought-out, requiring significant foresight before scheduling.

00:14:39.720 I’ve held numerous meetings where attendees arrive unprepared, and I end up explaining information that could have been reviewed prior. Rather than getting frustrated with attendees, we can adapt the meeting format to ensure that all participants are prepared because they required to absorb and engage with the material upfront.

00:14:54.370 I value the idea that everyone reads the materials collectively; it ensures everyone is on the same page, literally and figuratively, when conversations begin. A positive meeting culture fosters a productive work environment overall, making it worth contemplating how your organization can enhance its meeting structure.

00:15:09.920 When your company is small, simpler deployment strategies may suffice—like when you’re just using Heroku. However, as your team grows, you may require more complex solutions. As you start to require multiple deployments, you might find tasks you initially expected to be simple actually involve a multitude of steps.

00:15:35.890 Let’s illustrate this with a scenario at Chill Flicks. Suppose their mobile applications authenticate using a simple static token, which the security team finds insufficient. They suggest switching to OAuth2 for its token rotation capability. Fortunately, Chill Flicks has an OAuth server already, but it’s mounted within the Rails application that also serves the company’s marketing site.

00:15:54.130 This creates a problem as Chill Flicks grows; if mobile apps suddenly rely on OAuth, it could overload the marketing site with requests meant for only marketing purposes. Therefore, you’ll need to extract that Doorkeeper engine and separate it from the marketing application, which isn’t a straightforward task because third parties depend on the existing OAuth system.

00:16:15.700 You must construct a careful plan for this transition, encompassing multiple teams and codebases coordinated to execute a comprehensive migration. Documentation of this plan is essential, serving as a common resource for everyone directly or indirectly involved in this migration.

00:16:30.830 When authoring this document, target a general background suited for a diverse audience, including site reliability engineers, managers, and executives. Focus less on technical jargon and more on providing a clear, coherent understanding of what’s being proposed.

00:16:48.820 The goal is also to detail the steps necessary to achieve the final architecture you desire. When making changes to applications that handle significant traffic, it’s crucial to avoid overwhelming users during transitions. Last-minute drastic changes aren’t feasible; your approach must involve incremental steps to ensure user experience continuity.

00:17:05.050 These incremental deployments will govern the migration processes, ensuring minimal disruption. For instance, the first deployment could be a standalone app, using the existing database for creating tokens, allowing it to verify functionality steadily without affecting the original marketing app.

00:17:23.080 While the migration document should elaborate on crucial steps, it’s not necessary to write down every single detail. Instead, ensure it serves as a foundation for understanding the migration and a starting point for discussions among team members.

00:17:41.390 I find tools like Dropbox Paper useful for such documentation because they support Markdown syntax, providing a clean design that allows for real-time collaboration and feedback. Google Docs is also a viable option, but the tool’s specifics matter less than the practice of creating the documentation itself.

00:17:57.520 The migration I’ve been discussing is based loosely on a real situation from a previous company I worked for. The document was designed to ensure clarity and simplicity; remarkable complexity was not the objective.

00:18:13.080 Interestingly, many individuals on the team were resistant to the proposed architecture upgrade because it involved building a new app for high traffic on Rails—not typically associated with scalability. I initially faced skepticism, but I emphasized that this approach mitigated risk by leveraging familiar tools that our team had effectively utilized before.

00:18:30.240 By breaking down the deployment into manageable components our team was already familiar with, we minimized the risk of failure. In addition, we maintained continuity by using the same Doorkeeper Rails engine, ensuring that the new application behaved consistently with the older version.

00:18:48.850 When executing projects like this, especially with an expanding team, it’s paramount to remember that the process matters as much as the intended outcome. Neglecting the significance of intermediate steps can jeopardize the final product's success.

00:19:06.040 The steps taken to extract the OAuth server may have seemed high-stakes, as you’re creating a production app from scratch. Still, the structured, incremental approach ensures lower anxiety and allows you to learn from any errors in a safe environment. One of the beautiful aspects of programming is that we can test our mistakes in controlled scenarios without real-world repercussions.

00:19:24.220 In contrast, other professions, like surgery, demand a different level of precision under immense pressure. We design our processes; hence, let’s not make our deployment procedures high-stakes surgical operations. Instead, invest effort into making sure everything we do is as stress-free as possible.

00:19:42.740 Google publishes a comprehensive handbook on site reliability engineering that offers valuable insights. Its content, while seemingly mundane, becomes crucial for scaling at large organizations. They stress the importance of sanitizing and validating configurations within applications to avert minor issues turning catastrophic.

00:20:02.060 Something as simple as configuration management can have serious repercussions; for instance, an incorrect DNS entry cost Google for several minutes in 2005. Reading the manual is highly recommended; it cultivates a productive mindset regarding how to build resilient systems.

00:20:20.800 Another critical concept they discuss is the idea of an error budget. This involves determining guaranteed uptime for an application, then calculating the allowable errors. For instance, if your application guarantees 99.99% uptime, this translates to a margin of error of roughly 5,000 requests every month. Thus, you could test new deployments as long as the overall error rate remains within the budget.

00:20:39.500 As you scale into larger teams and hire Site Reliability Engineers (SREs), you'll encounter tension between their focus on stability and software engineers' drive to ship new features. The error budget becomes a valuable tool for mediating those differences. Instead of relying on feelings about what feels right, you can lend weight to decisions through data.

00:20:58.640 In scenarios where problems arise, it’s essential to locate and address the root cause. One effective approach to behavior during outages is utilizing blameless post-mortems. A few years back, Amazon experienced an S3 outage that stunned many users. After reviewing the post-mortem, they identified an engineer making a typographical error while deploying servers as the initial cause.

00:21:17.160 Revisiting the mistake doesn't yield constructive responses if we simply blame the individual for a typo. Instead, the real underlying cause is the possibility of bringing down S3 services with that oversight. Recognizing that foundation allows for improvement, leading to measures that improve resilience and prevent similar breakdowns.

00:21:35.850 Validation of inputs for configurations can significantly impact the overall reliability of a system. While we aren’t surgeons, our actions can still affect large systems. Promoting a culture of blameless post-mortems fosters an environment where individuals can freely learn from mistakes rather than fearing punitive measures.

00:21:53.720 A team comfortable with making mistakes in a psychologically safe environment will be more likely to adopt new practices and pursue innovation. In contrast, a culture where team members are apprehensive of making errors may stifle creativity and progress, leading to missed opportunities in shipping new products and features.

00:22:11.550 Over the past 30-odd minutes, I’ve shared insights that may seem mundane; leading a growing team can be frustrating when you’re confronted with simple tasks that become increasingly complex. It’s important to remember that challenges arise when there are significant numbers of people involved in systems. Don’t be too hard on yourself.

00:22:28.720 If you’re accustomed to a small team and suddenly find yourself navigating a hundred-person team, understand that it will take time to adapt to that change. The team’s structure has altered, and you must learn how to coexist and collaborate with this new team dynamic.

00:22:47.480 It is worthwhile to think systemically regarding how your team functions. When I mentioned earlier that small teams can easily leverage tools like Heroku, well, you might find that those deployments seem seamless when there are fewer people involved. Deployments are practically free when managed appropriately.

00:23:06.530 If deploying is simple and cost-effective, that creates an economic incentive to do it more frequently, resulting in cultural changes regarding shipping. If something goes wrong and people tend to blame others, don’t be surprised when individuals aren't forthcoming with critical information.

00:23:23.000 You have to understand when your organization becomes numbers-focused. If the focus shifts too much to metrics at the expense of building a great product, you may end up with severe failures. Consider the consequences that companies like Volkswagen experienced when they prioritized metrics over ethical standards, leading to significant losses.

00:23:57.300 You have to accept that humans are fallible. Embracing your team as a system made up of fallible components is essential. You can’t scale operations without enhancing your tolerance for error—this applies both to technology and to human resources within your organization.

00:24:15.060 Years ago, I came across a fascinating segment in the documentary Objectified, where Apple’s Industrial Design Lead, Jony Ive, mentioned that industrial designers at Apple usually allocate significant time to creating manufacturing fixtures and jigs—far more than actually designing the products themselves. This perspective was enlightening, as I previously thought they’d outgrow such details.

00:24:34.750 Success at scale often derives from addressing minor complexities rather than relying solely on major innovations. My discussion has consistently highlighted the more mundane aspects of daily operations, such as prioritizing effective meeting culture, encouraging personal investor relationships, and developing thoughtful onboarding processes.

00:24:52.800 Remember, company success rarely hinges solely on technology alone. Leadership is about much more than the core technology you employ. If you find these aspects tedious and prefer focusing on tech, be aware that managing a smaller team allows for this. However, if you're intrigued by the challenge of leading larger teams and navigating human complexities, I hope you consider taking on that role.

00:25:12.780 As an industry, we spent considerable time believing that technology held the key to solving every problem, yet we’re witnessing the consequences of that belief. Now, as we shift towards more human-centric approaches, we need to keep perspectives that see effective leadership fostering environments where humans can thrive.

00:25:34.370 Thank you very much for your attention today, and I hope you find a path forward within your teams that embraces not only technology but also the vital human elements of successful leadership.