RubyConf AU 2014: How Github (No Longer) Works

Talks

Zach Holman

RubyConf AU 2014: How Github (No Longer) Works

by Zach Holman

In his talk at RubyConf AU 2014, Zach Holman discusses the evolution of GitHub as the company transitions from a startup to a larger organization. He highlights the importance of maintaining transparency and communication throughout this growth, emphasizing that many startups lose their openness once they scale. Holman shares insights on several core themes regarding company culture and operational practices as GitHub grows from 30-75 employees to over 240. Key points include:

Startup Culture vs. Established Company: Holman critiques the trend of successful startups becoming less transparent as they grow and urges GitHub to maintain its openness.
Dunbar's Number: This concept is presented to illustrate the limitations of interpersonal relationships within a growing organization and the need to shift towards structured communication.
Operational Changes: Holman outlines how GitHub's deployment processes and communication strategies have evolved, focusing on asynchronous communication and remote work due to a high percentage of remote employees.
Remote Work Adaptations: GitHub's 60% remote workforce necessitates unique communication methods that do not rely heavily on synchronous interactions.
Buddy System and Induction: An emphasis on onboarding and longer-term buddy relationships is discussed to help acclimate new hires to the company culture and processes, contrasting previous methods.
Feedback and Team Dynamics: Holman highlights the importance of regular feedback and maintaining a supportive environment, especially to combat imposter syndrome among remote workers.
Team-focused Shipping: The strategy for rolling out new features has shifted from individual staff shipping to team-focused shipping to improve focus and collaboration.
Technology and Product Focus: The talk concludes with a focus on using tried-and-true technologies, resisting the temptation to adopt trendy solutions for the sake of modernity and emphasizing that product design should be prioritized over technology choices.

Overall, Holman's insights serve as a reflection on the changes GitHub underwent amidst rapid growth, and he underlines the significance of retaining core values and communication methods as the company scales.

00:00:12.160 Point of order: I do have strep throat, so that gets you back for everything you just said.

00:00:18.000 Secondly, I wanted to say that I was supposed to go on yesterday, and it sounded like Darcy and Felix both suggested they switch their spots from today so I didn't have to speak yesterday, which was awesome for me.

00:00:25.039 Now, if you haven't given any talks before, you may not realize how big of a deal that was for them, because 85 percent of talks are written about an hour before you go on stage.

00:00:35.680 So for them to volunteer to go up a day before is awesome. I thank you both and the organizers for not getting mad at me for suggesting I maybe not go yesterday.

00:01:00.239 Also, my mom is here, and she's amazing because even after all these years, she's still trying to figure out what GitHub actually is. Hopefully, this talk will shed some light on that.

00:01:06.799 I want to talk about building a startup. These are the rules, the recipe to success: to build a startup into a company and be very famous and awesome.

00:01:12.560 Number one: hire trendy rockstars. You're at a Ruby conference, so you're already well into step number one.

00:01:19.439 Then, be super transparent — that means writing a lot of interesting tech blog posts about how you built your cool company, speaking at a lot of conferences, and being well-known for your company and being awesome. Number five: roll around in money. You hire lots and lots of people, and then you never ever speak to these community members again. You ignore the community, watch everything go up all over the place, and then you start another company.

00:01:37.840 This is really stupid; I don't like this. I don't know why this is now a thing, but I've seen it happen a couple of times now, especially in Ruby, where some companies got really big and talked a lot about this sort of stuff at conferences five or six years ago, and you don't hear from them much anymore.

00:01:56.720 It's interesting when you think about it — there are millions of talks about one-person companies because that's easy. I could create a company right now and say, "Here's the story of how I created this company 20 seconds ago." But there are not as many talks about 10-person companies or hundreds of talks about 100-person companies. Not many talks exist about growing a company from zero to a thousand, because at that point, you're too busy being a CEO or something, and you just can't talk about this stuff.

00:02:19.760 My goal is that GitHub is not that type of company anymore. I want GitHub to remain open and still keep talking about the weird challenges and stuff we face because I think some of the aspects as we grow into a bigger company are the really interesting parts that you don't get from reading some dude's blog post.

00:02:35.680 And that's why I wanted to give this talk. I want to talk about these interesting aspects. So, I am Zach Holman, I work at GitHub. I've been at GitHub for almost four years — my four-year anniversary is in about a week or two, which is strange.

00:02:57.280 That's a lot longer than I ever thought I would stay at a company, and I've sort of talked about GitHub and how we work for two to three years at this point. You may know me from such slideshows as 'How GitHub Uses GitHub to Build GitHub', 'Scaling GitHub', 'How GitHub Works', and 'How to Build a GitHub' — a bunch of different aspects of GitHub.

00:03:14.400 Not necessarily because I really care about git repositories — I think git is kind of silly sometimes — but I would still give these sorts of talks even if we shipped bananas or sold cars, because I really love to talk about developers and building companies and making people happy. I think all that is really interesting.

00:03:36.960 But looking back, all of this stuff sort of stems from when GitHub was maybe 30 to 75 employees. I gave talks back then, but that's sort of when the ideas and foundations of what I talked about stem from.

00:03:49.760 We're 240 now, and 240 is a lot more people, and that has a lot of implications that come with it. Has anybody ever heard about Dunbar's number here? Dunbar's number is that famous number. If you haven't heard or don't know the definition, it's an imaginary employee number that Hacker News fetishizes — it will kill your company.

00:04:07.120 This came up every single time I ever talked about GitHub. I would say, 'Oh, that's adorable, but once you become 25 employees it's all going to change. Once you become 50 employees it's all going to change. 100, 150, 200, a thousand.' Therefore, I don't have to listen to what you're saying, etc.

00:04:31.440 Dunbar's number is basically a guy who studied interpersonal relationships in large organizations. It's the maximum number of relationships a human can meaningfully maintain. At that point, there's some limit to where you can keep all these relationships in your head.

00:04:55.679 The implication is that once you reach that limit, you can no longer rely upon yourself or your buddies to grow your company effectively. You have to start relying on structure and communication, and that stuff becomes really important to grow the company. This is probably about 150-ish, anywhere from 125 to 250 or something like that.

00:05:18.400 But there's some sort of limit, and 150 is a good number where you can realize that once you get to that point, it changes things. I don't know how many people are in really large companies here or startups, but I never thought I would be at a company bigger than 40.

00:05:31.520 So 150 is staggeringly large to me. I tell some people, and they just laugh at me and go back to their 60,000-person company, but this is a big deal for me.

00:05:47.919 Looking back, everything that I've talked about publicly can sort of fit into this idea of how GitHub worked. If you look at how we work today, that's another subset of things, and if you line up the Venn diagram, it probably looks something like this.

00:06:02.959 We're not going to suddenly be like, 'Oh, we’re changing everything — this is all ridiculous. Why did we ever say this?' It’s not that much different but, for me, I think the parts that did change are really fascinating — like the small incremental changes that make up everything.

00:06:14.720 I really think that progress happens along the edges, and if you can start identifying those edges earlier on in your company as it starts growing with the same sort of pain points that we may have faced, that gets you a step up on progressing into the next level of company development.

00:06:35.039 I don't know exactly what that means but this stuff is really interesting to me, and that's what this talk is about. So, the two-minute recap: if you haven't been totally ignoring everything in my life's purpose for the last two years, I’m going to recap this in two minutes because I can sum this up entirely apparently.

00:06:51.360 Number one: GitHub is 60% remote. This is probably the number one thing for GitHub and this is true today. We're still 60% remote, with people who don't live in San Francisco.

00:07:10.080 This sort of informs everything that we end up doing. We don't have stand-up meetings, we don’t have face-to-face meetings very much because a lot of times people aren't in the same office as you.

00:07:28.240 We try to build tools around the idea that we're not in the same time zone either, and this sort of informs how we work along the way.

00:07:40.319 GitHub's asynchronous communication is the other thing. Along the lines of time zones, if you work in twelve-hour shifts, I want our communication to be asynchronous, meaning I can send you something, forget about it, work on something else, and trust that you’ll get back to me later.

00:08:03.680 It’s not synchronous — I’m not waiting on you and you're not blocking me from doing the work I'm doing. So, GitHub has no managers. I would probably put an asterisk next to this, as we still have no technical managers who are assigning bugs and doing the classical managerial stuff.

00:08:45.039 I had those in my first job, and I absolutely hated them. My apologies to anyone who's a manager here, but there’s a famous Venn diagram that Ryan Tomac put in one of his talks where he says there's an overlap between the problems that GitHub has and the problems that managers are good for.

00:09:06.239 The overlap between those two is so small, and I think that’s still true today. After all of this, I want to talk about how things have changed since the time I actually said those things.

00:09:28.560 The first thing I want to talk about is deployment, because deployment is the constant — no matter how big you get, you still have to deploy code to production.

00:09:42.720 Whether that's shipping software like natively or web software, if you're lucky — like us — I had a slide that looked like this: limit your deployments to just your staff.

00:10:02.800 You can deploy your code to just your beta users or to one particular server in your cluster of servers or one specific app process on one server. The idea is that we want to limit the amount of impact your code has on the general population to ensure you don’t break things.

00:10:32.959 That's great, and I don’t disagree with that, but in hindsight I think our deployments have become a lot simpler.

00:10:50.079 For one, we rarely deploy to individual servers anymore. I'm mostly speaking from the perspective of an app developer — I don't have to deal with much of the low-level networking stuff.

00:11:01.560 We have a much simpler deployment process because it’s easy to say, 'Oh, we should probably set up all these controls and deploy only to little specific things.' It ends up not being that meaningful.

00:11:20.399 So, my targets are basically deployed to production. YOLO! Or you can deploy to a staff server in production, just to make sure that only people who have access can work on that code for a little bit, to ensure it's not going to break anything.

00:11:42.000 The reason I say this is that I think staging environments are the first to decay. By that, I mean that if our ops team is facing two problems—say, 'Well, staging is down' and 'Production is down'—they’re more often than not going to say, 'Well, staging will wait,' and it's going to stay broken for a while.

00:12:09.120 We have a staging environment, and it’s mainly for low-level networking, like if you're changing things with a Git protocol or working with stuff on disk. That's the stuff you want to try rolling out on staging first.

00:12:31.680 If you’re a typical Rails developer, just don’t bother with staging; push it out to production right away. It’s simpler, you’re isolated so only staff members can see it, and you have access to all your production data and production hooks.

00:12:48.160 Fewer deployment targets lead to simpler deploy lines, so you're not waiting on some staging environment to stay open. Everything is done through our chat robot, Hubot, so we can ask it, 'Where do I want to deploy?'

00:13:09.520 It’ll say, 'Oh, production's unlocked, staff 1 and staff 4 are locked by auto and Muan. Here are the branches that are deployed right now.' This is a nice, easy way to figure out where I can actually deploy my code and see it in production right now.

00:13:25.920 We currently have about five or so staff environments, including production. The new thing that happened in the last month is that we can also spin up dynamic branch environments, or at least not servers but environments.

00:13:42.080 So, I can say, 'I want to deploy GitHub/my branch to production,' and it will spin up a staff environment just for me on my branch. I can let that sit for a while to ensure it's not going to break and then merge it into master before tearing down that environment.

00:14:00.880 With so many people working on the GitHub environment at this point, it’s very easy to end up with lock contention, trying to figure out, 'I want to get my code out right now; I don't want to wait on you, I don't want to queue up in some dumb line.' That's a nice way to handle that.

00:14:31.119 I've talked a lot about feature flags already. This is a slide I had from a while ago where I said staff-only feature flags are great: they limit your exposure, keep you under real-world conditions, and avoid a bunch of hairy merge conflicts when you're ready to deploy to production.

00:14:54.399 This means that only staff can see the code that you deploy, but it’s on your production servers. That’s great and all, but in hindsight, I think deployments have changed way more.

00:15:10.480 I’m at the point now where I don’t really deploy any line of code without first deploying it to a staff ship server for a period of a minute to you know a day or something.

00:15:24.800 Like, most of our features now are staff ships for weeks or months at a time, and it’s okay to pull things out sometimes. We've had some staff-ship features that were built and awesome for 8, 10, 12 months, but then we realized that they were not good features.

00:15:41.840 This is terrible. We ripped them apart, pulled them out, and just didn’t ship them ever. That’s been great, but the fact that we could get it out in production and let the GitHub internal organization play with it and test it out has been really important for us.

00:16:03.840 And it's only grown in importance as we've become a bigger company. We've also mostly stopped using external beta testers.

00:16:17.360 Usually, we ask, 'Who in this room would like to see some new experimental GitHub feature ahead of everyone else?' and everyone raises their hands, but then you all never give us actual feedback. You're not like testers; you're just leechers.

00:16:34.240 It’s really hard to find someone who can give you valuable feedback, somebody who can say, 'This is not working for me. This might be a better direction to head in.' So we tend not to do this as much anymore. I think part of this is just that GitHub is so big now. We’re 240 people from designers, developers, to HR, and we can rely on staff shipping rather than having external beta testers who don't contribute as much feedback to us.

00:17:04.640 It's also much easier to break things for staff than it is for somebody external to the company, even if you give them the disclaimer that things may break.

00:17:26.720 The other thing that’s changed in the last three or four months is that we do much more team shipping than staff shipping at this point. By that, I mean take the two-factor authentication that we added to GitHub last year, which you should all be using if you don't already.

00:17:43.680 When we were about to ship that, we first shipped it to only people on the 2FA GitHub team. Literally, the people in the GitHub organization who were on the 2FA team, got to play around with it before the rest of the company actually got to see it.

00:18:01.440 That was great because, one, authentication is always a little scary. You don’t want to lock people out of their accounts with some dumb bug, but two, it sort of focuses the feedback.

00:18:21.680 I'll talk more about this at length for the rest of this talk, but it becomes problematic with so many people and so many bright people — 'bright' being a code word for opinionated people.

00:18:40.160 It’s easy to say, 'All right, I'm focused on this project,' but if this guy’s doing something over here and I disagree with what he’s doing, I’ll stop everything, tell him that he sucks, then go back to my stuff, and then he feels terrible for the rest of the day.

00:19:01.360 We want to stop doing as much of that and let people focus on more team-related stuff, so we've been moving more from just generally shipping to everyone internally to doing much more team-oriented shipping.

00:19:23.760 Another big thing I want to talk about is people. As everyone says, 'People are everything' or something like that — people are important, I guess.

00:19:44.960 I had a slide that said to reduce institutional knowledge. This is about bringing new hires into the company and the idea that you put a lot more documentation into wikis, issues, chat logs, and pull requests — not a bad idea.

00:20:13.760 Then I said, 'Your new hire will be stoked to dive in, start reading, and start contributing, so don’t get in their way.' That’s the part where I think we’ve changed a lot. I would say get in their way a lot more.

00:20:35.680 This has become painfully obvious as GitHub has gotten bigger and bigger. You want to be able to let people learn independently and grow independently without you getting in their way, but you also want to kind of point them in the right direction.

00:21:03.440 I think for too long we ended up kind of going hands-off and said, 'Well, we only hire smart people, so therefore they can figure it out.' But it's much, much harder to do that as you grow the company and there are so many different projects going on.

00:21:20.000 So keep information accessible but indoctrinate a lot longer. We have these buddy weeks where your first week, you have to come to San Francisco and we give you a buddy who shows you the ropes.

00:21:34.720 After that first week, we say, 'All right, go off and do stuff,' which is intimidating. I mean, my first week, I sat down with one of the founders who said, 'All right, run the app,' and then it didn't run for two weeks.

00:21:50.880 He was sort of off anyway. We can’t do that anymore; you need longer indoctrination as people acclimate to your culture and processes.

00:22:05.840 It's not just code; it's figuring out HR, it's trying to work out accounting, it's figuring out where to go for health insurance. So we still have a buddy for your first week, but we want to start exploring longer buddy relationships — to the point of three months or six months.

00:22:23.680 Not like hand-holding or anything like that, but just having a point of contact that you can talk to and ask, 'What do I do next? I just finished this project. What are the biggest pain points to focus on next?'

00:22:41.280 This has been something we've been coming around to. It takes a long time to revamp the hiring process, but it's been a learning process. I had a slide that said to trust your employees.

00:23:01.760 That was wrong. Never trust your employees! Just kidding; this is a really good idea. This stems from the idea that a lot of people, especially earlier on, myself definitely included, just didn't want people to micromanage me.

00:23:19.120 Just trust me to do the work, and I won't have to worry about any of that stuff. You want people to focus on the work.

00:23:38.720 The subtle drawback here is that you end up reducing a ton of feedback. The main thing I'm talking about here — I've seen a lot of people bring this up in their talks lately — is imposter syndrome.

00:23:59.760 If you don't know what imposter syndrome is, it's that idea that you don’t feel welcome; you feel like you don’t fit into a particular company or group because you know you have all these smart people around you.

00:24:21.840 That’s intimidating. It’s frustrating sometimes talking to somebody who’s brilliantly obvious, and then they're like, 'You just did X, Y, and Z. Why do you feel this way?'

00:24:39.680 They often say, 'I just didn’t have anybody empowering me, or really telling me how I was doing,' and they're surrounded by brilliant people. It's hard, especially for remote workers, since you don't get that informal check-in.

00:24:55.760 I think we try to come up with better processes to handle this, particularly for remote workers because the last thing we want is to have official IM conversations once a week, where we say, 'Now we're going to talk about you.' That’s not necessarily where the best feedback comes from.

00:25:13.920 For me, I like going out to lunch or the bar and asking, 'How are you doing? What are you interested in working on in the future?' That's the sort of informal setting that’s less stressful for everybody.

00:25:35.040 This can be harder for remote workers, so we’re trying to do longer buddy systems. This has been a topic of interest.

00:25:54.560 Usually, if you’re remote, we have one big summit a year, and then you'll usually come to San Francisco twice a year with your team, and maybe once or twice beyond that.

00:26:09.440 This is just to try to get as much as possible for remote workers; it’s great to have face-to-face contact and informal conversations — the time to relax and just kick it with somebody.

00:26:29.600 That's the stuff I think we can do a lot better at — experiment with more informal approaches.

00:26:36.480 For one, we've implemented robot iPads. I don't know if you’ve seen these — we have these iPads in the office. They can be controlled to walk around.

00:26:53.920 It's a fun gag; it can wander around and get some interaction. I would love to have just a window at the office between remote workers so that anyone can dial in at any time and interact with them.

00:27:14.720 The goal is to create those serendipitous interactions as much as possible because, while chat rooms can get you 90% of the way there, informal interactions are often lost online.

00:27:35.680 A couple of last things I've found helpful in hindsight: this quote is in one of our guidebooks, 'Do GitHub our main repository build websites like it's 2005.' I think that is the greatest thing ever.

00:27:50.240 We try not to get too concerned with the latest and greatest tech, because it just doesn't matter as much.

00:28:04.320 As time has gone on, our tech stack shrinks; we've been ripping out more and more unnecessary components because we know how to scale MySQL pretty well, and we know how to scale Ruby.

00:28:17.680 If Ruby doesn't scale, write in C. If C is not good enough, I don't know, maybe have a drink or something.

00:28:34.880 Beyond that, the more things you add, the more complications arise. We have a good story about how to scale certain problems we've been facing, so we do a lot fewer trendy languages and databases.

00:28:50.800 I think that’s totally fine; your product should be cutting edge, not your technology.

00:29:05.520 I think that’s where you should take risks. You should push for crazy UI instead of worrying about, 'We reached this one weird edge case where MySQL has some weird problem.' That's not really as important.

00:29:20.960 Give me a couple of months, and we’ll have emoji stability; it’s taken a long time to reach the point where GitHub’s stability is acceptable.

00:29:36.120 It takes a lot of money to deal with DDoS attacks and requires a lot of expertise to understand how to manage this sort of stuff.

00:29:51.600 But I think the best way forward is to work with tried and true methods because a lot of smarter people who've gone through this already can probably help you out.

00:30:07.760 The other thing is to resist 'just do it.' I don’t know of a better way to express this. I think GitHub has done a really good job of remaining skeptical every step of the way.

00:30:22.080 If somebody suggests doing something new, we question whether this is the right thing to do. The best example is when we got to about 75 people, we decided we needed to break out into teams.

00:30:41.680 It took us like a year, maybe two, to really figure out how teams worked for us. That’s totally fine; we tried several different approaches and could have easily said, 'Now we have teams, everything is perfect.'

00:31:02.240 It’s been good for us to be skeptical and challenge everything, bench-marking it against our values to build something that works for us.

00:31:18.880 Finally, I think your company should change. You should think about this a lot. But your values shouldn't change. It’s crucial to ask what needs to change.

00:31:40.160 I really think this isn't just a founder mentality or a CEO mentality, or even a team lead mentality. There are ways to contribute to these discussions, even if you're just starting out.

00:31:53.680 Begin looking around and asking, 'Will this scale as we grow? Is this making us happy? How are we doing things today?' Start conversations about how to make us a better company.

00:32:15.200 It kills me to see awesome people stuck in jobs where companies don’t care about their happiness. So start thinking about that more.

00:32:33.040 Thank you.

00:32:55.440 Thank you very much for my presenting and welcoming ideas.

00:33:01.760 We have time for two questions.

00:33:11.680 Thanks for the talk. You spoke about buddy relationships, which are a great way to coalesce team members into the culture.

00:33:14.400 I wonder, do you maintain remote buddy relationships? If so, what techniques have helped?

00:33:35.640 I think there are a couple of ways to deal with remote buddy relationships. One is that it’s mandatory to be in San Francisco for your first week on the job, just to get the culture.

00:33:45.040 That’s because there are so many non-development-oriented things that are that's helpful to have someone there with you to deal with.

00:34:01.360 I think we want to start codifying the idea of having a stronger buddy relationship. At that point, remote is not a big deal; it’s more about how, as a new employee, can I?

00:34:30.560 What project needs doing next, and so on? There are lots of levels of what you can call buddy relationships.

00:34:47.200 We have some people at the company who are really into pair programming, and they do that remotely all the time.

00:34:53.760 Specifically, two people — both ex-Pivotal people — which probably says a lot about them. One is in Colorado and the other in San Francisco.

00:35:12.480 They always team up and essentially everyone else is their buddy all the time because they don’t do anything without pairing.

00:35:23.040 So, there are different levels of what’s advantageous for that particular person.

00:35:34.640 Did you find that going from being a company that sold software to software developers to a company that sold software to enterprises changed how you did things?

00:35:54.400 Did your focus change in the way you presented the product or did it have no impact at all?

00:36:01.000 It did and it didn’t. Some context: we have GitHub.com and we also have GitHub Enterprise, which is our installable version that you can run on your own servers.

00:36:19.040 For the longest time, we thought of them as two different products, and over the last year we've been reducing that distinction as much as possible.

00:36:36.480 The dot-com team is swallowing the enterprise team, and we're realizing they should really be one product; a lot of the features on the enterprise side, like team management, are useful for GitHub.com.

00:36:53.840 So, it’s changed us, but it’s also put us in a better direction. Worries arise when the changes make us no longer work in line with how we operate on GitHub.com.

00:37:12.000 But the real answer is that it's opening our eyes to different use cases that we hadn’t considered before.

00:37:30.080 That’s really interesting, and we're definitely interested in that too because we have to manage our own organization on GitHub.

00:37:41.840 There’s a slew of different problems that come out of that. Some are relevant to GitHub.com; some are relevant only to enterprise, but the way forward is to think of them not as different teams and mesh them together.

00:37:57.120 Awesome! Are you sticking around for some of the day?

00:38:05.920 Yeah, I’ll be here! If you have any other questions, come harass me.

00:38:10.080 You'll be able to find me with my mom; I'm not allowed to leave her alone.

00:38:16.160 Mostly because she won't let me. You're more than welcome to approach me, too!

00:38:23.760 Please give a big round of applause for Holman and his mum.

RubyConf AU 2013