Moneyball At The Keyboard: Lessons on How To Scout Talented Developers

by Adam Jonas

The video 'Moneyball At The Keyboard: Lessons on How To Scout Talented Developers' by Adam Jonas emphasizes the importance of objective talent evaluation in both baseball and the tech industry. The presentation draws parallels between traditional scouting methods in baseball and how to better assess developers in software engineering.

Key Points Discussed:
- Introduction and Background: Adam Jonas, Managing Director of Engineering at the Flatiron School, shares his background in baseball scouting and the transition to the tech industry, emphasizing the need for objective evaluation metrics in hiring developers.
- The Moneyball Philosophy: Referring to Michael Lewis's 'Moneyball,' Jonas discusses how the Oakland Athletics utilized statistical analysis to identify undervalued players despite budget constraints.
- Traditional Scouting in Baseball: Jonas describes the 5 tools used to evaluate baseball players—hitting for average, hitting for power, running, fielding, and throwing—and explains how subjective evaluations can lead to missing out on talent.
- Identifying Developer Talent: The talk shifts focus to the Flatiron School and its five criteria for assessing potential developers: hireability, technical background, aptitude, passion, and culture.
- Anecdotes and Examples: Jonas shares stories of baseball icons like Derek Jeter, Trevor Hoffman, Albert Pujols, and David Ortiz to highlight the importance of context in talent evaluation and how traditional methods can overlook potential.
- Methods for Improvement: The speaker provides actionable strategies for better talent evaluation in the tech industry:
- Acknowledging the uncertainty of where talent originates.
- Controlling for sample size bias by increasing touchpoints and evaluations over time.
- Rethinking cultural fit to prioritize organizational values over personal attributes.
- Conclusion: Jonas concludes with the notion that successful talent evaluation requires introspection about what is valued within an organization and a commitment to objective criteria over instinctual bias. He encourages fostering a diverse environment to enhance creativity and decision-making.
Overall, the session urges viewers to adopt a more scientific and inclusive approach to identifying and nurturing developer talent based on the lessons learned from baseball scouting.

00:00:14.650 This is one of those offbeat talks, so we're going to expect a lot of energy. Do the wave or something! Let's say welcome to 'Moneyball at the Keyboard: Scouting Talented Developers.' I really do appreciate that you're joining me at this time slot. I know there are some awesome talks, and I appreciate you being here because there's no talk about you.

00:00:20.779 I'm Adam Jonas, the Managing Director of Engineering at the Flatiron School, where our primary goal is to inspire people to fall in love with code. We train some of the best junior developers around, placing them in awesome companies like Kickstarter, Etsy, Intel, and The New York Times. I am not a teacher, but my team has been working on transforming our internal learning management system into a full-fledged education platform called Learn.co. We're really proud of it and would love for you to check it out. This work has inspired me to start putting together these slides because I've spent a lot of time thinking about how we could create metrics to surface talented students.

00:01:02.420 As I often do, I fell back on a sports analogy, and today we're going to talk about baseball. We'll use baseball as an example of how talent is identified in that sport to improve our mental model for thinking about talent in software development. So why am I specifically talking about baseball? It's because, thanks to Flatiron School, software is actually my second career. I worked in scouting and player development for teams like the Brewers and the Twins, then ran an academy, spending most of my twenties living in and around Latin America, specifically the Dominican Republic.

00:01:36.710 I moved back to the US in 2010 and tried my hand at founding a company that helped draft players better and improve their bargaining positions, helping them make more informed decisions about their futures. While this venture went terribly wrong, it piqued my interest in code, which eventually led me to the Flatiron School. Here's what we're going to cover today: I'm going to tell you about the Moneyball philosophy and why it created a shift in baseball. Then we're all going to scout school together and see how professional baseball talent has been traditionally scouted. After that, we'll look at how talented students are identified at the Flatiron School. Finally, based on what we've learned, we'll discuss three ways to better understand developer talent and how to evaluate it going forward.

00:02:10.280 If you're still awake at that point, we'll hopefully do some Q&A at the end. One more thing before we dive in—there's a lot to cover—but I think it's natural to think about these sorts of conversations specifically in the context of hiring. However, we evaluate talent all the time: we assess our co-workers because it affects compensation, we evaluate open-source contributions, and we put value judgments on everything we see in resumes or hear from speakers. It's important to remember that we're wired to evaluate talent, and that's perfectly okay, but I challenge you not to limit this concept to just the interview process; it's much broader than that.

00:02:44.180 You might be familiar with 'Moneyball.' It was a book written in 2003 by Michael Lewis and later turned into a movie in 2011 starring the always handsome Brad Pitt. The essence of 'Moneyball,' while focusing specifically on baseball, is really about objectifying what was previously thought to be subjective. The subject of 'Moneyball' is the Oakland Athletics. They have smaller revenues and constrained resources compared to other teams they compete against, like the New York Yankees and the Boston Red Sox.

00:03:11.500 Because of these constraints, Oakland was forced to search for undervalued players in the market, leading them to utilize underutilized tools of statistical analysis. So, what did we learn from the 'Moneyball' approach? We learned that industry outsiders who had not been indoctrinated in the traditional way of doing things could identify inefficiencies in the old methods. We discovered that we could use better talent analysis tools to determine a player's objective value.

00:03:46.680 For example, what's the difference between a player who gets to first base via a single and a player who gets to first base via a walk? Intuitively, we favor the player who took the action—the one who swung the bat and earned their way to first base. But from a team's contribution perspective, it’s about the same; you have a guy on first base, and that's all that matters. When we think about these contributions, we often overvalue what we think is being earned.

00:04:07.450 This is precisely what was happening in baseball. Players with high batting averages were being overvalued, while those with high walk percentages were being undervalued. The 'Moneyball' approach capitalized on this gap between perception and reality, more accurately evaluating contributions to the team. This objectivity isn't perfect, though, because humans are complicated. Let's turn to a more humanistic approach by heading to scout school together.

00:04:42.550 Scout schools are a real thing, sponsored by Major League Baseball. I was lucky enough to attend in 2005, and you have to be invited by a team. You spend two weeks reviewing the theory of scouting in the mornings and applying what you’ve learned in the field during the afternoons—writing reports and watching different levels of the game. The first thing you learn is that all players are evaluated using five categories known as the 'tools': hitting for average, hitting for power, running, fielding, and throwing.

00:05:07.490 This is what everyone is evaluated on, and players who possess elite talent in every tool are rare. Willie Mays stands out as the prototypical five-tool player because he excelled in all areas, but most players have a mix of strengths and weaknesses that determine their roles and contributions to the team. Scouts have rubrics for these tools, and they provide them in a binder.

00:05:35.640 Let's take a closer look at the running tool. The grades for running range from 2 to 8, assessed by measuring the time from when the bat hits the ball to the player's arrival at first base using a stopwatch. However, there are complicating factors—right-handed batters are physically further from first base than left-handed hitters, and players don't always run at 100% effort. We need to consider when the right opportunities arise to measure these times, such as during lazy pop flies or base hits.

00:06:03.520 Traditional scouting has a reputation for being purely subjective, seemingly at odds with the 'Moneyball' approach. However, you can see that there is some standardization and consistency in the evaluations. The only way this can work is if the evaluations are documented consistently. Baseball has done an excellent job of maintaining a robust audit log for examining decisions over the years. This is an actual scouting report; the rubric is there on the left, and it outlines various categories of evaluation.

00:06:34.010 This allows us to understand why we would hire or pass on a player and where the candidate stands in relation to their potential future performance. Each report also provides a wealth of data—not just on the candidate but also for us as evaluators. Leaving a record can serve as proof and ultimately provide your redemption against any naysayers. I want to highlight three particular statements from scouting reports.

00:07:12.340 The first is about the subjective nature of scouting. Scouting often involves pattern matching, where the scout feels they've seen certain traits in players before. They might note something as being the face of a franchise, predicting the player would reach the major leagues by age 21. This prediction made for an 18-year-old who would eventually reach this milestone, becoming a remarkable player. This is reflected in the report of Derek Jeter, drafted sixth overall in 1992, debuting at age 21 and becoming a 14-time All-Star. The scout's accurate prediction provides credibility years later.

00:08:05.850 Let's look at three other examples to illustrate how scouts can miss players. Trevor Hoffman, a future Hall of Fame pitcher, was drafted as a shortstop before being converted to a pitcher after proving he couldn't hit. Albert Pujols was drafted 402nd overall, despite every team having multiple chances to select him. He became Rookie of the Year and the fastest player from his draft class to reach the major leagues.

00:08:38.910 David Ortiz was released by the Minnesota Twins before being picked up by the Boston Red Sox. Essentially, these players were missed due to not being seen in the proper context; they performed well, but various circumstances obscured their talents.

00:09:00.540 This reinforces the notion that talent and context are linked; there’s no such thing as general talent. For instance, Derek Jeter had a successful career but wouldn’t be someone you’d hire for certain jobs unrelated to baseball—even a simple task like taxes. Talent exists in context. Players are expected to contribute differently based on their roles. For a shortstop, fielding and throwing are crucial tools, while for a first baseman, offensive power-hitting drives their value.

00:09:53.430 This brings us to specialists, like pitchers who are evaluated primarily based on their arm strength. Jim Abbott had a successful ten-year MLB career, despite being born without a right hand. He represents the idea that context is essential; having a specialized skill can lead to success despite limitations. This begs the question: what's your context when assessing talent? Are you seeking a generalist for a startup that values flexibility, or individual contributors in a larger organization that prioritizes optimization?

00:10:36.070 Understanding the unique set of circumstances your team faces is critical. Let me share how we identify students at the Flatiron School. Our school connects with thousands of students each year and has a 6% acceptance rate. Over the last three years, we've accepted hundreds of students, all of whom have been job-seeking. Almost every accepted student has successfully found a job.

00:11:16.490 When discussing student admissions, I was reminded of my time in the Dominican Republic, where many players had limited training. How can we know a player has talent before they can even play the game? Identifying talent amidst little context is a challenge, so consider the five tools of the Flatiron School: hireability, technical background, aptitude, passion, and culture.

00:11:56.300 Before any student has written a line of code, these are the attributes we’re looking for. 'Hireability' refers to our goal of ensuring students can go from no experience to employment quickly, landing jobs with high starting salaries and rapid promotions, receiving positive feedback from their employers. We evaluate if they can hold a conversation about teamwork, showing they have the enthusiasm and energy to succeed in interviews.

00:12:36.520 Technical background is just as it sounds: how much experience do they have? Aptitude is about how well they can learn and integrate new information. Our tic-tac-toe coding challenge is a common example, where students often pull solutions from the internet. Our goal is to differentiate between what they genuinely understand and what was pieced together from external sources.

00:12:58.570 This approach is fine because coding isn’t always a solo endeavor. We should use similar tools that reflect real-world expectations. What we truly seek is pattern recognition: understanding similarities and differences, and identifying the immediate next steps for improvement. Our rubric for evaluation ranges from one to five, assessing their comprehension and ability to produce clean, object-oriented code.

00:13:42.030 Passion can be somewhat Sisyphean because learning to code is an ongoing process. Our best students thrive on mental challenges, surviving the mental punishment of grappling with new concepts. We look for evidence of their commitment to overcoming challenges. Culture is a more complex issue, as we don’t admit students; we admit entire classes, which need to reflect gender and ethnic diversity as well as diverse backgrounds and perspectives.

00:14:28.320 If we have a class all with finance backgrounds, we may ask some to defer, as too much homogeneity hinders learning and collaboration. An anecdotal finding reveals that around 80% of our top students engage in some creative art forms, be it professionally or through hobbies. These students demonstrate a unique approach to problem-solving.

00:15:06.790 Here's an overview of the actual ratings from our last class. By a show of hands, which aspect do you believe was the best predictor of success regarding speed to job placement, starting salary, and employer satisfaction? Was it culture fit, passion, technical background, hireability, or aptitude? The answer, surprisingly, was passion, and it was by a significant margin.

00:15:41.200 Passion was the most reliable indicator of whether a candidate would succeed, and it wasn’t even close. Interestingly, technical background had minimal correlation with the speed at which students found jobs or what their starting salaries were. Passionate students not only caught up with their more experienced peers but ultimately surpassed them.

00:16:15.460 While this rubric can be beneficial, it's essential to realize that not everyone operates within the same context. Hence, it’s vital to tailor evaluation criteria to your specific situation. Consider what truly matters to you. Possible areas to evaluate include performance reviews—though they can be fraught with issues—mission statements, value statements, one-on-ones, and retrospectives.

00:16:47.450 Ultimately, you cannot succeed in evaluating talent without understanding what you value and what’s essential to you. We've discussed many abstract concepts. Now, let’s focus on three key actions that will help you think about talent starting today and, perhaps, uncover some hidden gems.

00:17:09.890 First, admit that we have no idea where talent comes from. Second, control for sample size bias. Finally, rethink cultural fit. Consider Trevor Hoffman, who was drafted as a shortstop but converted to a pitcher. Is it completely out of the question that someone in a generalist role could excel if they focused on their strongest skill?

00:17:55.630 What about Albert Pujols? He was drafted from Maple Woods Community College, a school many might overlook. Or consider the unique circumstances of David Ortiz, who navigated various personal challenges that impacted his performance. We struggle to separate context from personal attribution. In baseball, this leads to “fundamental attribution error.” The same is true in hiring.

00:18:33.410 In the major leagues, only 600 players are active at any given moment, and only 72 of those make it to the All-Star game. Most of these are the top 12% of talent, including first-round prospects who were well-known. So, if a prospect is expected to be a star, everyone knows it—but true talent is often discovered outside of this spotlight.

00:19:11.930 This reveals the second concept: sample size bias. We are wired to make snap judgments, regardless of our attempts to remain objective. The challenge is to increase our touchpoints without overextending ourselves. It took baseball years to realize that one game is simply a snapshot. In an industry as nascent as software, we can sometimes think we can base evaluations solely on one interview.

00:19:50.260 It's important to admit that we are worse at evaluating talent than we think we are. A fuller picture is essential, and presently, we use various methods such as whiteboard challenges, phone screenings, open-source contributions, resumes, blogs, and other indicators to evaluate candidates over time. We need to ask: how much does this person love their work? Are they creative? Will they enhance our team?

00:20:29.000 Traditional interview methods can yield good insights if we ask the right questions. Cultural fit is tricky; it has become a loose term and should represent less of a match for personal fit and more for organizational values. These are complex issues that require serious self-reflection. Data indicates more diverse teams outperform homogeneous teams for jobs that require creativity and complex decision-making.

00:21:07.270 It's imperative to confront our biases, both personal and institutional. Research indicates that diverse teams outperform those that are more homogeneous in performance and creativity. To achieve this, we must tackle our biases head-on. This is daunting, but examining our internal data can illuminate the changes required. We often gravitate towards what we know—people with similar backgrounds and skills—but can that approach truly lead us to the best candidates?

00:21:44.570 Consider Jackie Robinson, signed by the Dodgers not just for his social significance but as a competitive advantage. He provided access to a talent pipeline that others shunned. While it was the right choice socially, it was a calculated risk with immense potential benefits—having access to talented players who offer value to organizations.

00:22:17.990 High-priced players may bring value, but it's unrealistic to consistently acquire the most expensive ones unless you are the New York Yankees or Boston Red Sox. Talent requires opportunities, so establish objective criteria, maintain detailed records, and introspect about your scouting abilities without cultural screens acting as barriers. Moving away from relying on instincts will foster a more inclusive industry and advance us toward a more complete meritocracy.

00:22:56.290 How can we approach evaluating technical background differently? My team and I have developed our own set of criteria—communication, ownership, adaptability, velocity, and quality. Given our small team and fast pace, these characteristics guide our evaluations, and we have assigned rubrics to assess candidates.

00:23:34.690 Moreover, we often hire out of our own student pool at Flatiron. Every candidate who completes our program becomes someone we know as a capable developer, but evaluating external candidates can be more challenging. So, when evaluating cultural fit, how does this candidate stand out from others in the organization?

00:24:03.270 It’s important to treat culture fit as a screening tool cautiously. Each member of our hiring panel, particularly recruiting managers, uses culture fit as just one piece of the evaluation process. Generally, I consider how a candidate could round out the skills of my team, and passion and desire to grow play significant roles in decision-making.

00:24:43.470 Reflecting on the evolution of Flatiron, I was part of the initial class which was untested and now we have a system in place. We can now support students more efficiently than before, covering double the material than earlier iterations and fostering greater educational gains.

00:25:12.150 Boot camps like Flatiron have transformed lives significantly, but it’s vital to discern whether the offerings meet the needs of each student. We conduct audits to track employment statistics and outcomes for students, collecting data on their salaries and results.

00:25:48.570 It appears that we are unique in performing such audits. As a result, we take this commitment to evaluating student success seriously. When it comes to grading, it's context-dependent—we apply different rubrics tailored to the roles we’re hiring for.

00:26:20.360 Thank you, everyone, for your attention. I really appreciate you joining me today.