00:00:02.810
Thank you so much, Dana! That was fantastic, and yes, you managed to fit so many pink slides! I loved it! Not that you can tell, I like pink. My mother is so disappointed that I love pink; she spent so many years trying to make sure her daughters weren't into pink.
00:00:08.130
She tried to hide it from us. She tried to do the same with sugar, but that didn't work. Now we are up to our final speech of the day, actually the presentation of the day. We've got Xavier Shay coming to talk to us. He works with the wonderful folks at Ferocia building Up, who you may have heard about. I've been talking about them and their great MarioKart competition and their excellent banking service over the last few days.
00:00:21.439
He's an active member of the open-source community and a member of the Aspect team. He previously worked on Bundler and is a contributor to Ruby on Rails. He recently moved back home to Melbourne after spending eight years in San Francisco, mostly as an engineering leader at Square. He blogs about code, books, and politics, but I don't think all at the same time. He's a runner, a swing dancer, a vegan, and a piano player. Please welcome, for our last presentation at the conference, Xavier!
00:01:16.230
Quick intro: I work at Ferocia, where I'm working on Up. You've probably seen this up at the back; we've been around a bit. If you scan the QR code here and haven't used it before, you'll get ten bucks, which is easy money as far as I'm concerned. Before Up, I was doing a lot of leadership coaching for executive teams, and before that, I was an engineering leader at Square.
00:01:31.650
Before that, I worked a lot with startups around Melbourne. My takeaway from all this is that learning from experience is hard. That's probably the most obvious statement you've heard today. However, I think about this a lot: how can we learn better?
00:01:39.120
From a personal perspective, how can I learn better? But also, in my role, I've had to coach many people who often ask, 'I want to get better. How do I get better?' They are usually very good at what they're doing, so it's unsatisfying to simply advise them to do more of it. I've thought a lot about this question, and I'll spoil the talk by telling you that I don't really have any great answers, but I wanted to share some of the things I've encountered, the things I think about, and what has actually helped me move through this problem.
00:02:02.280
To start with, yes, learning from experience is hard. But specifically, why is it hard? In what ways is it hard? I want to present a couple of different models for learning, environments, and decision-making, and see if we can glean anything from them to better understand what we're actually dealing with here. This is part one of the talk, addressing exactly why this kind of learning is challenging.
00:02:13.480
I'll start by talking about a concept called feedback loops. I'm sure you're familiar with feedback loops; these include things like red-green-refactor, review, CI/CD, and A/B testing. These are the nuts and bolts of software engineering, and I expect engineers to get these basic feedback loops down in a couple of years.
00:02:31.650
However, I wanted to spend this talk discussing what comes after that. If we take the concept of feedback loops and look at the other tasks we need to do to be successful at our careers, we see that the concept doesn't necessarily apply well at all. For example, what about code organization? What does a feedback loop look like there? What about introducing new technology, or features we work on, interviewing, hiring, and mentoring? Applying the feedback loop model here can feel unsatisfying.
00:02:50.320
For example, is having a shorter feedback loop better in these instances? It doesn't really fit. We can say that longer feedback loops are more ambiguous or harder to assess, which is true, but it's not particularly helpful. So, we can't just shrug our shoulders and say, 'Yep, feedback loops are hard.' I wanted to move beyond this and say that while the concept of feedback loops is useful in itself, it isn't enough for many of the harsher problems I deal with.
00:03:10.780
So, let's look at something different. I want to try a model of wicked environments. I learned about wicked environments via a book called "Range" and a paper by Hogarth and friends that details two settings in which inference occurs. In the first, information is acquired, and this represents learning; in the second, it is applied – predictions or choices are made.
00:03:39.420
This essentially states that we learn, then use that knowledge to make predictions or decisions. These are two separate environments. When those environments align perfectly, we create what we call a kind learning environment. For example, in the feedback loops we discussed, if you're learning how to perform a red-green refactor loop, you can practice it in exercises, and when it comes to actually doing it for real, it applies fairly similarly.
00:04:02.110
Similarly, games like chess enable practice and play to mirror each other consistently, hence creating a kind learning environment. These environments are generally easier to learn from because practice yields meaningful feedback.
00:04:32.920
Conversely, a wicked learning environment has mismatches, where what you're learning doesn't necessarily correspond with the problem you're trying to solve. It’s interesting because you may not even realize the mismatch is occurring. The paper continues to quantify how these mismatches can manifest.
00:04:49.110
To illustrate this, I’ll discuss different types of wicked learning environments using hiring as a concrete example. In this scenario, we have a learning environment defined by the interview process.
00:05:00.300
In this situation, I need to predict how well candidates will perform at work based on their interviews. If the learning environment aligns perfectly with the target environment, we would see a clear correlation.
00:05:36.820
However, we all know that interviewing is often imperfect; we don't see this exact line. So what happens when our learning environment is smaller than the target environment? In this case, using an interview process, we might not learn about candidates who don't pass the interview process. This creates a blind spot, meaning our predictions about candidate performance may not be accurate.
00:05:58.660
Alternatively, what if the target set is smaller than the learning set? In this case, a strong interview process may not reflect accurately on performance because it misses good applicants who decline to apply.
00:06:06.340
There can also be cases where both sets overlap but are distinctly different. For example, you might have candidates who pass the interview but shine due to an excellent mentoring program that skews your overall results.
00:06:33.490
There might be individuals who may not perform well based on the interview but excel at their roles due to a supportive environment. This complexity can be highlighted by the famous quote from W. Edwards Deming, the management theorist: ‘What should we do with the dead weight in our organization?’ His response was, ‘It depends, were they dead when you hired them, or did you kill them?'
00:07:06.690
This highlights the challenges when your learning environment doesn't match up with your target environment, leading to a lack of correlation between tests and performance, something you want to avoid. The interesting point here is that the model itself isn't quantifiable. You can't simply state that they intersect at 20% or whatever. Instead, the utility lies in listening to these mismatches and reflecting on which case you are dealing with.
00:07:49.000
From a technical, mathematical perspective, we see that missing data can present major challenges. There’s nothing concrete we can do about it. However, with interviewing, we actually understand the domain better, which gives rise to potential corrective measures.
00:08:02.110
For example, if we want to improve our blind spot in interviews with candidates that didn’t get hired, we can maintain relationships with these candidates and share our assessment from the interview process. I’ve personally seen cases where I believed the candidate would succeed in the future, so I kept in touch only to see them thrive down the line.
00:08:37.079
We might also consider taking a chance on those whom we're unsure about. If you have the luxury of being able to say, ‘Okay, let's try this out,’ you can potentially learn from these ‘edge' cases and often be pleasantly surprised.
00:09:18.630
Additionally, we can also be explicit about what we screen candidates for and experiment with our screening methods. For example, if I’m hiring for curiosity as an attribute, I might create a question like, ‘Tell me about something you learned recently.’ This question might not guarantee a good interview, but it gives us the chance to see how people respond.
00:09:43.660
With a more extensive pipeline, we can create a feedback loop that helps us understand how our questions correlate to performance, thereby allowing us to tackle the structural problem of interviewing more effectively.
00:10:06.790
Alright, this is the concept of wicked environments, which I thought was interesting. Now let’s apply this to our work with Ruby. In our jobs, most of us use Ruby and had to make architectural decisions about how to write code. When I think about the learning environment as Ruby itself and the performance environment as our success, I'm struck by things we may or may not learn due to our Ruby community focus.
00:10:54.690
For example, Ruby developers typically lean towards processes over threads for historical reasons. We also tend to avoid certain database practices, and very few experience working with stored procedures.
00:11:14.150
Even with test-driven development (TDD), we are generally proponents of testing, but have you considered how what we focus on limits our exposure to other methodologies? For instance, property testing is much bigger in more strongly typed functional languages.
00:11:33.520
Similarly, concepts like dependency injection in many modern Java projects or Command Query Responsibility Segregation (CQRS) aren't highlighted as much within Ruby. The challenge here is recognizing that while we excel in our community focus, we often miss out on broader software engineering methodologies. Thinking about this idea of wicked environments, my instinct is to expand my 'blue circle' of learning to better align with the performance goals.
00:12:06.190
But as I pondered more about it, I realized the idea that my blue circle would always expand over time didn’t ring true. I get better at solving problems with the tools I have, yet remain largely unaware or quite separate from other methods that exist.
00:12:17.770
Next, we navigate to the expert's curse, which I think has much to teach us about how we approach these topics. The expert's curse describes when a useful technique transforms into a philosophy. It occurs when a tool designed for problem-solving becomes a lens through which we assess all problems.
00:12:55.280
For instance, we might view everything through the TDD lens, automatically judging code as bad if it lacks tests. This approach reinforces dogmatic thinking rather than critical analysis of new situations. This not only leads to poor architectural decisions but blinds us to alternate solutions.
00:13:25.390
Next, there's the science of forecasting, based on making better decisions in wicked environments. Philip Tetlock is a leading author in this realm, investigating how to navigate unknown settings.
00:13:38.280
What I learned is that collaboration among diverse experts can yield better decision-making than allowing a singular philosophical lens to dominate.
00:14:00.460
This resonates with a truth in the industry where overconfidence can lead us to poor decisions. I’ll share this quote from 'Range': 'Experts viewed every world event through their preferred keyhole. Doing this made it easy to craft compelling stories about anything that occurred, told with adamant authority.'”},{
