Talks
The Dangers of Tribal Knowledge
The Dangers of Tribal Knowledge
by Annie SextonIn her talk titled "The Dangers of Tribal Knowledge" at RubyConf 2018, Annie Sexton addresses the issues that arise when critical information is only held in the minds of a few individuals in a team, a phenomenon referred to as tribal knowledge. This presents significant barriers to efficiency and self-sufficiency within organizations.
Key points discussed include:
- Definition of Tribal Knowledge: This is information that exists only within the minds of team members and is not documented elsewhere, creating potential bottlenecks and risks if individuals leave.
Impact on Team Dynamics: Annie describes scenarios where both senior and junior developers suffer from the burden of tribal knowledge—senior members face interruptions while the new ones struggle to find essential answers, leading to frustration on both sides.
Consequences of Poor Documentation: Essential information can be scattered across various platforms like Google Docs, Trello, and emails, making it difficult for team members to find necessary resources.
Creating a Self-Sufficient Team: To address the challenges of tribal knowledge, Annie proposes the establishment of a structured onboarding process made up of two main components:
- A Crash Course: This serves as an introductory guide to help new employees understand the basics quickly, enabling them to seek answers independently afterward.
- A Knowledge Base: An organized repository where comprehensive documentation is stored, ensuring it’s easy to find, update, and search for information.
- A Crash Course: This serves as an introductory guide to help new employees understand the basics quickly, enabling them to seek answers independently afterward.
Example from Heroku: Annie shares her experience at Heroku, highlighting how the support team introduced a crash course and a knowledge base for onboarding new support engineers. This included essential readings and procedures to facilitate better understanding.
Keeping Documentation Up-to-Date: The importance of regularly reviewing and updating the documentation is emphasized, including setting expiration dates for articles to ensure all information remains relevant.
Empowering New Team Members: Annie concludes by stressing that new employees should feel confident to voice their understanding (or lack thereof) to continually improve the documentation process.
Overall, the talk serves as a call to action for teams to document knowledge comprehensively, thereby preventing bottlenecks and enhancing self-service capabilities among members, which ultimately leads to a more efficient and productive work environment.
00:00:15
Hi everyone! Are you guys enjoying the conference so far? Woo! So am I, it’s great!
00:00:23
So, what? You said my name is Annie Sexton, and I spend a lot of time thinking
00:00:31
about how people absorb knowledge. That’s why I’m really excited to talk to you guys about tribal knowledge, why it’s so dangerous, and how to address it.
00:00:41
By learning to address tribal knowledge, you can scale your team quickly and help them become self-sufficient, which I think is the most important thing.
00:00:54
But before we get into that, I know she gave me a bit of an intro, but I’ll probably say the same things once more.
00:01:06
I'll tell you a little bit about myself. So, she mentioned I was a digital nomad.
00:01:11
I’m a retired digital nomad; I just bought my first place.
00:01:17
But I spent all of last year living out of a suitcase and globe-trotting; it was a lot of fun.
00:01:28
However, I got sick of airports really quickly.
00:01:36
So, I’m also a volunteer English teacher at a non-profit in my hometown of Austin, Texas.
00:01:41
I really enjoy teaching, and as she said, I’m a full-stack developer and have been for about five years.
00:01:50
I worked at lots of different web agencies and spent a lot of time on my own Rails applications.
00:01:55
But, of course, this was all before I was hired by the man. And by the man, I mean Salesforce, and by Salesforce, I mean Heroku.
00:02:06
And it is great! Totally candid photo here, definitely not staged. It’s an awesome place to work.
00:02:17
My coworkers are super smart and super friendly. I get to work on my dream platform, and Salesforce, our parent company, is really awesome.
00:02:33
They are like, 'Hey, what’s up for Salesforce? We have this guy,' so it’s great!
00:02:41
Overall, I really, really like work; it’s the best job I’ve had.
00:02:50
But there was some room for improvement in the onboarding process.
00:02:57
Heroku is a bit of a complicated platform; easy to use, right? We pride ourselves on making it easy to use.
00:03:05
But you know, there are a lot of things going on behind the scenes.
00:03:10
There was a lot of information I needed to know to ramp up for my job.
00:03:21
Let me tell you guys what I do at Heroku.
00:03:27
I'm what we call a Support Engineer, which sounds as vague as Solutions Architect and Success Developer.
00:03:35
But all that means is that I'm on the support team.
00:03:41
We are called engineers because we have to know what the customers are talking about.
00:03:55
We don’t have a tiered system when it comes to support; instead, we break it up into core support.
00:04:05
These are basically the generalists, or the first line of defense whenever tickets come in.
00:04:13
If we need specialized help, we can get someone with specific knowledge about Kafka, Postgres, or Heroku Connect.
00:04:21
If any of you caught Caleb Thompson's talk earlier, he is our resident Heroku Connect expert.
00:04:36
So, they hide him in the hall, but I am on the core support team.
00:04:49
As a generalist, it’s our job to be the power users of Heroku.
00:05:03
We basically have to be better at Heroku than everyone else in this room combined.
00:05:12
There’s a lot to know; there are a lot of moving parts, and it can be a little difficult.
00:05:23
And boy, howdy, lately the ticket queue has been getting out of control.
00:05:33
Which is why Heroku is hiring all of you RubyComers! Congratulations, you’re all hired!
00:05:42
That’s right, you get a job, and you get a job! And please don't sue us, this is just a joke.
00:05:51
So just to get you started as a new support engineer, we’ll need to make sure you’re all familiar with the platform.
00:06:02
I’m just going to give you a quick download of what I know about the platform.
00:06:07
So take notes! Heroku lets you deploy, run, and manage applications written in Ruby, Java, Python, Clojure, Scala, Go, and PHP.
00:06:18
These languages are provided by our build packs, but if you need further dependencies, you can use third-party build packs or even Docker.
00:06:31
If you'd like to use your own custom domain, you can do so by running 'heroku domains:add yourdomain.com' and creating a CNAME record.
00:06:43
Or for the alias, an A Name record for an ISA root or apex.
00:06:50
I feel like that’s not the best way of conveying this information. Maybe there’s a better way.
00:07:01
For the record, that’s not what Heroku onboarding looks like, thank goodness.
00:07:13
But let's be honest, there’s just too much about the Heroku platform for me to convey that information in a timely manner.
00:07:20
That would be a poor use of your time and mine, wouldn’t you agree?
00:07:35
Luckily, we don’t have to do it that way; we have the Dev Center.
00:07:43
How many people here have read articles in the Dev Center?
00:07:51
Thank you! So, we’ve got it all documented. We don’t have to convey it verbally, thank goodness.
00:07:57
But when I was being onboarded, there were things about the platform that were not in the Dev Center.
00:08:06
Information about the architecture, how things were interconnected, what were its limits, what were its edge cases.
00:08:14
Some of this was documented, but it was all over the place.
00:08:28
On top of that, I needed to learn so much more just to do my job as a support person.
00:08:39
The internal tools we use for debugging customers’ apps, how to use Splunk, and all these CLI commands I had never used before.
00:08:51
We also have our own support system because we are all developers. We decided to just build the system ourselves.
00:09:05
There was so much to learn.
00:09:17
So, there was a lot of tribal knowledge I had to absorb.
00:09:26
What is tribal knowledge? Let’s define it.
00:09:35
Tribal knowledge is that information that exists in the brains of your coworkers, your tribe, and nowhere else.
00:09:45
And this is no bueno! It’s dangerous.
00:09:54
I’ll give you an example: how many of you can relate to this?
00:10:02
You’re headphones on, you’re in the zone, you’re super focused, and then suddenly something distracts you.
00:10:11
Hey, hey, sorry to bother you, but I have a quick question: how do I dot, dot, dot?
00:10:21
You’re the senior experienced developer who reluctantly removes your headphones.
00:10:29
Concentration is broken; that sucks, right?
00:10:39
But let’s say the roles are reversed.
00:10:47
You’re the new kid, just trying to get people to like you, and you’re working hard.
00:10:55
Yet the answers you’re looking for aren’t documented anywhere.
00:11:08
What do you do?
00:11:13
If you're in that situation, guess what, you get to be the one to wake the sleeping dragon.
00:11:22
That’s always fun, right?
00:11:28
First of all, stop hiring dragons and be nice to your new people!
00:11:37
Tribal knowledge sucks for everyone.
00:11:46
Especially in a growing business, it is unfortunately common for all of this important information to reside in the brains of just one or two people.
00:11:55
Sometimes this is harmless, but sometimes it’s not.
00:12:04
For example, my website is down; I can't reach my developer, and I don't have access to the app on Heroku.
00:12:18
Let me start here by saying that credentials should never be tribal knowledge.
00:12:29
Unfortunately, this happens. We have to tell these well-meaning business owners: 'I’m sorry, your email isn’t listed as an owner or collaborator. I can’t provide you any details about the app.'
00:12:39
They will have to reach out to their developer, which is a frustrating thing to hear.
00:12:51
Tribal knowledge is dangerous for two reasons.
00:13:02
Number one, it creates bottlenecks for both senior and junior people.
00:13:09
It also creates liabilities, because when not if, when those people leave with that information, you’re screwed.
00:13:17
Now, some of you might be thinking, 'Listen, Annie, I don’t know how they do it at Heroku, but we do document our stuff!'
00:13:26
Okay, it’s just that these other people don’t know how to look things up themselves.
00:13:32
That’s fair; I totally get it.
00:13:40
Where are these articles kept? Oh, you know, Google Docs, Markdown files, GitHub issues, Trello cards, emails, Google Hangouts threads, Quip.
00:13:46
That’s a lot of places to store stuff!
00:13:52
And maybe it’s a little hard to find the information, but listen, I get it. Occasionally people are just lazy!
00:14:04
And that’s worth acknowledging.
00:14:16
The first thing that needs to happen is you need to set boundaries.
00:14:23
Without boundaries, you can't have a self-sufficient team.
00:14:34
Train your clients, coworkers, and managers to work around your schedule.
00:14:41
This is really, really important.
00:14:49
If you have some of this tribal knowledge documented, maybe people aren't able to find it.
00:14:55
It’s written down, but no one knows where to look.
00:15:04
It’s like the ELA de Muerta, a location that cannot be found.
00:15:11
This is fine if we’re talking about cursed treasure, but we’re not.
00:15:19
So what do you do? There are two main issues: there's people not knowing what is documented.
00:15:27
Then, there’s people not knowing where it’s documented.
00:15:34
These seem like really solvable issues, yet they are very common.
00:15:43
That’s why I want to go in-depth with you about how to address these problems and why they need to be addressed in this way.
00:15:55
Let’s assume this is a worst-case scenario: you have nothing documented. What do you do?
00:16:03
Well, since you can’t just give away the information, how do you transfer it?
00:16:11
By show of hands, how many of you, when you get a new appliance or a new car, read the entire user manual?
00:16:19
Okay, you guys are the outliers!
00:16:28
Personally, I like to see how far I can get without the manual.
00:16:37
I like to live on the wild side. Because let’s face it, most manuals suck!
00:16:46
They really do!
00:16:50
So instead, we need two things to ensure that the tribal knowledge we’re documenting is actually digestible.
00:16:58
We can't just plug someone in matrix-style and say, 'You know kung fu.' So what do we need? Instead, we need a crash course and a knowledge base.
00:17:09
If there's anything you remember from my talk, it’s these two things: a crash course and a knowledge base.
00:17:17
Now, in their simplest terms, these two things are just two different sets of documentation.
00:17:25
They can take any form; they could be webinars or videos, but they don't have to be.
00:17:34
It’s easier for instruction's sake to talk about them as two different sets of articles.
00:17:43
So, let’s talk about the first one: the crash course.
00:17:50
Now, I’m calling it a crash course, but you could easily call it a getting-started guide.
00:17:57
The point of this isn’t to document everything you need to know for a particular job.
00:18:06
It’s just enough information to get someone started.
00:18:12
So they can then self-educate and look up the answers on their own.
00:18:19
It’s not everything; it’s just a primer. It’s just the basics.
00:18:28
Now, who kind of sucks at explaining the basics? Engineers and developers!
00:18:36
I know who you are; listen, I’m totally generalizing, so take that with a grain of salt.
00:18:44
But the point is, we’re regularly criticized as a group that we can’t talk in layman’s terms.
00:18:52
They make TV shows about how poorly we explain things.
00:19:00
Terrible, awful shows I might add!
00:19:08
The problem is that we explain highly technical terms in highly technical terms.
00:19:16
And that doesn’t help.
00:19:24
Let me give you an example: I once asked a buddy of mine, who is an expert in Postgres,
00:19:30
'What’s the difference between truncate and delete in SQL?'
00:19:40
And this is what he replied: 'Truncate is more like a DDL (data definition language) statement than delete, which is a DML (data manipulation language) statement.'
00:19:54
Good? Yes? Okay, what now?
00:20:02
So, truncate is a fixed cost operation versus delete, which is closer to a parenthesis where n is the number of rows affected.
00:20:11
Okay, we’re getting closer, but I still don’t know.
00:20:19
Truncate is like, 'Give me a fresh table! Boom!'
00:20:29
That is what he should have started with!
00:20:36
And later he added, 'Oh yeah, truncate is like the big guns!'
00:20:44
But you see what I mean? We often go into describing the molecular composition of the bark, instead of the forest.
00:20:53
We need to be able to step back.
00:21:02
And by the way, I’m not throwing my friend under the bus here; he’s usually quite good at explaining things.
00:21:10
I just thought that particular example was great.
00:21:18
So, this is something we have to be aware of when we’re writing our crash course.
00:21:26
We need to be conscious of the terms we’re using.
00:21:35
The biggest thing about the crash course is it has to be digestible.
00:21:42
This even applies to non-technical things. For instance, if I told you guys that you can update your v2 mom through the Aloha portal and org 62,
00:21:51
you’d be like, 'What did you just say?' Unless you're a Salesforce employee, that means nothing to you!
00:22:00
And that’s normal; you have to think about this when you're writing your crash course.
00:22:10
You got to think of several questions: Who am I writing this for?
00:22:18
What is their background? What terms do I need to explain?
00:22:25
And what am I assuming they already know?
00:22:31
For example, do they know what the Aloha portal is?
00:22:38
So, the last thing I’ll say about the crash course is that it should be a crash course.
00:22:45
It’s not like a whole Udemy course that you take over several months; it needs to be short.
00:22:56
It should cover the material without going into too much detail.
00:23:05
Let’s talk about the knowledge base, which is the second part.
00:23:11
With the knowledge base, we were concerned with making the crash course not too long.
00:23:24
The knowledge base is where you put everything else; it's an expansive repository of articles that people can refer to.
00:23:33
This is a great place to store things like playbooks, procedures, and checklists.
00:23:40
Because we now have the knowledge from the crash course, we know how to navigate this.
00:23:48
To facilitate ease of use of our knowledge base, it has to be three things:
00:23:55
easy to find, easy to update, and easy to search.
00:24:05
I know one and three sound similar, so let me explain.
00:24:12
It's essential to pick one place to keep all your knowledge base articles.
00:24:22
Otherwise, you’ll run into bottlenecks.
00:24:30
You’ll still get people asking, 'Hey, do we keep it in Dropbox or Google Drive?'
00:24:37
We don’t want that.
00:24:42
Pick one place; it doesn’t matter which. Just choose, and stick with it.
00:24:49
Next, you have to make it easy to update.
00:24:57
Give your team write access to these articles and get out of their way.
00:25:05
The people who are using the articles should be the ones who have access to edit them.
00:25:12
You don’t want to create hurdles that slow them down.
00:25:19
Last, if you have reduced the number of places your knowledge base resides in, you want to ensure that location allows searching.
00:25:26
I don’t know what tool wouldn't have that ability unless it's a physical filing cabinet!
00:25:35
So long as you can search through that information, how you organize the articles doesn’t matter.
00:25:43
I absolutely love organizing, so I totally get it!
00:25:48
It’s a weird hobby of mine.
00:25:57
When I say you don't need to organize your articles, I really mean it.
00:26:05
You can always go back and reorganize them if you want!
00:26:15
Now that we understand the two pieces of addressing tribal knowledge, let's look at what this looks like in real life.
00:26:23
For example, the Heroku support team started to adopt this model unknowingly, definitely with my influence.
00:26:31
When we onboard new people, we have them read three articles; remember, the crash course should be short.
00:26:39
The articles include the Heroku Academy, which talks about the basics of the platform.
00:26:48
The features, pipelines, review apps, private spaces, that kind of thing.
00:26:54
We also have other articles more specific to support engineers, covering our tools and CLI commands.
00:27:02
This is a crash course that gets them started as support engineers.
00:27:09
Then we have our knowledge base, which consists of the help site articles.
00:27:18
We have them categorized, which is for your benefit.
00:27:25
It’s easier to open tickets that way, but you can just search for what you're looking for.
00:27:35
Searchability is key. We have both public-facing articles and internal ones.
00:27:43
Now, you might be wondering, wait a moment, Heroku has a Dev Center. What’s the point?
00:27:52
In Heroku, we realized that more critical answers didn’t fit in the Dev Center.
00:28:02
For example, we noticed a lot of applications using Unicorn as their web server experienced inexplicable request timeouts—specifically, H12.
00:28:14
This could only happen on Heroku after 30 seconds.
00:28:20
Why was this happening? The Unicorn can be a bit harsh.
00:28:29
What would happen is you would have these slow web workers.
00:28:36
Unicorn sends a 'SIGKILL' instead of allowing the process to gracefully shut down.
00:28:42
With fewer workers, requests would pile up, causing performance issues.
00:28:50
So, this is a Unicorn issue technically, but you wouldn’t see H12 anywhere else.
00:28:58
We decided to create a knowledge base article for it.
00:29:07
This article is an example of how we work.
00:29:16
We have many articles that help support our customers, which is fantastic.
00:29:25
Why do we have so many articles? Because all of our engineers at Heroku can create these articles.
00:29:37
This happens when we notice something important that's not documented.
00:29:46
It’s a natural instinct; you need to build that up in your team.
00:29:55
Having that has been incredibly helpful.”
00:30:05
Now, does this mean that Heroku’s onboarding process is seamless and totally hands-off?
00:30:11
Absolutely not! It definitely helps, but the knowledge base is crucial.
00:30:19
Let’s talk about how we keep these docs up to date.
00:30:29
You’ve created your crash course and your knowledge base. How do you ensure they stay up to date?
00:30:41
First, I’m going to sound like a broken record, but make them easy to update.
00:30:52
Again, give people write access and get out of their way.
00:30:58
The second thing is a really actionable tip: add expiration dates to all of your articles.
00:31:06
At Heroku, we mark articles due for review every three months.
00:31:17
When we see that notification next time we visit an article, we check to see if the information is still true.
00:31:27
You can certainly do this manually if you prefer.
00:31:34
Just write the expiration date at the top of your article.
00:31:43
Don't just put the last updated date. That doesn’t count!
00:31:53
Imagine buying milk, and on the carton, they put when the cow was milked.
00:32:01
What does that tell you about whether it’s still good to drink?
00:32:10
So, pick an arbitrary date: a week, a month, or a year from now.
00:32:18
And then stick with it!
00:32:26
Now, how do we make sure these docs are actually useful?
00:32:35
You can write them once, but they may not be perfect.
00:32:41
Let’s say you're the new kid—and you're reading through the docs.
00:32:49
If there's something you don’t understand, that’s important!
00:32:58
Take note of it and improve that documentation.
00:33:06
As a new person, you have to feel empowered.
00:33:13
If you’re a manager, help your team feel empowered to make these changes.
00:33:20
Their fresh eyes are valuable.
00:33:29
Don’t let feelings of being new get in the way of improving documentation.
00:33:40
One of the challenges at a place like Heroku is learning when it’s okay to say, 'I don’t understand this.'
00:33:48
Spoiler alert: it’s always okay to say that.
00:33:56
It takes a bit of bravery, though.
00:34:05
You know the impostor syndrome tells us, 'You’re stupid for not understanding.'
00:34:13
Some people suck at explaining things.
00:34:23
We need to hold them accountable for clarity.
00:34:30
When I ask for a specific explanation, I understand the context.
00:34:37
I make others clarify because I don't want to be confused.
00:34:45
There is something that I want to address.
00:34:54
I know there are folks who have worked hard to get where they are.
00:35:06
They can feel protective of their knowledge—like they worked hard for it.
00:35:14
But I hope it’s obvious the answer is an emphatic yes!
00:35:23
They will still be valued if they share their knowledge.
00:35:28
What you’ve learned is not something that can be easily conveyed in articles.
00:35:37
Getting good at what we do is hard and it's worth valuing.
00:35:46
As I mentioned, I spend a lot of time thinking about how people absorb knowledge.
00:35:54
If we have time and resources, we could create a whole academy for onboarding.
00:36:01
But we don’t have that; this is work!
00:36:07
How do you provide time to educate people with what they need to know?
00:36:16
You do that by teaching them the basics and giving them the resources.
00:36:23
That is how you address tribal knowledge.
00:36:30
Thank you guys so much!
