00:00:15
Hi everyone! Are you guys enjoying the conference so far? Woo! So am I, it’s great!
00:00:23
So, what? You said my name is Annie Sexton, and I spend a lot of time thinking
00:00:31
about how people absorb knowledge. That’s why I’m really excited to talk to you guys about tribal knowledge, why it’s so dangerous, and how to address it.
00:00:41
By learning to address tribal knowledge, you can scale your team quickly and help them become self-sufficient, which I think is the most important thing.
00:00:54
But before we get into that, I know she gave me a bit of an intro, but I’ll probably say the same things once more.
00:01:06
I'll tell you a little bit about myself. So, she mentioned I was a digital nomad.
00:01:11
I’m a retired digital nomad; I just bought my first place.
00:01:17
But I spent all of last year living out of a suitcase and globe-trotting; it was a lot of fun.
00:01:28
However, I got sick of airports really quickly.
00:01:36
So, I’m also a volunteer English teacher at a non-profit in my hometown of Austin, Texas.
00:01:41
I really enjoy teaching, and as she said, I’m a full-stack developer and have been for about five years.
00:01:50
I worked at lots of different web agencies and spent a lot of time on my own Rails applications.
00:01:55
But, of course, this was all before I was hired by the man. And by the man, I mean Salesforce, and by Salesforce, I mean Heroku.
00:02:06
And it is great! Totally candid photo here, definitely not staged. It’s an awesome place to work.
00:02:17
My coworkers are super smart and super friendly. I get to work on my dream platform, and Salesforce, our parent company, is really awesome.
00:02:33
They are like, 'Hey, what’s up for Salesforce? We have this guy,' so it’s great!
00:02:41
Overall, I really, really like work; it’s the best job I’ve had.
00:02:50
But there was some room for improvement in the onboarding process.
00:02:57
Heroku is a bit of a complicated platform; easy to use, right? We pride ourselves on making it easy to use.
00:03:05
But you know, there are a lot of things going on behind the scenes.
00:03:10
There was a lot of information I needed to know to ramp up for my job.
00:03:21
Let me tell you guys what I do at Heroku.
00:03:27
I'm what we call a Support Engineer, which sounds as vague as Solutions Architect and Success Developer.
00:03:35
But all that means is that I'm on the support team.
00:03:41
We are called engineers because we have to know what the customers are talking about.
00:03:55
We don’t have a tiered system when it comes to support; instead, we break it up into core support.
00:04:05
These are basically the generalists, or the first line of defense whenever tickets come in.
00:04:13
If we need specialized help, we can get someone with specific knowledge about Kafka, Postgres, or Heroku Connect.
00:04:21
If any of you caught Caleb Thompson's talk earlier, he is our resident Heroku Connect expert.
00:04:36
So, they hide him in the hall, but I am on the core support team.
00:04:49
As a generalist, it’s our job to be the power users of Heroku.
00:05:03
We basically have to be better at Heroku than everyone else in this room combined.
00:05:12
There’s a lot to know; there are a lot of moving parts, and it can be a little difficult.
00:05:23
And boy, howdy, lately the ticket queue has been getting out of control.
00:05:33
Which is why Heroku is hiring all of you RubyComers! Congratulations, you’re all hired!
00:05:42
That’s right, you get a job, and you get a job! And please don't sue us, this is just a joke.
00:05:51
So just to get you started as a new support engineer, we’ll need to make sure you’re all familiar with the platform.
00:06:02
I’m just going to give you a quick download of what I know about the platform.
00:06:07
So take notes! Heroku lets you deploy, run, and manage applications written in Ruby, Java, Python, Clojure, Scala, Go, and PHP.
00:06:18
These languages are provided by our build packs, but if you need further dependencies, you can use third-party build packs or even Docker.
00:06:31
If you'd like to use your own custom domain, you can do so by running 'heroku domains:add yourdomain.com' and creating a CNAME record.
00:06:43
Or for the alias, an A Name record for an ISA root or apex.
00:06:50
I feel like that’s not the best way of conveying this information. Maybe there’s a better way.
00:07:01
For the record, that’s not what Heroku onboarding looks like, thank goodness.
00:07:13
But let's be honest, there’s just too much about the Heroku platform for me to convey that information in a timely manner.
00:07:20
That would be a poor use of your time and mine, wouldn’t you agree?
00:07:35
Luckily, we don’t have to do it that way; we have the Dev Center.
00:07:43
How many people here have read articles in the Dev Center?
00:07:51
Thank you! So, we’ve got it all documented. We don’t have to convey it verbally, thank goodness.
00:07:57
But when I was being onboarded, there were things about the platform that were not in the Dev Center.
00:08:06
Information about the architecture, how things were interconnected, what were its limits, what were its edge cases.
00:08:14
Some of this was documented, but it was all over the place.
00:08:28
On top of that, I needed to learn so much more just to do my job as a support person.
00:08:39
The internal tools we use for debugging customers’ apps, how to use Splunk, and all these CLI commands I had never used before.
00:08:51
We also have our own support system because we are all developers. We decided to just build the system ourselves.
00:09:05
There was so much to learn.
00:09:17
So, there was a lot of tribal knowledge I had to absorb.
00:09:26
What is tribal knowledge? Let’s define it.
00:09:35
Tribal knowledge is that information that exists in the brains of your coworkers, your tribe, and nowhere else.
00:09:45
And this is no bueno! It’s dangerous.
00:09:54
I’ll give you an example: how many of you can relate to this?
00:10:02
You’re headphones on, you’re in the zone, you’re super focused, and then suddenly something distracts you.
00:10:11
Hey, hey, sorry to bother you, but I have a quick question: how do I dot, dot, dot?
00:10:21
You’re the senior experienced developer who reluctantly removes your headphones.
00:10:29
Concentration is broken; that sucks, right?
00:10:39
But let’s say the roles are reversed.
00:10:47
You’re the new kid, just trying to get people to like you, and you’re working hard.
00:10:55
Yet the answers you’re looking for aren’t documented anywhere.
00:11:08
What do you do?
00:11:13
If you're in that situation, guess what, you get to be the one to wake the sleeping dragon.
00:11:22
That’s always fun, right?
00:11:28
First of all, stop hiring dragons and be nice to your new people!
00:11:37
Tribal knowledge sucks for everyone.
00:11:46
Especially in a growing business, it is unfortunately common for all of this important information to reside in the brains of just one or two people.
00:11:55
Sometimes this is harmless, but sometimes it’s not.
00:12:04
For example, my website is down; I can't reach my developer, and I don't have access to the app on Heroku.
00:12:18
Let me start here by saying that credentials should never be tribal knowledge.
00:12:29
Unfortunately, this happens. We have to tell these well-meaning business owners: 'I’m sorry, your email isn’t listed as an owner or collaborator. I can’t provide you any details about the app.'
00:12:39
They will have to reach out to their developer, which is a frustrating thing to hear.
00:12:51
Tribal knowledge is dangerous for two reasons.
00:13:02
Number one, it creates bottlenecks for both senior and junior people.
00:13:09
It also creates liabilities, because when not if, when those people leave with that information, you’re screwed.
00:13:17
Now, some of you might be thinking, 'Listen, Annie, I don’t know how they do it at Heroku, but we do document our stuff!'
00:13:26
Okay, it’s just that these other people don’t know how to look things up themselves.
00:13:32
That’s fair; I totally get it.
00:13:40
Where are these articles kept? Oh, you know, Google Docs, Markdown files, GitHub issues, Trello cards, emails, Google Hangouts threads, Quip.
00:13:46
That’s a lot of places to store stuff!
00:13:52
And maybe it’s a little hard to find the information, but listen, I get it. Occasionally people are just lazy!
00:14:04
And that’s worth acknowledging.
00:14:16
The first thing that needs to happen is you need to set boundaries.
00:14:23
Without boundaries, you can't have a self-sufficient team.
00:14:34
Train your clients, coworkers, and managers to work around your schedule.
00:14:41
This is really, really important.
00:14:49
If you have some of this tribal knowledge documented, maybe people aren't able to find it.
00:14:55
It’s written down, but no one knows where to look.
00:15:04
It’s like the ELA de Muerta, a location that cannot be found.
00:15:11
This is fine if we’re talking about cursed treasure, but we’re not.
00:15:19
So what do you do? There are two main issues: there's people not knowing what is documented.
00:15:27
Then, there’s people not knowing where it’s documented.
00:15:34
These seem like really solvable issues, yet they are very common.
00:15:43
That’s why I want to go in-depth with you about how to address these problems and why they need to be addressed in this way.
00:15:55
Let’s assume this is a worst-case scenario: you have nothing documented. What do you do?
00:16:03
Well, since you can’t just give away the information, how do you transfer it?
00:16:11
By show of hands, how many of you, when you get a new appliance or a new car, read the entire user manual?
00:16:19
Okay, you guys are the outliers!
00:16:28
Personally, I like to see how far I can get without the manual.
00:16:37
I like to live on the wild side. Because let’s face it, most manuals suck!
00:16:46
They really do!
00:16:50
So instead, we need two things to ensure that the tribal knowledge we’re documenting is actually digestible.
00:16:58
We can't just plug someone in matrix-style and say, 'You know kung fu.' So what do we need? Instead, we need a crash course and a knowledge base.
00:17:09
If there's anything you remember from my talk, it’s these two things: a crash course and a knowledge base.
00:17:17
Now, in their simplest terms, these two things are just two different sets of documentation.
00:17:25
They can take any form; they could be webinars or videos, but they don't have to be.
00:17:34
It’s easier for instruction's sake to talk about them as two different sets of articles.
00:17:43
So, let’s talk about the first one: the crash course.
00:17:50
Now, I’m calling it a crash course, but you could easily call it a getting-started guide.
00:17:57
The point of this isn’t to document everything you need to know for a particular job.
00:18:06
It’s just enough information to get someone started.
00:18:12
So they can then self-educate and look up the answers on their own.
00:18:19
It’s not everything; it’s just a primer. It’s just the basics.
00:18:28
Now, who kind of sucks at explaining the basics? Engineers and developers!
00:18:36
I know who you are; listen, I’m totally generalizing, so take that with a grain of salt.
00:18:44
But the point is, we’re regularly criticized as a group that we can’t talk in layman’s terms.
00:18:52
They make TV shows about how poorly we explain things.
00:19:00
Terrible, awful shows I might add!
00:19:08
The problem is that we explain highly technical terms in highly technical terms.
00:19:16
And that doesn’t help.
00:19:24
Let me give you an example: I once asked a buddy of mine, who is an expert in Postgres,
00:19:30
'What’s the difference between truncate and delete in SQL?'
00:19:40
And this is what he replied: 'Truncate is more like a DDL (data definition language) statement than delete, which is a DML (data manipulation language) statement.'
00:19:54
Good? Yes? Okay, what now?
00:20:02
So, truncate is a fixed cost operation versus delete, which is closer to a parenthesis where n is the number of rows affected.
00:20:11
Okay, we’re getting closer, but I still don’t know.
00:20:19
Truncate is like, 'Give me a fresh table! Boom!'
00:20:29
That is what he should have started with!
00:20:36
And later he added, 'Oh yeah, truncate is like the big guns!'
00:20:44
But you see what I mean? We often go into describing the molecular composition of the bark, instead of the forest.
00:20:53
We need to be able to step back.
00:21:02
And by the way, I’m not throwing my friend under the bus here; he’s usually quite good at explaining things.
00:21:10
I just thought that particular example was great.
00:21:18
So, this is something we have to be aware of when we’re writing our crash course.
00:21:26
We need to be conscious of the terms we’re using.
00:21:35
The biggest thing about the crash course is it has to be digestible.
00:21:42
This even applies to non-technical things. For instance, if I told you guys that you can update your v2 mom through the Aloha portal and org 62,
00:21:51
you’d be like, 'What did you just say?' Unless you're a Salesforce employee, that means nothing to you!
00:22:00
And that’s normal; you have to think about this when you're writing your crash course.
00:22:10
You got to think of several questions: Who am I writing this for?
00:22:18
What is their background? What terms do I need to explain?
00:22:25
And what am I assuming they already know?
00:22:31
For example, do they know what the Aloha portal is?
00:22:38
So, the last thing I’ll say about the crash course is that it should be a crash course.
00:22:45
It’s not like a whole Udemy course that you take over several months; it needs to be short.
00:22:56
It should cover the material without going into too much detail.
00:23:05
Let’s talk about the knowledge base, which is the second part.
00:23:11
With the knowledge base, we were concerned with making the crash course not too long.
00:23:24
The knowledge base is where you put everything else; it's an expansive repository of articles that people can refer to.
00:23:33
This is a great place to store things like playbooks, procedures, and checklists.
00:23:40
Because we now have the knowledge from the crash course, we know how to navigate this.
00:23:48
To facilitate ease of use of our knowledge base, it has to be three things:
00:23:55
easy to find, easy to update, and easy to search.
00:24:05
I know one and three sound similar, so let me explain.
00:24:12
It's essential to pick one place to keep all your knowledge base articles.
00:24:22
Otherwise, you’ll run into bottlenecks.
00:24:30
You’ll still get people asking, 'Hey, do we keep it in Dropbox or Google Drive?'
00:24:37
We don’t want that.
00:24:42
Pick one place; it doesn’t matter which. Just choose, and stick with it.
00:24:49
Next, you have to make it easy to update.
00:24:57
Give your team write access to these articles and get out of their way.
00:25:05
The people who are using the articles should be the ones who have access to edit them.
00:25:12
You don’t want to create hurdles that slow them down.
00:25:19
Last, if you have reduced the number of places your knowledge base resides in, you want to ensure that location allows searching.
00:25:26
I don’t know what tool wouldn't have that ability unless it's a physical filing cabinet!
00:25:35
So long as you can search through that information, how you organize the articles doesn’t matter.
00:25:43
I absolutely love organizing, so I totally get it!
00:25:48
It’s a weird hobby of mine.
00:25:57
When I say you don't need to organize your articles, I really mean it.
00:26:05
You can always go back and reorganize them if you want!
00:26:15
Now that we understand the two pieces of addressing tribal knowledge, let's look at what this looks like in real life.
00:26:23
For example, the Heroku support team started to adopt this model unknowingly, definitely with my influence.
00:26:31
When we onboard new people, we have them read three articles; remember, the crash course should be short.
00:26:39
The articles include the Heroku Academy, which talks about the basics of the platform.
00:26:48
The features, pipelines, review apps, private spaces, that kind of thing.
00:26:54
We also have other articles more specific to support engineers, covering our tools and CLI commands.
00:27:02
This is a crash course that gets them started as support engineers.
00:27:09
Then we have our knowledge base, which consists of the help site articles.
00:27:18
We have them categorized, which is for your benefit.
00:27:25
It’s easier to open tickets that way, but you can just search for what you're looking for.
00:27:35
Searchability is key. We have both public-facing articles and internal ones.
00:27:43
Now, you might be wondering, wait a moment, Heroku has a Dev Center. What’s the point?
00:27:52
In Heroku, we realized that more critical answers didn’t fit in the Dev Center.
00:28:02
For example, we noticed a lot of applications using Unicorn as their web server experienced inexplicable request timeouts—specifically, H12.
00:28:14
This could only happen on Heroku after 30 seconds.
00:28:20
Why was this happening? The Unicorn can be a bit harsh.
00:28:29
What would happen is you would have these slow web workers.
00:28:36
Unicorn sends a 'SIGKILL' instead of allowing the process to gracefully shut down.
00:28:42
With fewer workers, requests would pile up, causing performance issues.
00:28:50
So, this is a Unicorn issue technically, but you wouldn’t see H12 anywhere else.
00:28:58
We decided to create a knowledge base article for it.
00:29:07
This article is an example of how we work.
00:29:16
We have many articles that help support our customers, which is fantastic.
00:29:25
Why do we have so many articles? Because all of our engineers at Heroku can create these articles.
00:29:37
This happens when we notice something important that's not documented.
00:29:46
It’s a natural instinct; you need to build that up in your team.
00:29:55
Having that has been incredibly helpful.”
00:30:05
Now, does this mean that Heroku’s onboarding process is seamless and totally hands-off?
00:30:11
Absolutely not! It definitely helps, but the knowledge base is crucial.
00:30:19
Let’s talk about how we keep these docs up to date.
00:30:29
You’ve created your crash course and your knowledge base. How do you ensure they stay up to date?
00:30:41
First, I’m going to sound like a broken record, but make them easy to update.
00:30:52
Again, give people write access and get out of their way.
00:30:58
The second thing is a really actionable tip: add expiration dates to all of your articles.
00:31:06
At Heroku, we mark articles due for review every three months.
00:31:17
When we see that notification next time we visit an article, we check to see if the information is still true.
00:31:27
You can certainly do this manually if you prefer.
00:31:34
Just write the expiration date at the top of your article.
00:31:43
Don't just put the last updated date. That doesn’t count!
00:31:53
Imagine buying milk, and on the carton, they put when the cow was milked.
00:32:01
What does that tell you about whether it’s still good to drink?
00:32:10
So, pick an arbitrary date: a week, a month, or a year from now.
00:32:18
And then stick with it!
00:32:26
Now, how do we make sure these docs are actually useful?
00:32:35
You can write them once, but they may not be perfect.
00:32:41
Let’s say you're the new kid—and you're reading through the docs.
00:32:49
If there's something you don’t understand, that’s important!
00:32:58
Take note of it and improve that documentation.
00:33:06
As a new person, you have to feel empowered.
00:33:13
If you’re a manager, help your team feel empowered to make these changes.
00:33:20
Their fresh eyes are valuable.
00:33:29
Don’t let feelings of being new get in the way of improving documentation.
00:33:40
One of the challenges at a place like Heroku is learning when it’s okay to say, 'I don’t understand this.'
00:33:48
Spoiler alert: it’s always okay to say that.
00:33:56
It takes a bit of bravery, though.
00:34:05
You know the impostor syndrome tells us, 'You’re stupid for not understanding.'
00:34:13
Some people suck at explaining things.
00:34:23
We need to hold them accountable for clarity.
00:34:30
When I ask for a specific explanation, I understand the context.
00:34:37
I make others clarify because I don't want to be confused.
00:34:45
There is something that I want to address.
00:34:54
I know there are folks who have worked hard to get where they are.
00:35:06
They can feel protective of their knowledge—like they worked hard for it.
00:35:14
But I hope it’s obvious the answer is an emphatic yes!
00:35:23
They will still be valued if they share their knowledge.
00:35:28
What you’ve learned is not something that can be easily conveyed in articles.
00:35:37
Getting good at what we do is hard and it's worth valuing.
00:35:46
As I mentioned, I spend a lot of time thinking about how people absorb knowledge.
00:35:54
If we have time and resources, we could create a whole academy for onboarding.
00:36:01
But we don’t have that; this is work!
00:36:07
How do you provide time to educate people with what they need to know?
00:36:16
You do that by teaching them the basics and giving them the resources.
00:36:23
That is how you address tribal knowledge.
00:36:30
Thank you guys so much!