00:00:11.679
All right, next up we have Brian Childress. Brian is an accomplished technologist who has spent time building and scaling software in many highly regulated environments. He holds multiple patents for software design, speaks internationally, and is a sought-after thought leader. Brian resides in Virginia and loves to travel and explore, often bringing his family along for the adventure. He will be giving a talk on 'Simplicity: The Key to Software Success.'
00:00:45.000
Good morning! I’m excited to be back in Asheville. I was here late last year with my family, doing what was probably the most touristy thing possible: we went to see Christmas lights at the Biltmore. It was absolutely beautiful! If you haven't been, I highly recommend it. I'm excited to be here because simplicity, the key to software success, is something that is very near and dear to my heart.
00:01:11.840
So, who am I? My name is Brian Childress. I'm from Richmond, Virginia, and I have been working in software for many years. I also serve as a technical advisor for a number of different organizations, including startups, small and medium businesses. A lot of what I'll be sharing with you today comes from my experience in this role.
00:01:36.880
To get us started, I like to have more interactive sessions. I hope we have enough coffee flowing through our veins now! Since we are all technologists, I'd like for us to build something together. I'm going to share a couple of options, and I just want you to raise your hands, hoot, holler—whatever feels appropriate for you—for the one that makes the most sense.
00:02:06.880
First, let's decide on an architecture: monolith versus microservices. Where are my monolithic fans? Yeah, all right! Any microservice folks here? Right, we have to build this thing to scale, so let’s go ahead and build it right the first time. You're in good company.
00:02:17.959
I feel like this is a gimme, but I'll give you the option anyway: Ruby or JavaScript? I have seen some Node.js folks out there. Ah, a delayed hand! All right, for databases, we’ve got to store data somewhere. How about relational databases? PostgreSQL? There you go, that’s my man! Non-relational databases? Oh wow, that’s the first time I’ve ever had none!
00:02:52.239
For our non-relational data, we need to get it out of the database somehow. How about REST? Plain old REST for the win! It works every time, most of the time. Anyone using GraphQL? Anyone excited about using GraphQL or already using it in production? A few folks, very cool! Thank you, everyone! We just built an amazing application.
00:03:14.760
However, as we went through that, I'm curious if any of you were asking, 'Why?' or 'What is this for?' Or my favorite answer to every technical question I'm asked: 'Well, it kind of depends.' As we were selecting the technologies, we were picking the ones we were most interested in, familiar with, and excited about. I want to suggest that we just built the world's most highly scalable, globally distributed to-do list application.
00:03:44.040
Some of the decisions we make may not be the right fit for the applications or the software we are building. At the end of the day, software is just a tool we use to solve problems. For many of us, we focus so much on the technology and the software that we forget about the problems we are actually paid to solve.
00:04:06.159
According to my extensive Google research earlier this morning, about 70% of software projects fail—one in seven projects fail! Has anyone ever worked on a software application that never made it to production? See, this is a safe space, we can all be true. Seven out of ten—that's disheartening! As engineers, we pour blood, sweat, and tears into the software we're building, only to see it never get into production or, even worse, see it get into production only to be quickly sunset.
00:04:27.280
There are a few reasons I found that many software projects fail. Often, it's due to unclear requirements. Businesses may have an idea that they want to solve—the next big problem or build the next great thing—but they aren't clear about what that means or what the end goal is.
00:04:58.240
We could have misaligned goals—not just unclear requirements but the goals themselves. As technologists, we want to play with the latest and greatest technology because that’s what we can control, but we’re not aligned with what the business needs.
00:05:03.120
Finally, one of my favorite reasons is that technology is just too darn complex. Many times, this is of our own making. We bring in technology that we don't need, which increases the complexity and makes it more difficult, leading to software project failures. One of my favorite software architects is Neil Ford. Has anyone ever run across Neil Ford? He has a friend named Mark Richards. I encourage you to look them up, they have some great books. One of my favorite quotes from Neil is: 'Developers are drawn to complexity like moths to a flame, often with the same result.'
00:05:54.440
Technology itself is already complex, so why do we as technologists encourage or get excited about making it even more complex? It's often because there are new technologies out there, new frameworks, and new libraries. We want to use them, and when we can’t utilize them in our side projects, we bring them into the organizations we work with. Who here is guilty of this? Who has brought in new technologies because they were bored?
00:06:31.840
Yeah, tired of building the same CRUD application, talking to the same database, that has the same database tables. We get bored and we bring in new technologies. Again, I mentioned that we are in a safe space. I want to ensure everyone here is honest: who here has ever suggested or brought in a technology because they thought it would look good on their resume?
00:07:11.759
I’m guilty, and I still feel bad for those organizations dealing with that technology years later because I wanted to put that technology on my resume, so I could say that I put it into production as I was on my way out to a new organization.
00:08:04.720
These are the things that ultimately hurt the business. When we take our own needs and desires and bring in complexity into the software we build, it ultimately hurts the business. So how do we get around this? How do we stay excited about what we’re working on, use technologies we love, and also meet the business needs? One way I like to do this is the Rule of One.
00:08:36.960
Let me explain: the Rule of One says that for a new project or a new large initiative we might have, we can change one thing. We get to change one framework, one library, one architecture—we're not going to change all of them for the new application. We won’t bring in the leading or bleeding edge technology unless we absolutely need it, as a differentiator for the applications we build. We get to change one thing because as a team, everyone likely has to learn that new one thing.
00:09:18.400
If I change the architecture, the backend runtime, or a database, that's a lot for an entire team to understand. This leads to the potential for a failed project.
00:09:31.200
I want to ask you if you've ever heard the buzzword 'real-time' or been asked to ensure that your application can serve data in real-time. Has anyone ever heard this? As engineers, we love this term 'real-time' because for us, the wheels start turning. We consider all the capabilities that AWS or other platforms can provide to get globally distributed data in near real-time to our customers, wherever they are.
00:10:11.000
But oftentimes, our definition of real-time is completely different from the business's definition. If we, as technologists, don't clarify what the business is actually looking for, we will create a system so complex that it becomes difficult for us to continue to grow, just because we solved the wrong problem.
00:10:40.320
Usually, for most businesses, real-time means they want some updated data on the dashboard when they log in at 8 a.m. with their first cup of coffee. For us, though, we're thinking about millisecond distributed data with eventual consistency.
00:11:01.760
I want to share with you a case study of a group I worked with in 2022. This was an interesting project where a company reached out to me on LinkedIn because they needed to build an application. They had already been working on it for about six months, but things were not going well. The fascinating part was that I had to turn the project around in about three weeks to meet a real, actual deadline.
00:11:60.800
At the time, there were teams spread across six different time zones, and I was the easternmost person. Most of the time when I was starting my day, the team was either ending their day or heading to bed. We faced a significant distribution problem. This project had a genuine deadline to hit, which made it more stressful than usual.
00:12:20.599
In August 2022, development began. Someone had an idea, commissioned the project, and started to spend money on finding resources. I joined in January of 2023, and we had to deliver by February of the same year—about three weeks away.
00:12:44.520
I have to say, when I came in, it looked like someone had gone down the AWS menu and just selected everything that looked cool. There were AWS services like Kinesis, DynamoDB, and tons of machine learning models.
00:13:03.120
To give you a sense of the project, we were ingesting public data, using some NLP machine learning models to determine sentiment, and storing that data in a database for display on a dashboard at the end of the day. We had about eight to twelve high-level users on the platform who wanted information, but we didn’t need that level of complexity.
00:13:32.200
My first step was to come in and assess what the real problem was that we needed to solve. My initial solution involved rearchitecting the application—it was necessary to immediately simplify things.
00:13:52.000
I needed to ensure that we factored in the number of users, the type of data being stored, the timeline, and the skills of the development team. I pulled out about 80% of those technologies.
00:14:08.120
Within just a few days, we had completely rearchitected everything. We were then able to continue to grow and scale, and in a couple of weeks, more developers were working across more time zones, contributing more code to the platform. Ultimately, we hit that February 2023 deadline.
00:14:28.800
Now, let's talk about one of my favorite buzzwords: scalability. I am currently working on a book about this. How many of you have ever been asked to ensure that the systems you are designing and building are scalable? Right? We all have our own definitions of scalability. It can mean various things depending on the context.
00:14:54.920
We hear 'scalable' in a lot of business conversations now—not just in technology, but regarding how we make processes and systems within the organization scalable. I like to think about scalability in three different ways. First, when we think about users, I need to add more users to the system while ensuring that performance remains the same or improves.
00:15:20.079
In my opinion, this is largely a solved problem. We have plenty of examples and different architectures we can draw from to tackle this issue. The other two areas are much more challenging to address. How can we add more features to the applications and software we build? The final area is how can we add more developers to the teams that are building features for our users?
00:15:51.000
So typically, when we think about scaling users, we consider how to add more compute resources. We might need to add things like more database capacity or the ability to process more events as our users become more distributed. This presents an interesting engineering challenge: getting our functionality and data closer to those particular users, thereby improving their performance and experience.
00:16:09.679
I feel this is largely a solved problem as well, but when we think about scaling features, it becomes more complex. We need to ensure that the features we develop align with business needs.
00:16:31.200
What I have often seen happen is that the features we develop and the terminology we use in those features don’t necessarily align with the business. What we call something on the technology side, such as a database table, is completely different from what the business refers to it as. When this occurs, we end up making a translation between the technology and the business.
00:17:07.760
For scaling features, I look toward maintainability and modularity as essential components. I want to share another interesting case study with you. This was with a health tech startup that had raised their Series B funding—in 2023, they raised about $17 million to grow their team. They had a fairly standard stack and a lot of fans here: Rails, PostgreSQL, AWS. It was a straightforward monolithic application.
00:17:30.959
They did well on their infrastructure; they had the necessary components to support the load. The one interesting thing was that it took about six months for a senior Rails developer to be able to confidently put code into the production code base. Six months! Has anyone here ever spent six months onboarding just to get code into production? That’s a long time.
00:18:10.560
As an engineer, it’s pretty disheartening when you can't confidently commit code for six months. I like to see developers being able to spin up their local environments in an hour or an hour and a half, with support from maybe someone else on the team. I want to see people able to commit code to a production code base within a day or a week at most.
00:18:49.200
Six months to be able to put code into the code base was pretty tough for this team, leading to substantial attrition and frustration. Lots of finger-pointing occurred, and it was really unfortunate. My role was to help them understand how we got to this point and how we could navigate back out.
00:19:20.120
What ended up happening was that the product and technical teams weren’t collaborating effectively. The product team was making decisions while the technical team simply implemented them. The codebase reflected this disconnect. Because it was a healthcare application, it was configuration-driven, which led to a situation where almost every user had their own version of the application.
00:19:49.080
This configurability, based on various factors like the state they lived in and their healthcare provider, created numerous issues. This not only impacted the technical side with the ability to onboard developers but also affected customer service.
00:20:07.760
The customer service team expressed frustration because they couldn't replicate users' issues due to the lack of logging and the difficulty of storing configurations. This led to countless issues with customer support, product development, and overall team synergy.
00:20:41.040
This brings me to the third area I think about for scaling: the ability to scale developers. This is often neglected consideration. When new applications start, it might be just a couple of developers; we spin up a GitHub and start working through things as they come. But we don't think about what needs to happen as we bring on new developers.
00:21:20.480
What does it take for a new developer to download the code repository and run a simple set of scripts to set up the application? If we have a complex architecture of multiple microservices or databases, what does that entail? We often say, 'Oh, the documentation will take care of that.' But is it really easy to understand?
00:21:50.560
For me, I have a couple of rules around this. One is that the code should be readable and understandable by a junior engineer. They might not be able to write or architect it in that way, but they should be able to come into a codebase and grasp what the sets of functionality are doing. A new engineer should navigate the codebase easily, understanding where things are placed.
00:22:28.559
If the code is structured well, it should feel as if it was written by one engineer, even if it was a team of ten or a hundred contributing.
00:22:55.680
It takes concerted effort to achieve this cohesiveness, and when the code reflects that unity, it's easier for new developers to navigate. They don't need to grasp all the complexities of both the code and the business; they just need to understand the business problem being solved.
00:23:41.480
Onboarding and technology acceptance are paramount. I've had to rewrite entire applications because we couldn't hire enough qualified developers to work on them. We might pick a highly concurrent technology or runtime, but if no one can work on it, it becomes a business cost.
00:24:10.000
As technologists, it's our job to guide the business in choosing the technologies and architectures that will support long-term success. I want to discuss one more case study with you—this involved an organization with a ten-year history and an offshore development team based in Eastern and Central Europe.
00:24:50.320
They aimed to scale up the number of developers involved and wanted to onboard developers from anywhere in the world, so they could reduce costs from one particular team or bring in special expertise. It should have been a simple task of adding someone to GitHub, but we soon discovered that our documentation wasn't available in a consistent language.
00:25:27.500
In fact, passwords and credentials were stored in plain text, accessible to anyone who had access to the documentation site. This was a considerable security risk that we had to manage before onboarding new developers.
00:25:52.760
What I find interesting about the desire to scale the developer base is that it puts immense pressure on our systems, communication, and documentation. Without those systems in place, everything quickly breaks down as we add more engineers, especially when they're not within close proximity.
00:26:42.320
To wrap this up, I want us to remember that at the end of the day, software is just a tool we use to solve business problems. It's complex enough already, so we shouldn't add additional complexity just because we're bored, we want to pad our résumés, or we're excited about some new technology we've seen at a conference.
00:27:22.320
I always encourage my teams to focus on the problem. How simple can we make that problem to understand across the organization? If anyone says, 'Well, the problem is too complex; you just won't understand it,' I push back. We really need to focus on the simplest way we can describe the problem and go from there.
00:27:44.080
I don't know if we have time, but I'm open to questions if there are any.
00:28:05.520
Yes, in the back? So I was curious—say you're on a development team and a technical advisor comes in, similar to the example you provided. How can those development team members equip themselves with the knowledge to refute a suggestion if they deem it overly complex? What do you recommend?
00:28:45.559
That’s a really good question, and I think it has two parts. Ideally, the advisor should be open to reception of pushback and engagement. However, from the development team perspective, if someone from a big-name tech company comes in with a VP of Engineering title, I can understand developers feeling intimidated about pushing back.
00:29:18.000
But I encourage engineers to focus on simplicity. That is, ask the 'why' question and do it at least five times. If they propose DynamoDB, for instance, ask why five times until you pinpoint the actual purpose it serves—and it should foster a conversation. It’s important for developers to bring their knowledge and experience, and not just say, 'I don't like that.' There should be valid reasoning behind the feedback.
00:30:15.720
Absolutely, we discuss keeping documentation up to date as it's leveraged by everyone in the organization. The first step is determining where we store our documentation. Ideally, that should be closest to where it will be used.
00:30:35.760
If I'm documenting a function, for example, I want to share not just how it's done but why it's done that way. Maybe there was an original bug that led to this solution, or there’s a GitHub issue tied to it. The documentation can be in a codebase's README, or in a shared documentation repository.
00:31:05.560
It's best to have a limited number of places where documentation exists, focused where it will be consumed by specific audiences, be they the technical team or the broader business.