Pre-factoring: Getting it (Closer to) Right the First Time

RubyConf AU 2015

by Coraline Ada Ehmke

In Coraline Ada Ehmke's talk, "Pre-factoring: Getting it (Closer to) Right the First Time," presented at RubyConf AU 2015, she emphasizes the importance of thoughtful software architecture in the development process. She critiques the binary views of software development—pitting structured methodologies like waterfall against agile methods. Ehmke argues that the shift towards Just-in-Time design has overlooked valuable aspects such as comprehensive planning and documentation. She frames architecture as a discipline akin to sculpting, where the aim is to create order from chaos and avoid code entropy.

Key Points:
- Architecture Definition: Architecture encompasses the composition, design, and interactions within a system, similar to a sculptor shaping granite into a statue.
- Compromises in Development: Ehmke stresses that every code writing decision involves compromises, which should be made consciously to avoid unforeseen technical debt in the future.
- Case Study - Panometer: She shares her experience with the Panometer project, illustrating how a lack of architectural foresight led to significant technical debt and inefficiencies, particularly in data handling and read optimization.
- Challenges of Agile Methodologies: While agile allows for rapid iterations, it can lead to issues when multiple developers work independently without a unifying architectural strategy, often resulting in messy, tightly coupled systems.
- Impact of Acronyms: The use of acronyms like DRY (Don't Repeat Yourself) can sometimes detract from code clarity and lead to convoluted implementations that prioritize abstraction over understandable code.
- Future Code Consideration: Ehmke encourages developers to think long-term about the survivability of their code, advocating for extensibility and maintainability as fundamental architectural concerns.

Conclusions and Takeaways:
- Architecture is essential and should not be an afterthought; it requires deliberate planning to prevent legacy code issues.
- Developers should create code that not only meets immediate needs but also sets a strong foundation for future iterations.
- The architectural approach should balance between necessary design and agile development, ensuring that enhancements do not lead to chaotic systems.
- Ultimately, Ehmke posits that thoughtful, disciplined architecture yields code that is functional, elegant, and maintainable, which aligns with creating positive experiences for both developers and end-users.

00:00:00.199 Um, I'm from the friendly part of America. I'd like to get to know my audience a little bit. I've met some of you at some of the functions beforehand, but it would be a big help to me if you could give me your names again, because I have a very bad short-term memory. So on the count of three, one, two, three! I'm never going to remember that, so I'm Coraline Ada Ehmke, at Coraline on Twitter, and my URL is sebastwherecoraline.codes. As I was introduced, I'm in charge of developer happiness at a company called Instructure. My team focuses on refactoring code, culture, and processes. This talk is about things that I wish the developers who came before me had done differently.

00:00:16.600 So I'm going to break some taboos today, and I'm probably going to lose some friends in the process, but I think it'll be worth it to spark some conversation. And yes, drop bears are real. It's not scary at all, though they look very cuddly. My talk is entitled, 'Pre-factoring: Getting it (Closer to) Right the First Time,' but we're really going to have a serious discussion about the architecture.

00:00:30.720 I've been doing development for about 20 years, and when I started, the only methodology we had was waterfall. We all know that waterfall is terrible and we don’t want to go back to that. It’s really hard to add features to a waterfall project; there’s a lot of friction around changing a design. You fail last, and it's really hard to fix things. However, there were some good aspects about waterfall as well. You had a planned roadmap for where your application was going to go over time, good documentation, and design artifacts capturing not only what the code was supposed to do, but why.

00:00:54.399 In moving away from waterfall to what I call Just-in-Time design or Just-in-Time architecture, we've lost some valuable elements along the way. I think the pendulum may have swung too far in the opposite direction, but we don't have to fall into binary thinking about architecture. It's not architecture astronauts versus cowboy coders; that’s a false dichotomy. Big upfront design is not the only alternative, and no design at all is also not a viable option. I’d like to explore how we can be thoughtful without over-planning.

00:01:15.080 This is the formal definition of architecture: the composition, design, and interactions of a system. I like to think of it as creating order from chaos. I envision it like a sculptor who sees a block of granite. Within that block, the sculptor perceives the statue and the image that wants to emerge from it. By applying will, discipline, and creativity, they can bring this image out of raw material. The natural tendency of a chunk of granite is to turn into a boulder, which will then break down into chunks of rock and ultimately gravel.

00:01:32.720 In the same way, the natural tendency of our code is to turn into spaghetti or legacy code that no one wants to touch, sliding into entropy. It requires discipline, will, and creativity to manage that tension. It’s almost Newtonian; when you push on one side, something has to give. Being thoughtful about managing that tension is really important because, whether you realize it or not, when we’re writing code, we’re making compromises. I’d rather make a compromise consciously up front than discover I’ve made an unconscious compromise down the road.

00:01:49.680 Gera said that music is liquid architecture, and architecture is frozen music. Music obviously takes a lot of technical skill, but it’s not just technical skill that makes music beautiful; it’s collaboration, composition, the mixture of melody, rhythm, and talent. I believe our architecture can be beautiful too by applying some of the same thought processes. However, our architecture today tends to be utilitarian and brutalist; it's something we tolerate instead of something that inspires joy.

00:02:06.160 Deliberate architecture isn’t fashionable anymore, but what happens when we disregard architecture altogether? Do we really end up with better software? I want to share a case study, a cautionary tale, if you will. The last project I worked on was called Panometer, which was basically a replacement at my company for Code Climate and Metric F. We did it in several phases.

00:02:24.000 We started out with design comps, looking at the charts and graphs that would answer the questions about code quality we needed answers to. We spent a lot of time iterating on the design—not only of the charts and graphs but also the type of data we were going to represent and how the user would interact with it. Our second sprint focused on a series of command-line analysis tools, including Society, which is a really cool gem I recommend checking out. It detects and maps out coupling between classes, both afferent and efferent coupling. We also used Fuku Zatsu, a non-opinionated measure of code complexity, along with Reek and some other tools.

00:02:56.080 Our command-line tools comprised our second phase, while the third phase involved something called Panda Pounce. Panda Pounce was essentially a job runner that you could plug metrics tools into. It would execute them and gather output, allowing us to chart metrics over time. The client-side implementation came next, where we had two teams working on the front end. A considerable amount of work was done on the front end.

00:03:18.200 When we began to implement the front end, we served fixture data through Rails controllers via a JSON API to a JavaScript MVC that then utilized D3 for visualizations. All good. Next was the Panometer API, which provided a means for all the command-line tools' data to flow into Panometer and be stored in a database. In Sprint six, we decided to tie it all together. The Panda Pounce generated JSON data which went into the database and was displayed through Panometer.

00:03:43.760 But the request-response cycle got painfully slow. We had queries like this—it had recursive queries finding the ancestors to a commit, and that’s really easy to test. Additionally, we had everything stored in a generic measurement model that contained raw JSON in the database. It was so large that I couldn’t fit it on one screen.

00:04:01.040 We ended up with records containing 200,000 to 202,000 characters of JSON. I looked at that, and I couldn't understand it or read it. It was not something I was parsing with Ruby. This should have alerted us during Sprint seven about the data model.

00:04:22.480 We optimized for writes when a reporting system should be optimized for reads. This led to frustration and a lot of rework, demonstrating how hard it is to take something apart once it’s all been glued together. We spent so much time iterating on the front end that the overall design of the system fell by the wayside, making us do a lot of rework.

00:04:38.760 I don’t think that my team’s experience is unique. We all, to some degree, fail in similar ways. We practice canonical development, doing Rails by the numbers, hurrying to produce an MVP, and building our houses on shaky foundations while spending a fortune on expensive paint. We treat a backlog like a stack, pop a feature off of it, implement it, and never really look down to see what’s coming next. There are many hands on the code, but no overarching strategy for how we’re actually going to implement it.

00:05:02.960 We often spend more time optimizing stylesheets than we do our data models. This doesn't mean we shouldn't optimize stylesheets; rather, we need a better balance between frontend and backend work. As we increasingly use JavaScript MVCs, we face complexities that allow for mistakes in two different places and two different languages.

00:05:21.360 There has been talk about service-oriented architectures today, and they won’t save you if you can’t construct a proper data model. An SOA is just going to break everything into ten pieces, adding latency on top, and you'll end up with the same problems in ten different locations. Agile, in theory, means we can determine the success or failure of an idea quickly, but this theory falls apart when working in a large system with many moving parts and people working in independent vertical slices.

00:05:35.920 We work in small iterations, hoping everything comes together perfectly in the end, but unfortunately, that rarely happens. Why do we do things this way? Much of our development is guided by fear—fear of premature optimization. What’s the opposite of premature optimization? In my experience, it’s no optimization at all.

00:05:55.520 If you don’t make time for optimization as part of your process, you won’t get to it later. As Gary Burnham said, 'We'll cross that bridge when it's burning beneath us.' Fear of overdesigning can lead us to not design at all. Fear of gold-plating also prevails; nobody wants to be accused of gold-plating, but an engineer focusing on the structural integrity of a bridge is not the same as gilding.

00:06:13.320 Focusing on the structural integrity of your application is not gold-plating; it’s planning for extensibility and maintainability. Sending M said that opinions about what code looks like don’t translate to concrete advice. While we have many patterns for refactoring methods in classes, we lack good patterns for architectural structures for object models or applications.

00:06:29.960 This drives me crazy; every text editor has a dropdown menu that says 'File New,' yet we’re terrified of it. We’d rather open an existing class and add more methods to it than create a new one. We should ask ourselves why this is the case. The hardest part of all, even harder than creating a new file, is deleting one.

00:06:47.120 We need to ask why our classes want to attract as many methods and dependencies as possible, motivated by the desire for importance—they aspire to godhood. It’s up to us to tame these classes, curate them, and create good little organisms within a healthier code ecosystem. This doesn’t happen accidentally.

00:07:02.720 A lot of talks over the past year have focused on monoliths and how to fix them. With Just-in-Time architecture, our code tends to form monoliths without a planning design. You will begin bolting on features and patching with duct tape, increasing complexity and coupling, ultimately building tomorrow’s monolith. Doc Norton quoted Steve McConnell stating, 'The problem with quick and dirty is that the dirty remains after the quick has been forgotten.' We make compromises along the way, which undermines our overall effort.

00:07:21.279 Fear of missing deadlines leads us to accept deadlines that do not allow us to write the code we desire. What if we stopped treating deadlines as constraints and began treating them as milestones? What would happen to our code's quality? Without time allotted to get it right the first time, we inevitably end up doing it wrong twice, and that’s not how I want to write code.

00:07:41.679 Imagine building a race car and deciding to focus on speed at the end of the project—if and when it becomes a problem. Trust me, that approach does not work. The example I want to share is the output from Society, showing class coupling in a mature four-year-old Rails application. This is where all our code is heading unless we're deliberate about its design.

00:08:01.360 Acronyms can be friends or foes. As developers, we love acronyms—they are cognitive shortcuts that make us feel safe and distance us from decision-making, reducing our risk of being wrong. However, at what cost? 'DRY' is one of the most overused acronyms I’ve seen. Developers jump through hoops using yield blocks to avoid copying a few lines of code, resulting in complex methods that lose readability.

00:08:20.920 Readability should be the ultimate goal of code. I write code for humans first and machines second. Thus, DRY is not about not repeating code; it’s about not repeating an idea or a semantic entity—it’s about clarity. 'YAGNI' is not about managing code; it's about managing yourself, avoiding the paradox of infinite choice. We can’t predict the future, but we can’t afford to pretend the future will never come.

00:08:42.960 There’s nothing wrong with being prepared. Don’t code as if today is the only day your code will be in production; it will outlive us all. I want to propose a couple of new acronyms for us: 'GEMINI'—Just Might Need It. Leave some doors open in your design; don't shut them all. Intuition is great; I use it as I code but tend to remember when I’m right while forgetting the times I’m wrong.

00:09:02.400 The other new acronym is 'YAGI'—You Ain’t Gonna Get To It. Temporary code is often the longest-lived code of all. If you don’t believe me, search your codebase for 'fix me' or 'to do.' If refactoring is not baked into your process and you’re not doing it as you go, YAGI becomes essential. So, what can we do to ensure a healthy, extensible architecture without falling into the trap of overdesigning?

00:09:20.280 Realize that today’s quick fix is tomorrow's legacy code. The word 'legacy' outside of programming is a gift from one generation to another. Think about those who will come after you. Do what you can to make their lives easier and avoid being on the other side of a 'git blame.' Consider the code you write today as a gift to your future self and keep the big picture in mind.

00:09:39.680 Fixing a method is easy, but fixing a design is hard. While we have well-documented strategies for refactoring methods or extracting classes, we lack solid patterns for fundamental aspects of our architecture. This lack of patterns means we must give ourselves room to fail, create opportunities for experimentation, and learn about the foundational building blocks of our applications.

00:09:55.560 I often hear developers say naming things is hard. Naming things is not hard if you understand the object’s purpose, if you know where it fits into your application’s narrative. It’s not about naming; it’s about comprehending. Once you know its role, naming it comes easily. Developers, especially Rails developers, tend to think of tables before objects. I view this as a flawed idea.

00:10:13.800 I want data models that answer questions; I seek persistence to serve my application, not to be its focal point. Active Record leads us down a poor path, with every model inheriting from it, treating it as a mere persistence object when it should contain business logic. The goal is to design objects based on how they’re used to answer user questions rather than relying solely on abstract database normalization principles.

00:10:32.440 This synchronization between data models and front-end design is crucial. By aligning them, we'll last create structures that respond to user queries more readily than if we focus solely on normalization. Recognize that small decisions accumulate, and they can have a massive impact. If you're wrong a thousand times on a small scale, it will be harder to undo than being wrong once on a larger scale.

00:10:48.799 Collaboration through code reviews is important. I don’t like pull requests; by the time code comes up for a pull request, a thousand minuscule decisions have already been made about its appearance and functionality. If you have alternate ideas or questions about design, by that stage, you may hesitate to provide feedback because someone has already invested significant effort.

00:11:05.560 We end up accepting code that is technically sufficient rather than excellent code that should meet our architectural vision or standards of quality. This leads us to slide into entropy. We must learn to fail safely and create safe spaces for failure, embracing the idea of being wrong and treating it as a learning experience. This should happen early, not at the end.

00:11:23.679 Waterfall is an example of discovering issues only at the end. Getting it right the first time isn't an Agile development goal; it stems from waterfall methodologies. Let's not fall into that trap. We don’t say this out loud very often, but there is never a dedicated refactoring phase if you don't refactor as you go. You’ll never have time in the future to fix the duct tape you put on your code today.

00:11:42.960 Architecture will exist whether we design for it or not. We can either let it happen to us or be deliberate about it. In the past, dedicated architecture teams provided some control over architecture; I don’t want to return to that. Today, architecture is democratized, and we’re all architects, which is positive.

00:12:02.720 This transition encourages us to be generalists. Yet, with the freedom to collaboratively and independently create architecture comes the responsibility to be the best architects we can be. For our stakeholders, our teams, and our future selves, we owe it to be thoughtful about architecture. Architecture is fundamentally about being thoughtful, applying discipline, will, and creativity.

00:12:20.680 It’s about finding the sculpture in the block of granite and working to help it emerge. I am confident that with a little change in our thinking and processes, being more thoughtful and deliberate, we can create code that’s not only functional and timely but also elegant and beautiful. That’s the kind of code I am proud to write and what makes me happy.

00:12:39.480 Thank you for your time, and I'm so happy to be here today. I'm Coraline Ada Ehmke. Think about what makes you happy in your coding journey. Thank you very much.

RubyConf AU 2015