Maintenance Matters: Maintaining Your Rails App and Your Sanity

DevOps

Annie Kiley

#development-best-practices

#testing

#documentation

#devops

Maintenance Matters: Maintaining Your Rails App and Your Sanity

by Annie Kiley

In the talk "Maintenance Matters: Maintaining Your Rails App and Your Sanity", Annie Kiley emphasizes the importance of code maintainability, sharing insights from her nearly ten years of experience in software development. She highlights that the time spent on writing maintainable code is a significant investment that can pay off in the long term. Throughout the talk, Annie presents ten strategies to make the maintenance of Rails applications easier and more effective:

Code Review: Emphasize the value of continuous code reviews as collaborative discussions rather than simple approvals, fostering team knowledge and improving code quality. Use GitHub settings to enforce peer reviews.
Linting: Adopt standardized formatting across projects to avoid formatting disputes. Tools like pre-commit can be used for automation. Enforce linting standards in Continuous Integration (CI) to keep code readable.
Good Tests: Define good tests as those that accurately reflect the application’s functionality. Annie underscores the necessity of integration tests to facilitate safe refactoring and upgrading. Ensure CI fails when tests do not pass to maintain code quality.
Helpful Logs: Focus on writing meaningful logs to aid in debugging without clutter. Annie shares an anecdote about managing log outputs effectively during a system debugging process.
Thoughtful Launches: Stress the importance of deploying code prior to its launch and using feature flags to manage and mitigate risks involved in feature deployments.
Timely Upgrades: Encourage regular system updates to avoid accumulating technical debt, utilizing tools like Dependabot to automate some of this process.
Helpful Shortcuts: Document repetitive tasks and promote the use of bin scripts for efficiency and ease of onboarding new developers.

Throughout her presentation, Annie reiterates that while these practices may seem fundamental and familiar, consistent application can significantly reduce maintenance burdens, ultimately contributing to a more manageable coding environment. She concludes with a note on the necessity of fostering good practices to better navigate the complexities of software maintenance and encourages developers to further explore the topic through her company’s resources.

Overall, her talk serves as a reminder that focusing on the fundamentals of software maintenance can lead developers to a more satisfying and less stressful coding experience.

00:00:03.600 Um, okay, hi! Before I get started, I'm just going to ask you guys a few questions. Raise your hand if you are currently working on a software project of some kind. Okay, good! I'm in the right room. How about this: are you working on a Greenfield project, say one that is less than three months old? So, are you working on something shiny and new? Nice, cool.

00:00:11.120 Okay, how about less than six months old? A year? Two years? Three years? Five years? Ten years? Okay, hands down... that’s a lot of maintenance! I was talking to my uncle who came through town, and I was trying to explain what my talk was about. I said, "Yeah, conferences are great! People tell you about things that were always there, that you didn’t know about, like the coverage module, or things to make sure this gets back on." You know, things that you could be doing better, like pair programming with non-violent communication. It’s great, and everybody talks about all these new things.

00:01:06.840 My talk is not one of those talks. My talk is about what we spend a lot of our time doing, which is maintenance. I’ve been building applications with Rails, Phoenix, Django, and a smattering of other frameworks for almost ten years, and the more software I write, the more I think that maintenance matters. Considering maintainability matters, and that minute for minute, the time and investment you put into writing maintainable code and maintainable software is some of the best time that you spend. That stuff often gets deprioritized, but it is really important.

00:01:40.720 So here I am, taking the sleepy 3 PM slot to talk about maintenance. I work at a company called Bot, which means I build software for other people. I’ve been working there for about seven years, and it’s great! One of the things I love about working at Bot is that we’re not experts in any one thing, but we are experts in building software. So, we spend a lot of time talking about how to do that better. For me, a lot of it comes back to thinking about maintainability, doing the little things right, and building strong foundations.

00:02:09.039 This is our office in Durham, and I know some of you are around, so feel free to come by anytime. We’re right downtown. Without spoiling my first point, my goal here is not to blow your mind. My goal is to talk about some of the simple things, the fundamentals. My guess is that you’ve heard and thought about all of the things I will say, but I would be shocked if anyone is doing them all of the time consistently. So, my hope is just to inspire us to do those little things as developers that make our job easier later.

00:02:50.560 Without further ado, here are ten quick hit things that you can do to make maintaining your Rails application just a little bit easier. Most of these ideas apply regardless of the stack. Number one: code review. If you’re lucky enough to work on a team, you should be doing code review all of the time. Code review does not have to be and really should not be just some senior developer signing off on a junior developer's work. Code review should be a conversation.

00:03:20.080 "Hey, I wrote this code. Here it is!" People can ask questions, "What’s going on here?" People can say, "Oh, I like this! I learned something new!" The benefits of code review compound: the more people see everyone’s stuff, the more everyone knows about the code base, and the more good conversations happen. Just this week, I’m working on a shiny new Greenfield app, and it’s great! I did a small thing, and one of the guys I’m working with said, "Hey, why isn’t this in here?" I explained why, and we ended up taking it offline, talked about it on Slack, and concluded with a better decision for the application that everybody knows, just because we had a code review.

00:05:04.120 But don’t just say you do code review; require it! Go into GitHub and make the settings so you shouldn’t be able to merge a PR without a second set of eyes on it. Sure, sometimes you're going to Slack someone and say, "Oh my God, give me a thumbs up! This has to go to production now!" That will happen. But making it easier to do the better thing and a little bit harder to do the wrong hacky thing has long-term implications on the health of your code base. This doesn’t matter if you’re working on a 10-year-old app—you can set this up right now!

00:05:44.640 Number two: linting. I am down to have a conversation with any of you about tabs vs. spaces or how to align your hashes or whatever. That can be a really fun code review. But when I’m trying to get stuff done, it’s not the time for those conversations. I like to have standardized formatting on every project and to enforce it. This way, you can have that conversation with a computer instead of people. We use pre-commit a lot to automate various tasks, which is pretty flexible; you can use it with different languages and frameworks. I highly suggest checking it out.

00:06:38.240 What’s nice is you get a local hook and then you can enforce it on CI, and everything just happens automatically. It’s not just that your build fails, but it's like, "Hey, I tried to commit this code, and it won’t let me until everything is formatted and organized." With Python, we use tools like Black and isort to organize our imports. It does a lot of good stuff, so I recommend using pre-commit for that. For Ruby, I really like Standard Ruby. What I like about it is that it cannot be configured. There’s no time to even have that conversation!

00:07:04.440 I feel like I once brought RuboCop onto an app to try to prevent having that conversation, and then we had a long discussion about the PR that brought RuboCop onto the app. Again, I’m trying to avoid that because I want to focus on the more interesting problems. I really like Standard Ruby because it just does it. I might be a Test Double—thank you!—for using it on Ruby. We also use pre-commit to lint our JavaScript and all that kind of stuff.

00:08:43.520 But similarly, don’t just say you do it; enforce it on CI. Your build should fail if your formatting standards are not met. Yes, this is annoying, but yes, it keeps your code much more readable and standardized over time. If you run this with fix locally, it’ll do the fixing for you; you don’t have to think about it! You can apply this to an existing app too. Just lint your whole app, set some formatting standards, run it with autofix, and now you have this maintainability improvement, even for older applications. So again, I think setting standards is important, but I think enforcing them is a big part of that, and over time, the payout for maintainability is very high.

00:09:36.880 I don’t have time in this talk, but ask me about one of my few open source contributions where I made the RuboCop hash alignment cop stricter than the default one, and a lot of people yelled at me on the internet because it broke their builds. Number three: good tests. There are a million things to say about tests, but I’m going to define good tests as those that fail when your application does not do what it should do and pass when it does do what it should do. You should be writing those tests.

00:10:38.560 There are a lot of different kinds of tests that are valuable and serve various purposes, but I've got to focus on feature tests or integration tests right now, because those, I think, provide the infrastructure that allows you to maintain, refactor, and upgrade an application easily. Again, don’t just have tests—your build should fail if your tests don’t pass. Yes, it can be annoying, but how many times have you pulled something you want to work on, and the build happens to be failing? The annoyance of that is way greater than the annoyance of whatever you have to do to fix the thing in order to ship your feature. Have that fight with a computer early.

00:11:55.520 This is another example of trying to make it easier to do the right thing than to do the quick, hacky thing. The easier you make it to write maintainable code, the better your life as a maintainer is going to be in the long term. GitHub should look like this: it should be yelling at you when you don’t have code review or when your tests aren’t passing. I know you can get around this, but you should have to click like three red buttons to merge something. Sometimes you have to do it, but you should feel bad about it.

00:13:20.160 As a bonus for writing tests: I’ve worked on a lot of languages and a lot of frameworks, and I really believe Ruby has basically the best suite of easy, usable in-language feature tests. This stuff is great! I like RSpec, and I really like how it allows testing of Rails application behavior easily. You know, writing the tests in the language you already know makes sense—so write good tests. If you're writing tests, you should enforce test coverage. High test coverage is necessary but not sufficient to have good tests, but I do think it helps.

00:14:34.920 This is probably the most controversial thing I’ll say: we like to use SimpleCov at the bottom. I like to enforce test coverage at 100%. It's a little brutal, but I think it’s fine if there are things that shouldn’t be covered, such as some fiendishly complex integration for asset storage that only matters in a deployed environment. You can wrap it in a no cov comment, so at least that is documented that it’s not tested.

00:15:45.360 You can also gradually increase your coverage. You can slap test coverage on an existing app, see what it is right now, and maybe your coverage is 54%. Set your test coverage at 54% and don’t let it fall below that, then increase it over time. This patience in terms of upgrades is not only for starting new applications; it's also important to manage existing ones. You can use an environment variable at the top of your Rails helper to enforce your desired coverage.

00:16:21.600 Number five: helpful logs. This is my favorite point. Logs are mostly for developers. How often has someone said, "Hey, weird behavior is happening" and you replied, "Okay" and then ten minutes later you’re gripping your logs for anything that might be relevant? A lot of us don’t think about our logs until we’re in that situation, and I think that’s a mistake.

00:17:56.320 Fun story: I was working on a system where people would buy zoo tickets, and then we would email them the tickets. A lot of money was moving around, and a weird thing happened. Someone asked me to figure out what was happening, so I turned on the debug-level logs on one of the staging servers, and then saw that at that log level, Rails prints out the contents of all email. If your emails include attachments like several pages of zoo ticket PDFs, it converts them to hexadecimal and prints them in the logs every time an email is sent. I ended up with tens of thousands of lines of this!

00:18:39.000 Now, you could still search in that, but that’s a terrible performance issue if it’s in the logs even if it was just for debugging on an integration server. It only took twenty minutes to figure out where the payload was, override it, and tell it to only print out the names of the attachments. All of a sudden, my logs were useful again. You deployed that, and I could see everything that happened around that time and fix the bug. So take out logs that aren’t providing any value. On the flip side, it’s trivially easy to tell a computer to write to a file if something is happening in your application that you want to log.

00:19:51.840 For example, in a multi-tenant application, just print out the tenant at the start of every request. It makes your logs so much more searchable—it’s incredibly useful! Rails makes this very easy. Pro tip: you should use blocks for your Rails logging statements. This way, it’s only relevant code that evaluates depending on the log level set. This approach ensures that large debug statements are evaluated only when necessary, so they won’t clutter your logs. Also, I’m a huge fan of creating custom log files for background processes, jobs, or cron jobs.

00:21:27.680 For instance, suppose every day at 2 AM I pull in a bunch of data and people use that throughout the day. If a price is wrong and a user says, "Hey, this price is incorrect," there could be many reasons for that. Knowing whether it relates to the cron job is essential, so for any cron job I produce, I’ll log something like, "Hey, I started the cron job," and write any relevant data immediately, which you can name something clear like `log_cron.log'. This way, if anything goes wrong, I can easily check that log file and know immediately if it’s the source of the issue.

00:24:03.160 I’ve found this to be incredibly valuable because it’s trivially easy to accomplish. Computers excel at writing to files; that’s essentially what they do. On another note, take care of your production logs: ensure they're rotating and backed up properly. If you've got a lot of servers, aggregate your logs! Paying a little attention to your logs ahead of time can be the difference between saying, "I don’t know why that’s happening, sorry" and being able to say, "I can't figure it out" with plenty of information at your fingertips.

00:26:43.560 Number six: thoughtful launches. When writing a feature, especially in a big existing system, consider how you’ll get your app to that place. If your plan involves a long-running feature branch, merging it into the main branch, deploying, running migrations, and hoping for the best, that can lead to very challenging situations. If things go wrong, you can be in for a very tough day.

00:27:12.240 My first thought on launches is, if at all possible, have all your code running in production before the launch. Even if it means maintaining some temporary data duplication. Get that done! That way, any new change you deploy is less likely to break existing functionality. Your launch can be as simple as exposing your new feature.

00:27:58.960 In this scenario, if something goes wrong, it’s just a matter of toggling a feature flag to hide it on the back end while you make corrections. This method is significantly less stressful than the alternative, which often involves emergency fixes late on a Friday afternoon.

00:29:51.200 There are many ways to achieve this, but simply put, figure out a way to implement feature flags. You could build this functionality into your application as a method in your controller, or simply check an environment variable. This approach requires no database support or complex dependencies. You can deploy your code long before launching your feature. In that case, launching the feature is just a matter of changing that environment variable and restarting the app.

00:31:13.920 So if any issues arise, the response process isn’t frantic—it's more of a priority for the next workday, which is vital. This can also get more sophisticated when working on systems with large audiences; for instance, when developing the Zoo Ticket sales system, we gradually routed traffic to the new system over a week—starting with a small percentage.

00:32:37.920 If something went wrong, we easily redirected traffic back. This kind of preparation can save you from a lot of stress on launch days; as a best practice, always try to ensure your code is running in production before the actual launch.

00:33:56.320 Number seven: timely upgrades. I wish I had a silver bullet to offer here. A few years ago, I wrote an entire blog post about an application that couldn't work with the new API service and required a thorough upgrade process. What seemed like an easy change turned into upgrading Ruby, Rails, and an entire environment—and it resulted in a staging environment that was out for an extended period while everyone else enjoyed their vacation.

00:35:37.024 So it is critical to prioritize upgrades! Tools like Dependabot can help automate some of this work. There is much to consider for your projects, and if you can inspire someone to tackle these upgrades sooner rather than later, it could prevent the stress of having to address major issues later.

00:37:03.040 Number eight: helpful shortcuts. If you're a developer, there’s likely something you're doing constantly in your application; document it! Rails has plenty of ways to customize the shell. By ensuring useful shortcuts, you’re contributing to the ease of development even for new team members or future you.

00:38:24.960 A common practice I advocate for is using standard bin scripts; this practice can streamline your workflow by reducing repetitive commands. Writing these shortcuts not only benefits you but also offers invaluable documentation and insight to other developers. When working with Docker, for instance, creating scripts to automate tasks means you need not remember lengthy commands.

00:40:11.840 Finish up these shortcuts and standards with solid documentation: I’ve rarely seen projects with great documentation and bad code. If you are thinking of documentation, then you’re often also contemplating the code itself. At the start of projects, I have a checklist to ensure we've covered the essentials for local project setup, testing, code coverage, and deploying.

00:41:38.560 One more important note: if a new developer is coming onto your project and they ask you questions on setup, that’s a prompt for you to open a pull request or a `PR` to solve that question in the documentation.

00:42:30.720 Finally, if you have standards, enforce them! Ensure all team members understand what the standards are, and take the steps to aid their adoption. The tougher you make it to take shortcuts or do hacky fixes, the less likely you are to deal with those messy situations over time. This payoff will only grow over time.

00:43:44.320 Now, here’s my list again. I don’t think I've mentioned anything new or groundbreaking. However, I recognize that I don’t always follow through on my intentions to enforce these principles, either. Yet, I do think it’s necessary to discuss this concept, especially since so much of our time is inevitably dedicated to maintenance.

00:45:25.760 If nothing else, I hope I've inspired someone to take action that fosters better practices—contributing to making it a good day rather than a bad one in the long term.

00:45:47.760 Lastly, we wrote a recent blog series where we expand on these points further and discuss some of the tools we like. You can find it at viet.com. I work at Viet and may be difficult to locate otherwise; feel free to reach out there.

00:47:15.840 And that's all. Thanks!

Blue Ridge Ruby 2023