A Tale of Two Feature Flags

by Rebecca Sliter

In the video 'A Tale of Two Feature Flags' by Rebecca Sliter, presented at RubyConf 2015, the speaker explores the benefits and challenges of using feature flags in software development. Feature flags allow development teams to release features gradually and mitigate risks associated with large deployments. However, they can introduce unexpected complexities that can hinder productivity if not managed properly.

Key points discussed include:

- The dual nature of feature flags: They provide the ability to test and release features with more control but can also create technical debt when overused or mismanaged.

- How feature flags work: Initially, these flags are simple configurations in files that dictate which features are active. They enable teams to deploy code incrementally and test features with specific user groups before a full rollout.

- Real-world example: Sliter shares a project experience that started well but escalated into complexity as they added multiple feature flags to manage dependencies on external services, testing scenarios, and user interfaces.

- The importance of simplicity: She emphasizes the necessity to limit the number of active flags for clarity and to avoid overwhelming complexity, advocating for a single flag per feature whenever possible.

- Consequences of neglect: Long-lived feature flags can create significant technical debt, as exemplified by the Knight Capital case, where an outdated feature flag led to a catastrophic failure.

- Best practices: To manage feature flags effectively, teams should set deadlines for removal, designate maintainers, and regularly review their status to prevent legacy issues.

The main takeaway is that while feature flags offer significant advantages in software delivery, disciplined management and a streamlined approach are essential to harness their benefits without incurring complexity or risk.

00:00:11.990 Hello everyone.

00:00:14.450 My name is Rebecca Sliter, and I'm an engineering manager at Kickstarter. Before joining Kickstarter, I was a consultant, which involved traveling to clients of all shapes and sizes to discuss software development practices.

00:00:20.280 Because of that experience, I've grown to care greatly about the tools and practices that help teams deliver better code more efficiently.

00:00:25.560 In a nutshell, I believe development should be as simple as possible. Therefore, I try to avoid writing clever code, focusing instead on clear communication of ideas, domains, or tools across the development team.

00:00:29.910 This brings me to the point of my talk: A Tale of Two Feature Flags. Specifically, we'll explore what happens when a team implements a set of features behind flags and ships them. Today, we'll discuss what makes a flag successful, comparing it to other feature flagging strategies that may not work out well.

00:01:07.710 Software development, in many ways, has never been better. Over the last few years, we have developed languages, patterns, and release strategies that abstract away much of the grunt work, allowing us to focus on delivering great products. Yet, these conventions, patterns, and languages can be misused if not guided appropriately.

00:01:21.080 While it's the best of times, it can also be the worst. Feature flags, for instance, are designed to ease our workload but can unexpectedly introduce complexity that detracts from our overall productivity.

00:01:32.190 Let's first focus on what makes feature flags powerful. Features begin with an idea, igniting the excitement of building something great and delivering value to users.

00:01:39.540 However, this journey often leads developers to face the hard truth: writing software requires significant effort. Developing a feature from start to finish can be time-consuming, necessitating coordination across the entire team.

00:01:57.420 Once the development work is complete, you're often left with a sizable bundle of code ready for deployment—and it can be risky to release that all at once.

00:02:04.880 Even if the code has been tested, many moving parts are involved, and small bugs are likely to slip through undetected. The uncertainty of how everything will perform in a live environment is daunting. What would be ideal is to test ideas on a small group of users first.

00:02:20.100 However, large monolithic releases hinder this possibility. Moreover, your implementation may depend on new databases or third-party APIs, making simultaneous deployment risky.

00:02:28.710 Communicating these risks to stakeholders can be challenging; they often push for grand features to be released. You might find yourself as the naysayer, caught between business ambitions and technical realities.

00:02:37.680 Yet, we all know the solution: put features behind a feature flag. At a high level, flags are configured in their own files—often YAML files—where you can decide which features to enable or disable.

00:02:44.600 Initially, these files are straightforward, containing boolean values that define the state of different features. As development unfolds, rather than fully implementing a feature, you query these boolean configurations to determine the user experience.

00:02:59.750 Depending on the state of the flag, users will either access the existing UI or the newly implemented feature. This method allows for iterating on your development without risking a complete overhaul upon release.

00:03:14.850 Feature flags help address many challenges we face in software releases. If a feature is complex, we can deploy it iteratively, alleviating the burdens of massive deployments that require extensive configurations upfront.

00:03:26.460 Once a feature is ready to be shown, you can restrict access through the flag, making it available only to specific users, perhaps internal staff for testing.

00:03:39.750 As you gain confidence, you can gradually expand access until everyone can see the feature. Afterwards, you remove the flag and the surrounding conditional logic, making the feature universally available.

00:03:48.080 Additionally, feature flags can integrate smooth A/B testing methodologies, where users are shown different versions of a feature to assess performance before making final decisions.

00:04:04.840 Another practical use of feature flags is employing them as kill switches, allowing teams to deactivate problematic components of their code without affecting the entire application.

00:04:17.739 For instance, if a page experiences a spike in load due to high traffic, you can disable specific features like commenting without taking the entire page down.

00:04:28.200 These benefits make feature flags seem like the best of times. However, what happens when a team introduces too many flags or mismanages them? Let's discuss what occurs in practice when a team decides to ship code behind a feature flag.

00:04:39.820 Often, teams begin their journey with the best intentions. I worked on a project last fall involving faceted search for a retail client, where search results came from several independent sources.

00:04:51.020 Given the uncertainty surrounding the performance of these external services, we decided to encapsulate the feature behind a feature flag, allowing us to test its impact on user experience.

00:05:05.400 However, as with many projects, changes in scope emerged. The external dependency wasn't ready when we needed to move to the next facet, which required a new separate flag to enable its deployment.

00:05:20.490 What began as a tidy implementation turned into a mess with multiple feature flags. While this may not seem inherently bad, it complicated our development process significantly.

00:05:32.790 This tight coupling of features also meant that we later struggled to determine how to display the user interface should one flag be active and the other inactive.

00:05:47.540 It required extensive discussion with stakeholders and resulted in a truth table to ensure we were clear on the business logic, which ironically introduced more complexity to our code.

00:06:05.080 Testing the resulting features became overwhelming due to the intricate dependencies, making it easy for bugs to slip through. Our QA team struggled to verify whether the end user experience matched expectations.

00:06:18.270 This situation illustrates my first key point: treating independent flags as a means to iteratively release features can be exceptionally challenging.

00:06:31.650 Around this time, our team recognized we had limited options: either scrap the project and push back on the project manager or continue carefully by testing the feature extensively.

00:06:42.929 We chose to proceed with care, but it quickly became burdensome. While manual testing allows for thorough checks, it’s impossible to guarantee that everything is perfectly checked, and mistakes will inevitably happen.

00:06:55.340 We found ourselves spending significant time testing instead of obtaining valuable feedback from our clients or users, which is fundamentally what putting something behind a feature flag should accomplish.

00:07:08.850 Through this, we learned an important lesson: feature flags should simplify the process, not add unnecessary complexity. If implementing a flag consumes more time than the feature it encapsulates, something is amiss.

00:07:21.010 The key takeaway from my experience with faceted search is that we should stick to one flag at a time for a given feature to maintain simplicity and clarity.

00:07:32.360 Now, let’s delve into the implementation of one of those flags. The search feature I previously mentioned was designed to deliver different types of results we referred to as facets.

00:07:43.560 To manage these facets, we initially implemented a single feature flag backend. However, as we adopted this method, it spread throughout our codebase.

00:07:54.740 The challenge arose when each team used a different strategy for feature flags, leading to difficulties in unit testing since each test had to know the flag state.

00:08:10.240 What was supposed to be a simple implementation ended up resulting in tests coupling deeply with the flag's state, complicating maintenance and future enhancements.

00:08:23.000 Removing toggles became a laborious process every time we dialed the flag up to 100 percent. We found ourselves repeatedly updating backend code to accommodate toggles, rather than cleaning up after the implementation.

00:08:39.150 This resulted in additional time spent reviewing code and trying to evaluate the test suite, further complicating the simple task of removing a flag.

00:08:51.840 Ultimately, the flag's presence became a burden, complicating our testing landscape and delaying timely delivery of code, as we grappled with these intertwined dependencies.

00:09:05.220 Feature flags are designed to streamline our workflows, yet without proper management, they can morph into significant burdens. Making flags permanent often adds unnecessary complexity to systems.

00:09:18.529 Teams often hesitate to remove flags because they fear they might break critical functionalities, leading to a scenario where flags intended to be temporary become immortalized.

00:09:32.540 Some flags morph into kill switches, which can hinder team performance due to the lingering complexity that arises from keeping outdated flags in the codebase.

00:09:46.010 Occasionally, kill switches can be beneficial when used purposefully, such as when Stack Overflow employs a kill switch to disable posting while undergoing maintenance.

00:10:02.720 However, it's paramount that these switches are consciously architected to remain long-term, unlike the temporary nature of a feature flag.

00:10:17.220 When thinking about flagging strategies for temporary feature flags, I would recommend isolating their usage within the codebase.

00:10:30.700 A flag should exclusively reference a single part of your application, reducing potential confusion around its functionality and improving removal efforts.

00:10:46.529 Additionally, treating flagged code as new code, separate from production, encourages clarity around what a flag does and when it should be removed.

00:11:03.880 When it's time to remove a flag, it should be clear what needs to be removed, enabling a smoother workflow and reducing bugs or corruption in the code.

00:11:20.279 Now, let’s pivot to the question of maintaining clean code with feature flags. A frequent issue arises when teams neglect to remove flags after their expected lifespan.

00:11:36.320 Flags often remain in codebases, creating technical debt that can accumulate unnoticed. Over time, this complicates the mental burden when developers interact with the codebase.

00:11:49.170 Teams may convince themselves to leave flags functional for a chance to review performance later. However, unsupported flags represent a blockage that requires ongoing mental energy.

00:12:07.520 Considering the example of Knight Capital, it became evident that ignoring obsolete flags can have dramatic repercussions. Their system experienced a historic failure due to long-lived feature flags.

00:12:18.780 In their case, a feature flag was left on for eight years until a different team inadvertently enabled it, leading to catastrophic financial consequences.

00:12:31.920 Actions were unintentional, but they highlight how the presence of lingering flags not only complicates management but shrouds potential pitfalls in risk.

00:12:44.560 What became clear is that long-lived feature flags should not be ignored. Teams must adopt a proactive strategy to ensure that obsolescence doesn't result in unwarranted failures.

00:13:00.080 A best practice for mitigating these risks is to establish procedures that enhance visibility and accountability surrounding feature flags.

00:13:14.410 For example, introduce deadlines and assign maintainers to each flag to clarify responsibility over its state and future deployments. These actions contribute to an organized approach that promotes continual maintenance.

00:13:31.300 The process at Kickstarter involves regularly checking active flags and escalating accountability when they haven't been touched in some time.

00:13:48.022 By following this strategy, teams can make significant strides toward cleaner code, reducing the risk of features sitting unmonitored indefinitely.

00:14:01.200 Ultimately, while feature flags allow teams to ship code quickly, they require discipline and structure. Without a robust approach to their management, they can become more of a hindrance than a help.

00:14:14.822 It’s important that teams remain vigilant about flag usage to scale effectively and foster a culture that prioritizes removal and efficiency.

00:14:28.720 In conclusion, embracing feature flags as a part of your software development strategy means striking a careful balance between their utility and their potential downsides.

00:14:41.800 As long as we recognize the complexities and adhere to a disciplined methodology, we can harness feature flags to release better code faster and more reliably.