Why We Need DevOps Now: A Fourteen Year Study Of High Performing IT Organizations

Gene Kim

Gene Kim

1 talk

#devops

#continuous-integration

#continuous-deployment

#software-development

#collaboration

#technical-debt

#agile-practices

Why We Need DevOps Now: A Fourteen Year Study Of High Performing IT Organizations

by Gene Kim

In 'Why We Need DevOps Now: A Fourteen Year Study Of High Performing IT Organizations', Gene Kim presents his research on optimal practices within high-performing IT organizations, focusing on the integration of development and operations in achieving both service excellence and rapid feature deployment. The video highlights the following key points:

In conclusion, Gene Kim's presentation serves as a comprehensive guide to understanding and implementing DevOps practices, advocating for a transformative culture that dismantles traditional silos between development and operations, leading to better organizational outcomes.

00:00:16.380 Hey, just a quick thing before we get started.

00:00:22.410 I was actually observing this before. I thought I’d do the smart thing and copy my presentation onto a USB drive. I sat calmly while the presentation was being set up, but then I lost my USB drive.

00:00:35.030 Over the last 13 years, my passion has been studying a group of organizations that we used to call 'Gene's List' of people with great kung-fu. I've been studying organizations since 1999, so what makes these organizations great?

00:00:45.720 They have excellent project due date performance, great operational stability and reliability, solid security posture, and compliance. One of the questions we wanted to ask was: what are they doing so differently that's resulting in such phenomenal outcomes?

00:00:57.239 One of the things that surprised us at first was asking where these high performers come from. We noticed that the leaders of these high-performing organizations typically came from one of three professions: non-commissioned officers in the military, chemical or nuclear engineers, or auditors.

00:01:14.190 Can anyone speculate what these three professions have in common in terms of the values they hold dear? That's right, many people mentioned 'discipline' and 'rigor.' Non-commissioned officers give live ammunition to 18-year-olds, and in my journey, I've learned that in restaurant operations, they often hesitate to give knives to 18-year-olds.

00:01:40.140 Chemical engineers use long, elaborate recipes; if you miss sequencing two steps, an endothermic reaction can become exothermic. As a joke at Purdue University, we would say, 'How do you know if a nuclear engineering professor is from the nuclear engineering department? Count their fingers - if they don't have all ten, they're a nuclear engineer.' Auditors have been known to work with detailed checklists to maintain control.

00:02:19.080 From about 2005 to 2008, I helped raise about a million dollars to benchmark over 1,300 IT organizations to see if we could link controls with performance. We found that rigor and discipline seemed to predict performance. This finding became part of the Visible Ops Handbook published in 2004.

00:02:57.740 This journey ultimately took me deep into the heart of the DevOps movement. During this period, I observed a downward spiral occurring in virtually every IT organization, regardless of size or industry. What I want to share with you now is the nature of this downward spiral and how it can be described in three steps.

00:03:25.340 Act one begins in operations. How many people here would say their primary passion and vocation is in operations, and how many would say it's in development? Anyone working in information security? Those of us in information security often see ourselves in the middle of this conflict.

00:03:48.620 In operations, we are frequently placed in a situation where we have to support fragile infrastructure in production. Fragile infrastructure is problematic because if it blows up, no one knows how to fix it, and no one knows how to build a new one. These fragile services are usually under immense pressure from the most critical business initiatives of the organization, which means computer operations and projects are relying on these fragile systems. Consequently, when critical operations are dependent on fragile components, several terrible outcomes frequently occur.

00:04:40.640 Companies begin to miss commitments made to external stakeholders, such as customers, Wall Street, or analysts. This situation leads organizations to compensate by making even larger, more audacious promises, often made by product managers who might not have a solid understanding of technology.

00:05:39.660 And so, we often find ourselves in a scenario where developers are pushed to the brink. The joke in the industry is that show me a developer who isn't causing an outage, and I’ll show you one who’s on vacation. In reality, when urgent, date-driven projects are put into the queue with already committed outside market dates, tensions escalate quickly. As a result, developers typically spend all their time working on features, leaving no time for critical operational requirements such as operability and manageability.

00:06:25.850 This leads to a significant accumulation of technical debt. Visualizing technical debt can be quite revealing; at the end of every project, organizations often find themselves adding another piece of fragile infrastructure into production. This accumulation is problematic not only because it creates more fragility but also because it leads to longer and more complicated deployment processes. Many organizations might witness a deployment process that escalates from five minutes to a day, then two days, and so forth.

00:07:22.700 The loops become positively reinforcing; the worse the deployment process gets, the less likely teams are to deploy often. This creates warlike conditions between operations and development. It's as if developers don’t realize the product they just committed to is causing a data center outage for an entire weekend.

00:08:29.360 As this downward spiral continues, nobody achieves their goals. Developers struggle to get their features into production, testing becomes increasingly challenging, operations are inundated with maintaining services, and technical debt exacerbates the issues at hand. Meanwhile, information security often finds itself at the back of the line, ultimately dealing with compliance rather than security. This dynamic creates a sense of hopelessness and despair for those downstream in operations or information security.

00:09:25.570 This conflict is prevalent in virtually every organization. All organizations, regardless of size, face conflicting needs: the urgent demand for rapid business responses, which typically falls on development, versus the need for reliable, stable, and compliant systems, which is the realm of operations. This tension creates a brutal dynamic: IT organizations are often caught in a cycle where time-to-market supersedes stability and security.

00:10:10.540 Every company ultimately becomes an IT company because all business decisions have some IT impact. Fifty percent of capital spending is technology-related, which contradicts any notion that IT isn't a core competency. Furthermore, when business stakeholders express their needs, it often feels like IT is in the way, hampering the flow of work.

00:10:54.030 Historians will likely pinpoint 2009 as a pivotal year for the DevOps movement, largely due to the influential presentation by Patrick Debois at the Velocity Conference. He emphasized the need for high-performing companies, like Flickr, to release frequently and successfully. The impact of that presentation resonated widely, changing perspectives on how deployment could be approached.

00:11:13.290 In fact, Amazon's John Jenkins boasted about deploying every 11.6 seconds, which amounts to an astounding 23,000 deployments per day! This pace illustrates just how critical the conflict in deployment speed versus operational stability truly is.

00:12:13.030 Discussions need to focus on breaking the chronic conflicts that hinder fast workflows while maintaining system reliability and security. Several organizations that have embraced DevOps have proven that success is attainable, even among those who initially claim to be different. Understanding that the principles of DevOps can be applicable in any organization is essential.

00:13:13.490 Let’s discuss the underpinning principles of DevOps. In the book 'The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win,' we introduce 'The Three Ways' which serve as a guiding framework for these principles. The first way centers around the left-to-right flow of work. It explains the importance of transitioning requirements through development into operation so that value is delivered to customers effectively.

00:14:14.000 The focus is on understanding the flow of work from left to right and optimizing it so that work does not bounce back and forth between systems. We must strive for a continuous improvement approach to increase flow and efficiency.

00:15:14.000 The second way emphasizes how to take the learning from production to prevent future problems and improve the effectiveness of our processes. In this vein, it is essential to understand the needs of the next downstream work center, often an internal customer. This principle allows for a more robust understanding of how operations influence all parts of the workflow.

00:16:17.650 At Toyota, the idea of stopping the assembly line to fix problems reflects a proactive approach, preventing defects from moving downstream and accumulating issues. For operations and development, fostering understanding and accountability between teams is crucial. Each team must recognize how their work influences downstream tasks, ultimately contributing to system reliability.

00:17:24.100 By shortening feedback loops and amplifying the learning process, organizations can significantly enhance their capabilities. Products like Netflix have emphasized the importance of internal product hygiene, ensuring that all development practices are of high quality before being shared with any work center.

00:18:32.300 Learning from mistakes and iterating upon them is a vital tenet of this second way. Emphasizing a culture where mistakes are treated as learning opportunities allows for both personal and organizational growth. This ability to respond quickly to feedback and continually improve is the backbone of effective DevOps practices.

00:19:25.240 As organizations implement these practices, the collaborative spirit between developers and operations grows, ensuring effective communication and coordinated efforts. This collaboration often results in faster problem resolution and a more efficient integration of both development and operations processes.

00:20:34.900 The third way focuses on culture, creating an environment where experimentation and continuous learning are encouraged. Creating a culture of risk-taking where employees can freely explore new ideas allows for a more innovative atmosphere. This not only leads to improvements in current processes but also promotes mastery through repetition.

00:21:58.320 Creating this culture is fundamental for long-term success.

00:22:00.000 Successful companies recognize the necessity of taking risks, failing fast, and learning from those experiences while cultivating an environment where feedback is welcomed. Establishing processes for continuous improvement requires all employees to embrace their roles as innovators and contributors.

00:23:10.580 The principles laid out in DevOps apply across various sectors beyond tech startups. For instance, organizations like Intuit have successfully implemented rapid innovation practices, achieving significant improvements in their service delivery and operational capabilities.

00:24:05.890 Openness around innovation allows for adaptability, quick iterations, and painted responsiveness to change in any market condition. No organization is too small or too traditional to recognize the potential gains from adopting a DevOps mindset.

00:25:09.490 The importance of integrating these principles into everyday operation cannot be overstated. They offer a framework for moving past traditional barriers that have limited the capacity for responsive operational practices.

00:26:05.160 Establishing a shared vision and collective ownership of outcomes is paramount. This culture not only benefits individuals, it enhances the capability of the organization as a whole to deliver value efficiently and effectively.

00:27:00.430 Learning from industry leaders about their high-performing practices can spark cultural shifts in many organizations. Embracing experimentation as part of daily workflow creates avenues for continual improvement, ultimately enhancing the overall quality of service provided.

00:28:09.020 The future will undoubtedly require organizations to address the inherent challenges posed by legacy systems while navigating the fast-paced demands of modern technology. Companies must commit to creating environments conducive to continuous learning in order to stay competitive.

00:29:11.800 By fostering this kind of environment, we establish practices that put people at the center of effective changes in how we work. For many, these changes align with their personal aspirations for growth and achievement, assisting not only the organization but the individuals within it.

00:30:16.820 A culture that values both collaboration and accountability provides clarity of purpose. Such an orientation ensures that as teams work toward shared objectives, they can achieve far greater results.

00:31:27.950 Adopting practices that promote transparency around each stage of work fosters engagement among all team members. When reflecting on failures and successes alike, constructive dialogues emerge, encouraging iterative improvements.

00:32:44.580 Establishing a philosophy centered around continuous improvement prepares organizations for various challenges they may face in the future. Developing mechanisms for catching issues before they escalate enables both operational and developmental aspects to thrive in an adaptable environment.

00:33:46.490 In this light, organizations are encouraged to foster diverse teams and exploratory tasks around innovation and efficiency. Structured yet open experiments yield valuable insights that can inform operational protocols and service options.

00:34:36.430 As processes evolve, emerging knowledge is shared across teams facilitating smoother transitions and richer outcomes. Understanding that collaboration is key, we need to invest in systems that offer ongoing feedback.

00:35:20.180 Ultimately the narrative behind DevOps reflects a journey toward mutual understanding and respect across functional barriers. By building bridges between developer and operational teams, we can create an environment where everyone has a stake in the organization’s success.

00:36:11.770 This framework sets the stage for continuous enhancement within processes—be it through automation or collaborative efforts—such an alignment drives operational throughput. Crafting such a structure equips organizations to respond effectively where their strengths lay.

00:37:49.340 The goal, then, is to harness these principles to cultivate a transformative organizational culture. Consider organizations that have thrived in these settings, they’ve embraced change and implemented systems that reflect resilience and agility.

00:38:33.590 This is the world we wish to create—an environment where everyone is empowered to innovate while being supported with structures that facilitate their contributions.

00:39:23.310 Fostering continuous feedback is vital for organizations aiming to maintain relevancy in an ever-changing landscape. The opportunity to learn from the past while shaping the future creates momentum toward realizing collective aspirations.

00:40:16.550 Fundamentally, this is about ensuring that no one in the organization feels isolated or trapped by the constraints of their environment. Everyone should feel they play an integral role in driving the organization forward.

00:40:56.670 As we move into an era characterized by rapid advancement, nurturing adaptability and openness will be paramount for everyone involved. Everyone in the organization will need to work together to ensure coherence in objectives and outcomes.

00:41:54.840 The vision is to foster a communal spirit wherein ideas flow freely and where every mind contributes to enhancing efficiency across teams. This collaboration serves to eliminate barriers and drive the organization toward mutual successes.

00:42:56.160 Creating this framework is contingent on everyone within the organization understanding their roles and frameworks that motivate them to improve themselves while pursuing innovations.

00:43:41.340 Ultimately, the focus on storytelling as a means of communication unlocks different dimensions of understanding. All of which points towards fostering intrinsic motivations tailored to individual and organizational progress.

00:45:01.900 As we conclude, I appreciate your engagement and interest in optimizing processes through the adoption of developmental operations principles. I hope you leave here today with actionable insights to implement within your organizations.

00:45:36.490 Thank you all very much for your attention and participation!

MountainWest RubyConf 2013