Search
MountainWest RubyConf 2013
ChatOps at GitHub
Edit
#chatops
#devops
#infrastructure
#deployment
#automation
#continuous-integration
Summarized using AI

ChatOps at GitHub

Jesse Newland • January 28, 2020 • Earth

The video titled "ChatOps at GitHub" features Jesse Newland discussing how GitHub's operations team effectively uses ChatOps, specifically through the implementation of Hubot and Campfire, to streamline and automate their workflow. The talk, presented at the MountainWest RubyConf 2013, highlights the challenges of keeping pace with rapid feature development while ensuring stable infrastructure changes. The key points from the presentation include:

  • Definition of ChatOps: ChatOps is presented as a practical method to automate operational tasks by using a bot called Hubot within group chat environments like Campfire.

  • Evolution of Hubot: Initially a simple Campfire bot, Hubot has developed into a primary control service for infrastructure management, capable of more than just playful interactions.

  • Operational Workflow: Jesse explains a typical deployment cycle involving updates to their internal Git package, showcasing how they utilize Puppet through Hubot in a controlled rollout process.

  • Continuous Integration: The integration of CI bots within the chat allows for real-time feedback on branch builds, and Hubot can initiate deployments once confirmation is received.

  • Real-time Visibility: One of the significant advantages of ChatOps is that team members can view all actions taken in real-time, which enhances communication and reduces the need for extensive documentation or onboarding.

  • Actions and Monitoring: Besides deployments, Hubot performs various tasks such as checking the status of branches, monitoring server loads, and handling alerting via PagerDuty.

  • Incident Management: The transparency afforded by ChatOps during incident response helps maintain a clear flow of information, which is crucial, especially in remote working environments.

  • Cultural Shift: Adopting ChatOps nurtures a culture of shared responsibilities and collaborative problem-solving, allowing tasks like duty swapping to happen seamlessly.

  • Accessibility: The mobile capabilities of Campfire provide teams the flexibility to manage alerts and services on the go, enhancing operational responsiveness.

In conclusion, Jesse emphasizes that ChatOps, through Hubot, fosters a collaborative and efficient operational environment that benefits everyone involved. He encourages viewers to explore Hubot for their own uses, highlighting its potential to enhance operational efficiency at any team or organization.

ChatOps at GitHub
Jesse Newland • January 28, 2020 • Earth

Ops at GitHub has a unique challenge - keeping up with the rabid pace of features and products that the GitHub team develops. In this talk, we'll focus on tools and techniques we use to rapidly and confidently ship infrastructure changes using Campfire and Hubot as our primary interface, and the benefits we've seen from this approach.

Help us caption & translate this video!

http://amara.org/v/FGcZ/

MountainWest RubyConf 2013

00:00:20.480 Hey everyone! I'm Jesse, and I work in operations at GitHub, also referred to as DevOps, or whatever the kids are calling it these days. Today, I'm going to be talking about ChatOps at GitHub.
00:00:30.960 So, what is ChatOps? ChatOps is the pattern we've been using to automate common operational tasks with Hubot. Before I go any further, I’d like to see a show of hands: how many of you use some form of group chat, like IRC or Campfire, in your daily development work? Cool! How many of you deploy from that environment? Okay, how many of you deploy from the command line?
00:01:04.280 Alright! Hubot started as a simple Campfire bot, but it has evolved into something much more significant. Despite some people's perception of Hubot as merely a bot that shows funny pictures and cat memes, it serves as our primary control service for our infrastructure.
00:01:09.640 Understanding the principles of ChatOps from an outside perspective can be quite challenging. So, let's start with an example. Suppose we've made updates to our internal Git package, and we believe it’s ready to go. However, we want to roll it out in a controlled manner and establish confidence in it. We use Puppet for configuration management—apologies to the chef folks in the house—yet we utilize Hubot to interact with Puppet. This is how we manage these changes in a ChatOps style.
00:01:42.440 We have a GitHub repository for our Puppet configurations, which is the process we all know well. We check out a branch, change the version number of the package, make a commit, and then push that up to GitHub to create a pull request. While we're writing the pull request in Campfire, we have a CI bot that streams the output of the build. Every branch pushed to GitHub is automatically sent through CI, which is incredibly helpful.
00:02:26.640 Now that you know the branch is green, you can use Hubot to deploy that branch to production. In this case, we update the Puppet master to set up an environment called 'gh13.' Hubot provides deployment output, informing us that we’ve restarted the Puppet master. Puppet has a 'noop' (no operation) mode that allows us to establish confidence when rolling out changes, ensuring that there are no unintended side effects. This mode simulates what would happen if you executed Puppet with this branch.
00:03:00.920 After running that with Hubot and getting the output, we see the version number of Git changes. Now, once the noop gives us confidence that things will work as intended, we roll out the new Git version to one file server, keeping the potential impact of regression small. Hubot displays the output again, confirming that the package has changed, so the new package is live.
00:03:41.039 Next, we check that everything is functioning correctly. We've added support in Hubot for displaying graphs from Grafana and Graphite. We request a graph from Hubot and verify that everything looks normal and within acceptable bounds. If we're confident that this version looks good, we proceed to roll it out everywhere.
00:04:06.760 From there, we merge the pull request, letting everyone know what happened. CI runs again to ensure that the master branch is green. Once CI passes, the master branch is automatically deployed to production. Hubot keeps you informed that you were the one who merged the pull request, even though Hubot executes the actions for you. When it's all done, you receive the same notification.
00:04:34.480 Now that the new version of Git is on the master, we can push it everywhere Puppet runs. Puppet operates on an interval, akin to Chef, but sometimes when we are doing a controlled rollout, we want to force it all at once. So, we roll it out everywhere, and again, Hubot informs us that the package version has changed across all different file servers.
00:05:10.640 To assess the impact of the deployment, we check the load on all file servers which looks good. We peek at the logs to ensure nothing else is broken, and we also check that we are still receiving hooks for post-receive and upload packs, confirming that everything is functioning as expected.
00:06:38.240 So, that's ChatOps in a nutshell. It’s quite simple. Essentially, it's about doing all the things you do daily in Campfire. Hubot is the star of our Ops team, performing numerous tasks for us every single day. We interact with Hubot to work with other simple tools, APIs, and various scripts we have created.
00:07:05.120 All these internal tools engage with Hubot, but it also interacts with external APIs seamlessly. By utilizing Hubot effectively, we’ve seen distinct advantages, particularly as our use of Hubot within Ops has grown. I showed you the entire flow of rolling out a new version of Git across our infrastructure. One exciting aspect of ChatOps is that everyone can see what happens in real-time without needing to tell others that a task has been completed.
00:08:06.320 Everyone is passively aware of these operations happening, and there’s no need for extensive onboarding on how to deploy, as this is immediately visible to anyone in the chat room. We don't just use this method with Puppet; we employ ChatOps for many other tasks in our infrastructure as well.
00:08:33.360 For example, we can check the status of any branch built, deploy a branch of any GitHub app to a specific server, and get graphs of the app's recent performance. We can also monitor the status of Unicorn across all front ends, check the status of a rescue queue, or monitor loads across different servers.
00:09:59.680 We can examine current connections to a front end that we think may have issues since we deployed a branch to it. By comfortably viewing logs for our services on that front end, we can determine if we’ve broken anything, and if we did, we can quickly take it out of the load balancer.
00:10:25.679 We also determine who is on call, allowing us to apologize promptly for any issues we cause. If we're unsure whether we've broken something, we may check Pingdom to confirm if everything is down or if it’s just a minor issue.
00:11:05.000 Once we identify which branch caused the problem, we can revert to the master on the affected front end, check the logs to ensure everything has returned to normal, and feel accomplished when we fix the issue. Once verified, we can reintegrate the corrected front end back into the rotation and update the status site.
00:11:58.840 After resolving any outages, we might notice unusual activity, such as a strange IP address linked to a spammer. This could lead to some light-hearted conversations among the team while we reflect on our responses during incidents.
00:12:09.200 All these interactions make up the ChatOps culture—building tools that simplify infrastructure operations, allowing easier access to Hubot rather than coding in a terminal or using a web browser.
00:12:29.880 This is the essence of the talk: by integrating tools directly into conversations, everyone is collaborating and learning continuously, even if they are remote. Ryan Tomo, the creator of Hubot, had this vision since the initial commits to Hubot. It started as a simple repository of shell scripts for managing our infrastructure but has evolved into a crucial component of how we conduct training.
00:13:57.000 When explaining how to perform a task in Campfire, I typically demonstrate it in real-time, allowing others to see the process. This transparency leads to better communication of efforts; when performing tasks in a terminal or dashboard, you have to verbally inform others afterward.
00:14:20.240 By using Hubot in a chat room, everyone can see the actions being executed in real-time, eliminating the need to ask about status updates or determine who is performing which tasks. In our remote work environment, it’s vital to maintain these open communication lines. GitHub operates without physical cubes, and a significant portion of our team is completely remote, allowing us to hire talent from anywhere.
00:15:04.880 ChatOps proves invaluable during outages or any scenario requiring tactical responses. People often fall into the habit of focusing solely on tasks in silence, making it difficult for others to know what’s happening. However, when responding to incidents using Hubot commands, that communication occurs naturally, creating a flow of information.
00:15:49.839 Another compelling advantage of ChatOps is the ability to mask ugly interfaces, allowing us to construct streamlined interactions. For instance, we have Alerts in place that notify appropriate personnel in a non-intrusive way, making it easy for developers and Ops engineers to engage without cumbersome systems.
00:16:23.800 Moreover, we've integrated Hubot with PagerDuty, enabling commands to easily gather information. One feature allows personnel to temporarily take on pager duties (for example, if someone needs to step away) without going through complex processes. This has cultivated a culture of sharing responsibilities as people give and take on these alerts as needed.
00:17:23.000 One last advantage of ChatOps is getting mobile support for free. With the Campfire app, it’s easy to monitor alerts or services directly from your phone, making it far simpler to address any issues that may arise—even while traveling.
00:18:06.000 ChatOps ultimately makes it easier to be on call and to interact with your infrastructure effectively. To summarize, Hubot embodies the spirit of ChatOps perfectly.
00:18:36.080 The Bots, both in spirit and functionality, summarize the utility and potential benefits that can significantly enhance operational efficiency. As we continue to develop this practice, the rewards keep evolving, benefitting everyone involved.
00:19:03.960 So to get started, head over to hubot.github.com, where you’ll find instructions on how to create your own Hubot. Thank you for your time, and if you ever want to utilize ChatOps fully, you can always come work with me at GitHub.
JN
Jesse Newland
1 talk
Explore all talks recorded at MountainWest RubyConf 2013
+28