Talks
ChatOps at GitHub
ChatOps at GitHub
by Jesse NewlandThe video titled "ChatOps at GitHub" features Jesse Newland discussing how GitHub's operations team effectively uses ChatOps, specifically through the implementation of Hubot and Campfire, to streamline and automate their workflow. The talk, presented at the MountainWest RubyConf 2013, highlights the challenges of keeping pace with rapid feature development while ensuring stable infrastructure changes. The key points from the presentation include:
Definition of ChatOps: ChatOps is presented as a practical method to automate operational tasks by using a bot called Hubot within group chat environments like Campfire.
Evolution of Hubot: Initially a simple Campfire bot, Hubot has developed into a primary control service for infrastructure management, capable of more than just playful interactions.
Operational Workflow: Jesse explains a typical deployment cycle involving updates to their internal Git package, showcasing how they utilize Puppet through Hubot in a controlled rollout process.
Continuous Integration: The integration of CI bots within the chat allows for real-time feedback on branch builds, and Hubot can initiate deployments once confirmation is received.
Real-time Visibility: One of the significant advantages of ChatOps is that team members can view all actions taken in real-time, which enhances communication and reduces the need for extensive documentation or onboarding.
Actions and Monitoring: Besides deployments, Hubot performs various tasks such as checking the status of branches, monitoring server loads, and handling alerting via PagerDuty.
Incident Management: The transparency afforded by ChatOps during incident response helps maintain a clear flow of information, which is crucial, especially in remote working environments.
Cultural Shift: Adopting ChatOps nurtures a culture of shared responsibilities and collaborative problem-solving, allowing tasks like duty swapping to happen seamlessly.
Accessibility: The mobile capabilities of Campfire provide teams the flexibility to manage alerts and services on the go, enhancing operational responsiveness.
In conclusion, Jesse emphasizes that ChatOps, through Hubot, fosters a collaborative and efficient operational environment that benefits everyone involved. He encourages viewers to explore Hubot for their own uses, highlighting its potential to enhance operational efficiency at any team or organization.
00:00:20.480
Hey everyone! I'm Jesse, and I work in operations at GitHub, also referred to as DevOps, or whatever the kids are calling it these days. Today, I'm going to be talking about ChatOps at GitHub.
00:00:30.960
So, what is ChatOps? ChatOps is the pattern we've been using to automate common operational tasks with Hubot. Before I go any further, I’d like to see a show of hands: how many of you use some form of group chat, like IRC or Campfire, in your daily development work? Cool! How many of you deploy from that environment? Okay, how many of you deploy from the command line?
00:01:04.280
Alright! Hubot started as a simple Campfire bot, but it has evolved into something much more significant. Despite some people's perception of Hubot as merely a bot that shows funny pictures and cat memes, it serves as our primary control service for our infrastructure.
00:01:09.640
Understanding the principles of ChatOps from an outside perspective can be quite challenging. So, let's start with an example. Suppose we've made updates to our internal Git package, and we believe it’s ready to go. However, we want to roll it out in a controlled manner and establish confidence in it. We use Puppet for configuration management—apologies to the chef folks in the house—yet we utilize Hubot to interact with Puppet. This is how we manage these changes in a ChatOps style.
00:01:42.440
We have a GitHub repository for our Puppet configurations, which is the process we all know well. We check out a branch, change the version number of the package, make a commit, and then push that up to GitHub to create a pull request. While we're writing the pull request in Campfire, we have a CI bot that streams the output of the build. Every branch pushed to GitHub is automatically sent through CI, which is incredibly helpful.
00:02:26.640
Now that you know the branch is green, you can use Hubot to deploy that branch to production. In this case, we update the Puppet master to set up an environment called 'gh13.' Hubot provides deployment output, informing us that we’ve restarted the Puppet master. Puppet has a 'noop' (no operation) mode that allows us to establish confidence when rolling out changes, ensuring that there are no unintended side effects. This mode simulates what would happen if you executed Puppet with this branch.
00:03:00.920
After running that with Hubot and getting the output, we see the version number of Git changes. Now, once the noop gives us confidence that things will work as intended, we roll out the new Git version to one file server, keeping the potential impact of regression small. Hubot displays the output again, confirming that the package has changed, so the new package is live.
00:03:41.039
Next, we check that everything is functioning correctly. We've added support in Hubot for displaying graphs from Grafana and Graphite. We request a graph from Hubot and verify that everything looks normal and within acceptable bounds. If we're confident that this version looks good, we proceed to roll it out everywhere.
00:04:06.760
From there, we merge the pull request, letting everyone know what happened. CI runs again to ensure that the master branch is green. Once CI passes, the master branch is automatically deployed to production. Hubot keeps you informed that you were the one who merged the pull request, even though Hubot executes the actions for you. When it's all done, you receive the same notification.
00:04:34.480
Now that the new version of Git is on the master, we can push it everywhere Puppet runs. Puppet operates on an interval, akin to Chef, but sometimes when we are doing a controlled rollout, we want to force it all at once. So, we roll it out everywhere, and again, Hubot informs us that the package version has changed across all different file servers.
00:05:10.640
To assess the impact of the deployment, we check the load on all file servers which looks good. We peek at the logs to ensure nothing else is broken, and we also check that we are still receiving hooks for post-receive and upload packs, confirming that everything is functioning as expected.
00:06:38.240
So, that's ChatOps in a nutshell. It’s quite simple. Essentially, it's about doing all the things you do daily in Campfire. Hubot is the star of our Ops team, performing numerous tasks for us every single day. We interact with Hubot to work with other simple tools, APIs, and various scripts we have created.
00:07:05.120
All these internal tools engage with Hubot, but it also interacts with external APIs seamlessly. By utilizing Hubot effectively, we’ve seen distinct advantages, particularly as our use of Hubot within Ops has grown. I showed you the entire flow of rolling out a new version of Git across our infrastructure. One exciting aspect of ChatOps is that everyone can see what happens in real-time without needing to tell others that a task has been completed.
00:08:06.320
Everyone is passively aware of these operations happening, and there’s no need for extensive onboarding on how to deploy, as this is immediately visible to anyone in the chat room. We don't just use this method with Puppet; we employ ChatOps for many other tasks in our infrastructure as well.
00:08:33.360
For example, we can check the status of any branch built, deploy a branch of any GitHub app to a specific server, and get graphs of the app's recent performance. We can also monitor the status of Unicorn across all front ends, check the status of a rescue queue, or monitor loads across different servers.
00:09:59.680
We can examine current connections to a front end that we think may have issues since we deployed a branch to it. By comfortably viewing logs for our services on that front end, we can determine if we’ve broken anything, and if we did, we can quickly take it out of the load balancer.
00:10:25.679
We also determine who is on call, allowing us to apologize promptly for any issues we cause. If we're unsure whether we've broken something, we may check Pingdom to confirm if everything is down or if it’s just a minor issue.
00:11:05.000
Once we identify which branch caused the problem, we can revert to the master on the affected front end, check the logs to ensure everything has returned to normal, and feel accomplished when we fix the issue. Once verified, we can reintegrate the corrected front end back into the rotation and update the status site.
00:11:58.840
After resolving any outages, we might notice unusual activity, such as a strange IP address linked to a spammer. This could lead to some light-hearted conversations among the team while we reflect on our responses during incidents.
00:12:09.200
All these interactions make up the ChatOps culture—building tools that simplify infrastructure operations, allowing easier access to Hubot rather than coding in a terminal or using a web browser.
00:12:29.880
This is the essence of the talk: by integrating tools directly into conversations, everyone is collaborating and learning continuously, even if they are remote. Ryan Tomo, the creator of Hubot, had this vision since the initial commits to Hubot. It started as a simple repository of shell scripts for managing our infrastructure but has evolved into a crucial component of how we conduct training.
00:13:57.000
When explaining how to perform a task in Campfire, I typically demonstrate it in real-time, allowing others to see the process. This transparency leads to better communication of efforts; when performing tasks in a terminal or dashboard, you have to verbally inform others afterward.
00:14:20.240
By using Hubot in a chat room, everyone can see the actions being executed in real-time, eliminating the need to ask about status updates or determine who is performing which tasks. In our remote work environment, it’s vital to maintain these open communication lines. GitHub operates without physical cubes, and a significant portion of our team is completely remote, allowing us to hire talent from anywhere.
00:15:04.880
ChatOps proves invaluable during outages or any scenario requiring tactical responses. People often fall into the habit of focusing solely on tasks in silence, making it difficult for others to know what’s happening. However, when responding to incidents using Hubot commands, that communication occurs naturally, creating a flow of information.
00:15:49.839
Another compelling advantage of ChatOps is the ability to mask ugly interfaces, allowing us to construct streamlined interactions. For instance, we have Alerts in place that notify appropriate personnel in a non-intrusive way, making it easy for developers and Ops engineers to engage without cumbersome systems.
00:16:23.800
Moreover, we've integrated Hubot with PagerDuty, enabling commands to easily gather information. One feature allows personnel to temporarily take on pager duties (for example, if someone needs to step away) without going through complex processes. This has cultivated a culture of sharing responsibilities as people give and take on these alerts as needed.
00:17:23.000
One last advantage of ChatOps is getting mobile support for free. With the Campfire app, it’s easy to monitor alerts or services directly from your phone, making it far simpler to address any issues that may arise—even while traveling.
00:18:06.000
ChatOps ultimately makes it easier to be on call and to interact with your infrastructure effectively. To summarize, Hubot embodies the spirit of ChatOps perfectly.
00:18:36.080
The Bots, both in spirit and functionality, summarize the utility and potential benefits that can significantly enhance operational efficiency. As we continue to develop this practice, the rewards keep evolving, benefitting everyone involved.
00:19:03.960
So to get started, head over to hubot.github.com, where you’ll find instructions on how to create your own Hubot. Thank you for your time, and if you ever want to utilize ChatOps fully, you can always come work with me at GitHub.
