00:00:09.519
Good morning! My name is Josh Nichols. Here's a picture of me wearing a tuxedo and wielding a sword. This was from an awesome wedding, incidentally. I work at Rails Machine.
00:00:16.480
Officially, I'm the Chief Technology Officer, but unofficially, I'm the Chief Technical Pickle.
00:00:26.960
Today, I'm going to be talking about ChatOps. Now, the first thing you might be thinking is, 'What the heck is that?' What is ChatOps?
00:00:33.680
Whenever I come across a new term, I usually think, 'Okay, I'll look it up in the dictionary.' Well, it's not going to be there, so let's try breaking it down into these two parts.
00:00:40.559
The first part is 'chat.' Now, when you think of chatting, you might think of talking like I’m talking to you right now, using spoken words, which can sometimes be slow and inefficient.
00:00:46.239
However, that's not what I'm talking about. I'm referring to electronic chat – using platforms like IRC, Campfire, Jabber, or any of those tools. So when I say 'chat,' I mean internet chat.
00:01:05.280
The second part is 'ops,' short for operations. This is not the kind of operation where you have a doctor slicing you open, nor is it about servers. If you know me or Rails Machine, you might initially think I’m referring to server operations, but that's not it.
00:01:16.320
I’m talking more about the day-to-day operations of running a business; the essential tasks that keep everything running smoothly. For example, if you have a product, you need to support your customers, handle development, manage sales, or if you are a consultancy, work with ticketing systems and customer interaction. At Rails Machine, we deal with servers, applications, code, Rails, and all that good stuff.
00:01:39.680
So when we put them together, what do we get? ChatOps. It involves using electronic chat for day-to-day business operations. But that, in itself, could be pretty boring.
00:01:44.720
If you were just using Campfire for all that, it wouldn’t be worth discussing. So what makes it interesting? What if let's say you're discussing business in chat and that discussion includes a robot helper to provide additional context or even do some of the tasks for you?
00:02:06.080
I like to say, 'Anything is possible with enough code.' The essence of ChatOps is using code to create a robot helper that resides in your chat, assisting you with your work on a daily basis.
00:02:50.160
So what's the big deal? That's why I'm here to hopefully provide you with insights into what it actually looks like and why it's interesting. Before we delve into that, I'd like to share a story about Rails Machine and how we arrived at ChatOps and how we're using it today.
00:03:11.920
Rails Machine has been around for a while; I believe we are one of the first Rails-specific hosting companies. Everything began quite simply, with a Virtual Private Server (VPS) configured for running Rails. We also had the Rails Machine gem, which was akin to the idea of Rails having a fifteen-minute blog.
00:03:36.000
We had deployment in 50 minutes; it achieved solid outcomes. However, as our customers grew and their needs became more complex, we had to adapt. They required additional support and more hands-on assistance, which led us to offer what we call managed hosting.
00:04:03.920
In addition to providing physical servers, we started offering consulting to help take a Rails app from development to production, including deployment and monitoring. If a system goes down in the middle of the night, our customers won’t need to worry about it; we’ll handle it and help identify what went wrong and how to get it back online.
00:04:25.919
Additionally, there’s a lot that goes into managing Rails applications in production, and we offer operational consulting to help ensure that they are available, performant, and capable of growing consistently.
00:04:44.000
However, we’ve faced some challenging times recently. Rails Machine has always been a small company; our current team is just four people while we serve hundreds of customers. We have been small for a long time; I don’t think we’ve ever exceeded ten employees.
00:05:02.800
About a year ago, we had a situation where three of our four engineers opted to take different professional opportunities. The only engineer left is me, with two thumbs, standing here to give this presentation.
00:05:24.320
During that tough period, which was certainly challenging, I realized the true value of our employees lies in the knowledge they possess. It's not merely about knowing how to complete tasks, such as being proficient in Rails or Apache; it's about understanding how the business operates.
00:05:53.680
When you lose that knowledge, you can manage to keep going with what you have, but it's comparable to having a blank area in your brain. It happens in companies; you must be resilient and keep moving forward. All this while, we are still trying to deliver quality service to our customers.
00:06:06.080
We were handling support tickets, on-call work, monitoring, consulting, building tools, and engaging in open-source development. Broading all of these tasks into one representation is cluttered; it's tricky to visualize all that happens on a daily basis. The core of the issue is that there wasn't a single dashboard or centralized system to manage everything.
00:06:36.000
This made it increasingly difficult; it felt almost as if we were trapped. When you’ve got numerous tools, learning how to use each effectively and interacting with them can become a grueling task. I realized I had to remember multiple URLs, search for items, click, copy, and paste information from one tool to another.
00:07:06.880
We managed to navigate through the challenges thanks to our outstanding staff, but losing that talent only compounds the issue. If you're short-handed, the feeling of being constrained becomes overwhelming, and I can deeply relate to that.
00:07:27.480
Personally, I felt like I was trapped; I was struggling and overwhelmed. The toll it took on my personal life was tough; I wasn't sleeping well and gaining weight, and I hadn’t been on a date in years—it was just awful. I felt stuck in carbonite, devoid of options.
00:07:48.240
In a way, it felt pretty dark. If Rails Machine had a movie saga, we were at the end of 'The Empire Strikes Back'. However, things were set to improve, and I wouldn't be here today if they hadn’t.
00:08:08.160
This challenging situation persisted for several months. Earlier this year, it became evident that we needed to make a change. Our CEO, Bradley Taylor, came in and declared that 'This has to change.'
00:08:20.319
I'll never forget that morning when I woke up and checked my email and saw a subject line that read 'New World Order'. This email outlined a new division of responsibilities: 'Kevin, you're doing this. Ernie, you're doing that. You’re no longer responsible for this aspect.' It was amazing how a slight shift in perspective made things so much better.
00:08:44.480
The only odd thing about that email was that my name wasn't mentioned at all, which momentarily made me wonder if I still had a job or what was happening. Fortunately, it turned out I just needed a break.
00:09:06.240
For a while, I stepped away to gather my thoughts, and eventually, I decided to recharge—perhaps hanging out in a hammock on the beach could help me refocus my life and work.
00:09:26.560
However, I found it challenging to disconnect because I tended to check support tickets—only to remind myself that wasn’t my responsibility anymore. It was tough, but eventually, I started contemplating the solution, which led me to the concept of ChatOps.
00:09:39.840
After discussions with Bradley and my friend Jesse Newland at GitHub, we came to the idea of utilizing chat as a platform for managing tasks. We didn’t have a centralized dashboard to navigate tasks, but we constantly used Campfire in our daily operations, so what if we could leverage that platform?
00:10:03.520
I needed to formulate a plan and define some goals before embarking on this journey. I wanted the work to be enjoyable, to give me a sense of joy and fulfillment. My heart needed to glow with joy, not just in terms of the development but also in terms of sharing our work and achievements with others.
00:10:34.720
I wanted this workflow to be iterative because the workload at Rails Machine fluctuates considerably. We do consulting, and tasks come and go unexpectedly; it's challenging to pin down a week’s worth of work on a single project at a time. Hence, I needed something feasible that I could work on steadily each time.
00:11:01.920
Moreover, I wanted the ChatOps process to be well-documented—not just in a plain text file, but more interactively so that its functionality explains itself. Whenever someone uses this ChatOps tool, it should help new users seamlessly understand how to participate.
00:11:26.560
Now with those goals in place, I could explore each individual task to see if it aligned with our goals. If a task didn’t meet those criteria, I would reconsider whether or not to pursue it.
00:11:58.240
Several months prior, GitHub had released their Campfire bot, Hubot. After eagerly discussing it, I finally realized it was an obvious path for us to follow. We had a Campfire bot built with a Ruby library called Fire Tower, but I wanted to move on from that.
00:12:09.440
The original bot wasn’t very enjoyable to work with, so I ended up taking over maintenance of it. Hubot's use would offer itself as a much more enjoyable project and a way to engage a wider community, where others have been blogging about its functionality.
00:12:27.200
There was a repository of scripts out there, some of which are whimsical and delightful, while others are genuinely useful. I didn’t expect that, after getting comfortable with Hubot, I might end up as a maintainer for the bot itself.
00:12:47.440
Now, rather than attempting to do live coding here, I can demonstrate Hubot’s functionality live. Unfortunately, I won’t have enough time for an in-depth 'getting started' session with Hubot, as that could take an entire talk by itself.
00:13:10.080
However, I’ll give you a rapid glimpse into how it functions. Hubot is a Node package, and we have a package.json file for managing dependencies in Node. This file outlines the required dependencies for our Hubot implementation.
00:13:41.680
We’ve a Hubot scripts file that lists all the functionalities we want to implement. The Hubot setup engages with a source scripts directory, from which it utilizes Coffee or JavaScript files.
00:14:07.920
Having already set my goals, I can then think of how to ask questions in Campfire and how to answer them. For instance, it’s a workday morning where I’m a bit tired, contemplating whether I want to make a latte or iced coffee.
00:14:30.720
I want to know if it’s good weather for iced coffee or if I should stick to hot drinks. There’s this website called 'Is It Iced Coffee Weather?' that serves this purpose. Unfortunately, they don't have an API, so I’ll have to do some simple screen scraping.
00:15:01.840
Therefore, I wrote a script to retrieve this information using the Cheerio library to extract the necessary elements from the website to respond to my inquiries.
00:15:34.080
The bot responds to the command with whatever relevant output it gathers. Initially, I wanted to make sure I provide a means for the bot to respond accurately based on my current location since it doesn't know automatically.
00:15:57.680
Once I have a functioning bot, it's essential to think carefully about its name. As a fan of Borderlands, I decided to name my bot Claptrap. So now I can ask, 'Is it iced coffee weather?' and Claptrap replies, presuming that the bot is alive.
00:16:21.680
This interaction is simple and powerful, providing me with the necessary information promptly. Now that I have the coffee query sorted out, it’s time to delve into work-related monitoring.
00:16:46.680
I need to check the status of certain servers and ask if they are being monitored, how much memory they have available, and so forth. We have an internal application called Imperial Probe Droid that tracks the state of these servers.
00:17:11.680
If I need to know about the Claptrap server, I can easily search within this application to find a concise overview of its status. This system will help me establish whether or not it's being monitored effectively.
00:17:32.240
However, the traditional way of looking this up involves logging into the app and running manual searches, which can be tedious. Wouldn’t it be much easier if I could just retrieve this crucial data directly from Campfire instead?
00:18:07.200
To make that happen, I wrote a script that pulls the necessary information. The method follows a boilerplate structure to ensure clarity and conciseness throughout its function.
00:18:35.760
Within the script, I ensure documentation is available for user guidance. Creating a range of command styles with regular expressions facilitates interaction with the bot. This allows it to respond depending on user input accurately.
00:19:00.720
Additionally, there are various methods available within the context of this application to assist with querying and gathering relevant data. This helps streamline the process of obtaining information rapidly.
00:19:27.680
Next, we have a paging system, known as PagerDuty. This system ensures we have coverage, so someone is always on call to address issues as they arise.
00:19:50.160
It's beneficial to inquire who is on call, so I implemented a way to ask Claptrap who is currently responsible for monitoring. This functionality adheres to similar patterns established in previous scripts for consistency.
00:20:12.640
We also made the decision to automate announcements, notifying the team whenever someone comes on shift for monitoring. This helps ensure everyone is aware without needing to query the bot constantly.
00:20:39.680
Additionally, we rely heavily on New Relic for performance monitoring. It allows us to gauge the health of our servers and address slow performance issues. However, it can be challenging to keep track of which customers have active New Relic accounts.
00:21:05.680
By implementing a script that enables us to identify and find information about customer accounts, we can minimize the hassle of manually querying New Relic and gathering insights.
00:21:32.880
This allows us to effectively ask if a customer has a New Relic account and quickly obtain the necessary details to correlate back to their application.
00:22:00.320
As we are making developments, I wanted to conclude with a fun aspect. Our office environment often features shared audio speakers, and we frequently receive inquiries about the current song playing.
00:22:28.480
A fun script allows us to discern the current song's name by utilizing some form of audio recognition or track identification to provide quick answers.
00:22:58.320
Through interactions involving metadata about the songs being played, we generated an implementation that eagerly recommends information back to users within our chat application.
00:23:26.960
Encouraging a fashionable culture via data sharing and enthusiastic collaboration makes working with our tools much more enjoyable.
00:23:57.120
In conclusion, ChatOps has transformed our day-to-day tasks into more manageable and interactive experiences, which contributes significantly to morale and productivity within our team.
00:24:07.920
So the main takeaway from my presentation is to enjoy what you do while progressing iteratively through challenges. Since adopting ChatOps, I’ve shed 30 pounds, met an amazing partner, and now serve as CTO—things are going really well.
00:24:57.040
If anyone is utilizing Hubot, please reach out to me! I’d love to hear about your experiences. Here’s my contact info: joshuailsmachine.com and don’t forget to enjoy our technical pickles!
00:25:22.400
Are there any questions?
00:25:31.440
Regarding the question about my gaming adventures, I have unfortunately not played Borderlands 2 yet, but I look forward to it once I return to the office.
00:26:07.440
There was also a question about keeping the bot active. We use ‘god’ for monitoring the process, which ensures it stays alive, but disconnections can still occur occasionally, something I'm actively working on.
00:26:36.960
If there are no more questions, I want to express my gratitude to you all for being such an engaging audience.
