Black-boxing with Ruby

by James Britt

In the talk titled Black-boxing with Ruby, James Britt shares his experiences of integrating Ruby with non-Ruby applications, emphasizing the notion of 'black boxing'—interacting with external applications without direct access to their inner workings. Meanings include:

Introduction to Black-boxing: Britt defines black boxing as interacting with applications that do not cater to specific API needs, often relying on their external functionality to solve problems.
Personal Background: He has been involved with Ruby since the early 2000s, maintaining several Ruby-related websites and exploring various tools beyond Ruby.
Practical Applications: The talk shares the need for tools like Track—project tracking software—and WordPress—a blogging platform. Britt notes that these tools, despite not being Ruby-based, have features that significantly benefit his projects.
Challenges and Solutions: He discusses the allure of building project tools from scratch, acknowledging that it can lead to unnecessary complexity and effort. Instead, Britt advocates for leveraging existing solutions, arguing that communal maintenance and updates are better than going solo.
Case Studies with Track: Britt details how he created a command-line interface (CLI) to interact with Track, simplifying the ticket management process without the cumbersome UI of web forms. Through Rake tasks, he developed tools enabling easy ticket submissions and retrievals, thus keeping his workflow efficient.
Learnings from WordPress: Britt talks about using WordPress for a commenting mechanism for his online book after discovering its advanced comment handling capabilities. His approach involved manipulating HTML elements and sending requests to WordPress, showcasing creativity in utilizing non-Ruby tools effectively.
Conclusion and Insights: Britt encourages experimentation and curiosity within development, urging others to utilize existing tools when they offer clear benefits. He concludes on an optimistic note, stating the successes in connecting Ruby with other technologies enhance productivity and open up new avenues for innovation. His experiences provide a balance between leaning on established platforms while still embracing Ruby.

00:00:07.839 Hey, my name is James Britt. I'm going to talk a bit about what I call black boxing with Ruby.

00:00:14.799 A little bit of background about myself: I run some Ruby sites, probably most notably Rubydashdoc.org, and I've been writing Ruby code since about 2000 or 2001. I've also written a number of Ruby articles and parts of books.

00:00:28.400 The basic idea of what I want to discuss is sort of an overview, recounting some experiences I’ve had while trying to work with applications, primarily web applications, that aren’t written in Ruby.

00:00:34.640 These applications don’t necessarily expose the kind of API or interface that I want. Yet I want a decent Ruby API from my client code, say, for command line tools or some other applications.

00:00:42.480 So the definition here of black boxing is just interacting with some other app that doesn’t have my particular interests in mind. For my purposes, a black box is just an unknown entity that you may only know how it works from its outside behavior.

00:00:54.480 You might not have any access to its internals or any opportunity to really hook into it. I think, for our purposes, this is a good enough definition.

00:01:13.119 The story behind this is that I needed some sort of project tracking tool. In another case, I wanted some kind of commenting system for a particular application. In general, I liked a whole bunch of stuff that just wasn’t written in Ruby.

00:01:25.600 There’s a lot of nice stuff out there. There’s Track, a project tracking tool that some people have mentioned and I know a lot of people use WordPress for various things, which is actually quite good if you’re doing blogging or certain kinds of presentation work.

00:01:43.759 I like Amazon’s EC2 Elastic Compute Cloud for various things. There are some Ruby applications for it, but they aren’t quite as complete as I’d like. One of the sites I run, Rubystuff.com, is a portal aggregator for a bunch of Cafe Press shops.

00:01:56.320 All these cases are things I want to work with using tools that were written from moving my end to things that weren’t necessarily Ruby-friendly on the other end.

00:02:07.120 One question that comes up when you talk to people about this is: Why not just build everything myself? Right? If I can’t find a project tracking tool that has just the right API I want, why not just build it?

00:02:28.959 The reason is, well, because that’s just insane! I know that many people, when they work on projects, half the fun is building the tools to help you build your main project.

00:02:48.080 But it’s a serious milestone you can just chase forever, spending more time building tools than actually building your project, and I’ve been there and done that. So I don’t really want to chase that all the time.

00:03:06.000 Why not use a Ruby app? Well, the truth is, there’s not always a good selection available. People do talk about writing a Ruby substitute for Track. There are Ruby options that aspire to what WordPress offers.

00:03:21.200 But I’m not sure if they’re quite there. Even if they are, there’s another aspect to this which is just plain mind share.

00:03:33.920 Even if you find an application written in the language of your choice, when a bug is found, when there’s a security problem, or when there’s a feature request, how many people are going to be out there to fix it or to contribute?

00:03:46.000 I’m pretty confident that if I’m using Track and there’s a problem, all those developers are going to jump on it. The best part is I’m not going to be one of those developers having to fix it, which leaves me free to focus on the things I want to do.

00:03:58.000 So what it really comes down to is that when you need a particular tool, it’s secondary to what you’re mainly trying to accomplish or just has some particularly useful feature.

00:04:14.400 Yes, it’s nice when you can do everything in pure Ruby, but the reality is that isn’t always what’s going to work best for you. You needn’t be blinded by this kind of language fundamentalism.

00:04:21.199 Basically, the P languages are not the work of Satan, but don’t let that get out of this room, okay? I’ll deny it.

00:04:33.919 Another big motivation for me personally is the curiosity of wondering if you can take this and hook it up with that, and experiment with various combinations. Anyone who had a chemistry kit growing up knows what happens when you mix the blue stuff with the red stuff.

00:04:50.240 A lot of that thinking applies to mechanical devices too. Oddly enough, I grew up being known as the one who always broke stuff, even though I was also the one who always fixed stuff.

00:05:04.960 I view a lot of what I do as an opportunity to experiment with certain ideas.

00:05:10.880 When you do things like that, you end up in odd development territory and wind up learning or doing things you wouldn’t have considered if you were to take perhaps the 'right' path or the obvious route.

00:05:31.199 So keep that in mind as you go through this.

00:05:36.720 I think for most speakers when they give a presentation, this is what they want from the audience. Right? This is what maybe it’s all about.

00:05:42.720 That’s fine, but this, however, is the path to enlightenment, so I’m good with that.

00:05:53.440 Let’s start off with Track. Track is an excellent project tracking tool. You can argue with me afterwards if you want, but I’m quite happy with it.

00:06:05.120 It does basically what I want, gets out of the way, has nice features, lets me look at my SVN repo, has a wiki, accepts commissions—it works.

00:06:15.360 But there’s way too much mousing required for many things. If you’ve ever tried to add a ticket to Track, you know you have to open a browser, click on stuff, and go through all of that.

00:06:26.880 It’s fine for one-off things, but it is particularly useful if you open up your Track repo to clients and give them an opportunity to submit things.

00:06:38.960 However, it wasn’t really what I wanted to do, and just in general, web pages are poor data entry tools.

00:06:55.120 So I asked myself: Hey, I wonder if I could use Track from the command line? The situation really was this: I find myself working on projects, coding away, and when I come across something that I see as a problem.

00:07:12.240 I don’t want to fix it right away, but it’s also not the main problem I’m facing right now. Perhaps I notice that the CSS alignment on a certain page is just wrong. I want to make a note of it.

00:07:27.680 I don’t want to write it in a temporary notes file and then later on remember to post it on Track. What I wanted, while at the command line, was to type it out and have it magically appear on Track.

00:07:43.840 So that was a practical thing I needed to do, so I wouldn’t break the flow of my work. And now the answer is: Yes, you can use Track from the command line.

00:08:01.599 This is an application/library I’ve written to interact with Track, primarily from the command line and through Rake tasks.

00:08:14.720 Let me say that I actually get this to work. I tried to do some screen capturing, as I don’t like live demos. The medium of live demos has a really bad history.

00:08:30.640 So I’ll see if I can get this to perform well enough. To give you an idea of what it’s like, you can type a Rake task, track: tickets.

00:08:45.000 It goes up to the server and pulls back a list of all the open tickets. That was something I just wanted to have available at any time while working on something.

00:09:02.960 Now, likewise, I want to be able to add tickets from the command line through a series of prompts.

00:09:14.720 Here’s another high-tech audiovisual presentation: similar to a Rake task, you can type track: ticket.

00:09:29.440 It prompts you for a bunch of information, and there are some reasonable defaults I’ve put in there.

00:09:44.880 This may look practically unreadable, but trust me—there’s a gracious prompt asking you what milestone it belongs to and what the component type is: is it a task or a defect?

00:09:59.000 It will then post it back up there, and that’s exactly what it does.

00:10:10.880 Personally, I found that much nicer than having to open a web browser every time I have a ticket.

00:10:27.520 Even if you leave the browser open, it just wasn’t what I wanted to do.

00:10:39.840 Some of the capabilities show how it works. It’s not rocket science—it’s just HTTP mechanized, going up there and pretending to be a web browser.

00:10:57.599 It goes up there and simulates a request. There’s a config file involved that loads your username, password, and the URL for the Track repo, along with a couple of other things to make it easy.

00:11:14.560 This setup worked well for me because I’ve gotten to interact with an application ostensibly written in Python.

00:11:31.680 I’d be lying if I said I didn’t have to actually look at the Python; I did.

00:11:45.760 So there’s the URL if you want to poke around. I realize that URL isn’t going to tell you how to get the code or the gem, but I will have to update that.

00:11:59.360 It’s kind of raw and in a beta/alpha state, so you can let me know if and when you find problems with the code.

00:12:15.920 This is great—it solved my problem. However, there were some glitches with it. One thing I noticed is that every time the Python developers updated Track, it was different.

00:12:31.920 It was like it was new, and new enough in different ways that it would break things. When I first started with Track, the form field for submitting a ticket was quite straightforward.

00:12:45.839 I had to change servers, re-install Track, and when I went back to use it, it just wouldn’t work.

00:12:58.120 It turns out there’s now a hidden form field that performs verification against a cookie, which has a grid for your specific process.

00:13:14.960 Fixing that was not phenomenally difficult, but it was just an annoyance.

00:13:29.680 As I spent more time with Track, I became more aware of plugins. I figured out the Python egg installation process and how to get things done right.

00:13:44.720 There’s quite a lot of nice stuff available. For example, most of my Track sites are on a URL with HTTP basic authentication, using .htaccess files to restrict access.

00:13:59.160 But for the Track site, I wanted a general open page and a nicer login. Track has a plugin that handles that, but the irony is that Track Dracula doesn’t actually work against the Track site.

00:14:20.040 This is largely because of a different means of authentication. So, there’s a certain amount of brittleness introduced when you try to do this sort of thing with a web page.

00:14:35.680 I guarantee that the people maintaining the page aren’t thinking of what they’re presenting as an API. For them, it’s just whatever it is that people click on.

00:14:55.040 Some lessons learned: First, HTTP mechanized truly kicks ass. If you haven’t spent any time with it, you really should.

00:15:10.920 If you’re manipulating HTML that comes down from pages, they do a really good job of fixing randomly sloppy HTML.

00:15:25.040 You can reach in and grab images, forms, tables—whatever information you need.

00:15:39.600 Second, if it’s on the web, it has an API. Granted, yes, it’s fragile, but at least consider that as you look at things.

00:15:56.240 Anything that’s out there that you can access through a browser, particularly with tools like HTTP Cad and mechanized, should lead you to think of how to automate the interaction with this website.

00:16:09.840 When you fetch a page, you’re just getting text back. It’s text that follows some well understood rules. You can grab it, manipulate it, and post it back.

00:16:25.200 The tools available now in Ruby for this have gotten really good. But yes, there’s fragility to deal with, as someone could decide to shift around where the tables are or change what a form field is called.

00:16:42.000 Everything can kind of fall apart, but the upside is it is very easy to get started. It didn’t take me that long to get a basic version of Dracula up and running.

00:16:55.680 Most of what I ended up doing afterward was just trying to clean it up so I could open it up and let others use it.

00:17:09.520 To echo what Gregory was talking about earlier, there’s real motivation in doing that. While I wrote this for myself, I would like to see it do more.

00:17:23.200 I don’t want to be the only person working on it. It would be useful to others. To the extent that I can make an effort to get others interested and give back, it becomes more valuable.

00:17:37.120 Even those who are working on projects—a lot of times when you start out to build something just to scratch your own itch, it’s about self-interest.

00:17:51.440 There’s a way to look at this to make it worth your while to get some docs up, have a nice page, and make it easy for people to get started.

00:18:04.720 So yes, there are still some features missing—that’s okay. I’ll add them when I need them. Basically, what it does for me now is I can get a list of tickets, post a single ticket back from the command line.

00:18:18.000 Another very useful thing is I can create a YAML file that describes a whole slew of tickets and post them in bulk.

00:18:35.440 This was valuable because there is a plugin for Track that lets you attach time estimates and figure out velocity for your projects.

00:18:50.560 I found it very useful when going through a spec, as I had a contract to run down a list of requirements and start putting times against them.

00:19:06.720 I would then post the whole thing up into Track, let it figure out my estimates, and work off of that. It seemed like a reasonable way to store information without having to type it repeatedly.

00:19:24.640 However, the idea of entering all these tickets manually was just too much work for a lazy person like me.

00:19:39.680 I’d rather invest that time and energy into writing code. Now, yes, I could sit down and make a big bug list of what’s wrong with my project, type it into a simple structured YAML file, and have it all uploaded.

00:19:55.680 What Dracula does is it loops over the YAML file, creating single postings as it goes without me having to worry about it.

00:20:09.760 Another problem here is that I haven’t found an easy way to request a list of current milestones from Track. So I cheat: I go on the web page, handcopy it, and post it into my config file.

00:20:24.000 This way, when Track runs, it knows what to display. It’s handy because I found that automatic posting would fail if I did not provide a proper milestone for any ticket I was submitting.

00:20:39.840 The code now ensures that I provide a milestone that makes sense or it doesn’t bother trying to post at all.

00:20:54.240 One nice thing at the end of doing all this is that by using such a tool and avoiding trying to go against Track directly from any particular piece of code.

00:21:06.960 When the time comes that Track changes or adds new features, I simply have to change one library. I only need to update that in my client code.

00:21:21.680 All my different projects that use these Rake tasks will hopefully remain oblivious to any changes.

00:21:34.479 This helps buffer a lot of the changes that would occur. You can at least isolate the fragility of working against someone's web page as an API.

00:21:49.440 Ultimately, Track is going to have a plain XML RPC interface as a standard part, if it isn’t in there already. When that time comes, it will make things simpler.

00:22:06.720 But at least I don’t have to wait for that point.

00:22:20.720 The next application I wanted to work with was WordPress.

00:22:34.239 Years back, I started working on a book titled 'Beginning Ruby'. It was just a bad time to be working on books.

00:22:47.920 I finally decided to put this on the web and wanted to allow people reading the book to leave comments on the content.

00:23:01.919 So I decided to roll my own commenting system, starting out with a Nitro application for publishing that uses a form of enhanced Textile and hooks into its own commenting system.

00:23:17.440 I had gotten to a point where it was pretty much done when I came across the Django Book.

00:23:34.080 Now you might think I’m talking about doing it in Django, but no! The Django Book has an amazing commenting system.

00:23:50.720 If you go to djangobook.com, they have this online book, and while it’s in development, you can leave comments with a super sweet AJAX interface.

00:24:04.800 When I saw this, I thought it was a million times better than what I was doing, so I decided to steal it.

00:24:18.480 I contacted the author of the Django Book, said, 'Dear author of the Django Book, I’m going to steal your code. Are you good with that?'

00:24:36.680 He replied positively, but I should note that Jack Slocum had written extensions to the Yahoo UI library, which powers much of this stuff.

00:24:55.760 So I dropped by his site and gave him a donation.

00:25:14.000 I grabbed the code and started working on it. This is what the Django Book looks like; if you click on any of these things, you get a nice little comment box that stores the comments for that chapter.

00:25:31.320 However, it wasn’t quite what I aimed for.

00:25:46.560 In the Django Book, they have the entire chapter ready and available while it collects comments, attaching comments based on position.

00:26:00.560 If you move a paragraph or insert text, all of that information might get lost.

00:26:16.080 My approach, rather, was to give HTML elements a unique ID derived from the content, which is unique enough.

00:26:33.600 The comments will match these IDs, rather than relying on element locations, which could be too fragile.

00:26:47.680 One reason I chose WordPress is that I think it’s among the best for this purpose.

00:27:04.560 I’ve seen other people talk about using it, as it provides a site not meant just for comments.

00:27:20.960 You can easily route users to a WordPress site, which has nice admin tools, plugin support for spam handling, and comment management.

00:27:35.920 Why build all of that yourself?

00:27:50.079 The process I came up with was to fetch or create the source HTML. I ran it through an element processor that examines the page text and rewrites it by adding unique IDs and class attributes.

00:28:05.680 It writes out a new page, and in theory, you could take any plain HTML page that can go through HTTP and turn it into a system layered with commenting.

00:28:20.800 When you visit beginningruby.com and poke around, you’ll see it operates somewhat similarly to the Django Book site, with changed colors.

00:28:36.240 Clicking on the comment box lets you see comments attached to that specific chunk of text identified by a hash key.

00:28:52.320 A bit more about this process with WordPress: the client code runs locally or on the server and modifies element nodes by sending requests to an application I simply called Comet.

00:29:11.200 The Comet engine sends requests to an instance of WordPress, creating new posts with the title of that ID.

00:29:27.360 Now, the comments attached to content on the Beginning Ruby site are actually comments attached to posts on this hidden WordPress instance, with each post related to a specific chunk of text.

00:29:42.160 When a user wants to add a comment, the static page comes off Apache as plain HTML, and the JavaScript that comes down behaves similarly.

00:30:00.480 When you make a request back, it gets routed through mod_proxy to this comment server.

00:30:17.920 You now see two types of signaling going on: just plain HTML delivery and the comment behavior overlay.

00:30:31.280 My professional graphics should clarify this if my explanation wasn’t clear.

00:30:44.880 Any questions?

00:30:58.720 Yes, regarding the commenting system.

00:31:14.880 The system does exactly what I wanted it to do.

00:31:30.160 My thought is, if I change the text, the comments that applied to it might no longer apply.

00:31:44.400 When it fetches again, these could become obsolete. However, it’s a problem I’ll worry about when it arises.

00:32:00.000 Any other questions?

00:32:05.760 Thank you very much.