Postcards from GorbyPuff

RailsConf 2016

by Aja Hammerly

The video titled 'Postcards from GorbyPuff' is a talk by Aja Hammerly, presented at RailsConf 2016. In this talk, Aja introduces GorbyPuff, a fictional character embarking on a world tour, and illustrates how to use various Google Cloud tools such as Cloud Vision, BigQuery, and Google Translate to track his journey.

Key points discussed include:

Introduction to GorbyPuff: Aja introduces GorbyPuff, a character who decides to explore the world while leaving clues for the audience through postcards. Aja emphasizes the playful theme inspired by 1980s educational software.
Google Cloud Vision: The talk covers the capabilities of Google Cloud Vision, including landmark detection and label identification in images. Aja provides code examples that demonstrate how to use the gcloud gem to interact with Cloud Vision API for identifying landmarks and getting attributes about images.
Label Detection Examples: Aja showcases how Google Cloud Vision can recognize various objects in images, sharing personal anecdotes of using it for images of her cats and comparing how accurately it identified different objects based on visual input.
Text Detection and Translation: As Gorby travels, Aja demonstrates how to use optical character recognition (OCR) to identify text in images and how to translate this text using Google Translate API. Aja emphasizes the limitations of OCR and translation technology, noting possible inaccuracies but also the fun of decoding messages from Gorby’s postcards.
BigQuery for Data Analysis: Aja explains how to use BigQuery to analyze real-world data, specifically for the highest temperature recorded on a specific day, by writing SQL-like queries. This showcases the power of analyzing large sets of real-time data using Google Cloud tools.
Conclusion and Call to Action: Aja wraps up by discussing the capabilities of Google Cloud Platform and its compatibility with Ruby applications, highlighting new offerings such as App Engine for Ruby. She encourages the audience to explore Google Cloud tools for their projects and invites questions.

The talk combines humor with technical knowledge, showcasing the versatility of Google Cloud tools through engaging storytelling and practical demonstration.

00:00:09.889 Welcome to my talk, this is 'Postcards from GorbyPuff.' Those of you who are here have decided that you'd rather laugh and learn some things than go to the other talks right now, and I approve of that choice. I'm Aja Hammerly, and my GitHub handle is @mashabongo. All the code for this talk is available in my examples repository under 'railsconf16.' I tweet at @mashabongo, and I really enjoy it when people tweet at me during my talks. My phone is over there, so it won’t interfere with the mic or interrupt the talk. My blog is at mashabongo.com. The slides for this talk are also available on my blog if you want to follow along.

00:00:20.070 I really like dinosaurs, and I work for Google Cloud Platform. We will have a workshop tomorrow afternoon in the workshops track if you want to check out our tools and spend some time working with people who know the platform better. It's a great way to get help. We'll also have a booth, so you should totally come by, ask us questions, and grab some free stuff, because that’s what booths are for.

00:00:50.460 If you're one of those people who doesn't enjoy talking to people face-to-face, I know you're out there. Check out our Slack; we have a Ruby room where several folks who are working the booth will be available to answer your questions and help you out. Since I work for a large company, I must include this slide: lawyer cat says any code in this talk is copyrighted by Google and licensed under Apache 2.0. By the way, if you hadn’t noticed, this is the 'cat pictures' talk — there’s probably an equal amount of cat pictures and code.

00:01:08.490 So, who knows who this is? This is Gorby. He’s a character who lives in Seattle, and the lovely Tender Love is giving the keynote on Friday morning. You should definitely attend because I currently have fewer pictures of Gorbachev than this talk has! I’ve used them all for this conference, and Gorby was upset that he couldn’t come to RailsConf, so he decided to set off on a world tour of his own. In the spirit of 1980s educational software, we will be asking: 'Where in the world is GorbyPuff?' He will be leaving us some clues in the form of postcards.

00:01:58.620 I had an argument with someone who gets the joke I’m making here, about the computer program I’m alluding to, so raise your hands if you know the reference. Okay, enough of you know, so first clue: this statue-type thing. You know, it’s a postcard, so here’s the back of it. GorbyPuff is hanging out with his new stone friend. Tell me, the cars are high in iron. It’s made to figure out where Gorby is. Where's Gorby? Much like in that quintessential 80s computer game, we have access to a super problem-solving computer, and ours is called Google.

00:02:29.430 We’re going to be using Google Cloud Vision for this. Google Cloud Vision is a web API that can label landmarks and faces. It does face detection but does not tell you who the face belongs to. It only tells you it’s a face. It can also label a wide variety of things in images and can find adult content if that is relevant to you for your website. I will also be using the gcloud gem. If you’ve tried using Google APIs with a programmatically generated gem and found it subpar, I recommend that you come back and try gcloud. This gem is designed by Rubyists, for Rubyists.

00:03:04.320 It’s hand-coded. If you know Mike Moore or Chris Smith, who are from Utah, they’ve worked on this gem and have done a fantastic job. It truly feels natural. Here’s the code that you need to look at for the postcard from Gorby. I’m going to walk through it. First, we require the gem, which allows us to use Vision. I create a new gcloud object, specifying two arguments: one is the project, which organizes your resources inside Google Cloud, and the other is a private key that lets me authenticate into that project. This is partly for billing — Vision is not free.

00:03:42.350 I create a Vision object, and I’m going to pull an image. This image is being pulled from local disk, but you could also fetch it from a web source. Here’s the line that does all the magic: I'm calling 'annotate' on the Vision API, passing it the image and telling it that I want to find one landmark. It seems there was only one in that picture. If it finds something, I’ll print out the description using puts; I will also announce a location, which will give me the latitude and longitude because the picture can tell you what it is and where it is.

00:04:09.630 Because it’s been a while, this is what the image looks like. The output indicates that it actually knows that this is the Fremont Troll — the Fremont Troll happens to be about three blocks from the Google Seattle offices. If you’ve seen the movie '10 Things I Hate About You,' you’ve seen this troll, as it plays a significant part in that film. So, where in the world is GorbyPuff? Gorby is right here! We get in our virtual 8-bit plane, fly over, and meet a guy sitting around who tells us a grumpy cat gave him something to pass on to us. Yes, the grumpy cat — he gave us this postcard.

00:04:54.150 The guy mentioned that tender love wanted to hang out with this creature and was very jealous of how not flat its face was. On the back of this postcard, we find an item, wondering if it’s a dog or a rat, as it has such a pointy face. Now, we need to figure out what kind of creature it is. We’re going to use the label detection feature in Google Cloud Vision, which can label various things, including breeds of animals.

00:05:28.680 This is an example: those are my cats, and this was taken about a week ago. They were hanging out on the deck, and I wanted to run this picture through Cloud Vision to see what it would label. These are the labels it provided: cats, probably pets. One of them is mostly black; that's really just the plastic cover on my planters. They are mammals, and the AI indicated that: there are animals in that picture; they're probably playing; they're small to medium-sized cats; one is a tabby cat.

00:05:49.350 The interesting thing is that whenever it provides these labels, it gives a percentage of certainty indicating how confident it is that a given label applies. Here's yet another different example: this was a picture I took while skiing. The labels it generated included: snow, mountain, weather, piste, winter, geological phenomenon, season, and adventure. Since that’s a double black diamond in the background, I was pleased it labeled it as adventurous. If we think of snow, we typically think of it as white, but in this picture, there is very little white due to lighting. However, Cloud Vision, using machine learning techniques, managed to recognize it as snow.

00:06:35.340 In another example, I took a picture of some buses. The labels it provided included: transportation, buses, commercial vehicles, and possibly an engine. I was amazed that I could do all this with just a few lines of code — specifically, just seven lines, including the require statement. This is basically the same setup as before, but the only line that changes asks it to look for labels. I want to clarify that I'm currently using a not fully released version of the library.

00:07:12.680 When they fully integrate it into the gem, all you will have to do is say 'annotate,' and it will find all the interesting information for you. In my use case, since I don't have just a single landmark, I’m iterating over the labels it found and printing them out. Now, back to that postcard: what the heck is it? Some of you probably have a good guess, while others may have no idea. This turns out to be a wombat. Cloud Vision successfully identified it as such!

00:08:15.870 I also tried it with a Pangolin; the software thought it was a turtle or an armadillo, which, given the appearance of a pangolin, is actually a fair amount. While I’m not too upset about that, it did identify the wombat with about 83 to 93 percent confidence. Now, getting onto our virtual 8-bit plane again, we’re off to Australia where we meet someone who mentions, 'I saw a cat matching that description, and he told me to give you this.' It’s another postcard from GorbyPuff. This one shows some sort of sign in a language I don’t currently understand, and on the back, it says that Tender Love left a list of countries he likes to visit.

00:09:04.289 He was just laying around the apartment. Do you have any idea what this sign says? We can use text detection, or optical character recognition (OCR). This technology finds characters in images and determines what they are. It will also figure out the likely language of the text. We’ll run a similar setup as before but change the line from looking for labels or landmarks to looking for text.

00:09:38.339 We will print out the text as well as the locale. The locale is a combination of language and place, so countries with multiple official languages will have multiple locales, while those with only one will only have one locale. Here’s the image again and the text that came back. It’s not great; it’s not particularly accurate. OCR is tricky, especially with images that are not neatly aligned or if they use non-Roman characters.

00:10:06.920 You’ll notice that the locale came back as ja, which stands for Japan — not surprising since Tender Love enjoys visiting Japan and speaks the language. Gorby is trying to figure out where his owner had been, but I don’t speak Japanese. So how can we proceed? Who has used Google Translate? You know the trick — taking text through Google Translate from one language to another and playing the game of telephone. I’ll show you how to do that programmatically using the gcloud gem for translation.

00:10:54.310 Creating a new gcloud object for translation is really straightforward. Running some text through Google Translate is easy; the 'from' argument is completely optional. It usually guesses reasonably well based on the characters. Since I have the locale, I’ll provide it to increase the likelihood of a correct translation. However, if you’ve used Translate, you know it’s a tough problem. Human language is complicated; while it can get close, sometimes it needs a little assistance, and I expect what comes back will be translated text of significant volume.

00:11:58.829 When I analyze what’s returned from translation, I notice it’s more than just the text on this sign; actual text from buildings and signs in the background got pulled in as well. So, I know Gorby has a flower garden, he’s playing on the beach, and he’s a cat in a giant sandbox! Now, let’s go catch up with Gorby and see what he’s doing! Flying our 8-bit plane again — someone informs us that the cat we were looking for was here, but he’s been busy hitting the books.

00:12:33.750 The highlight of this talk was going through Gorby’s Tumblr; he has both a Tumblr and Twitter and even more followers than I do. One of my official company goals this year is to surpass Gorby in Twitter followers! Gorby’s decided to share some trivia with us: What place on earth recorded the highest temperature? It’s actually May 4th, 2000, which is relevant to the slide because Gorby discovered he's not particularly adept at Ruby karaoke, so instead, he's here for Ruby pub trivia.

00:13:53.030 There’s a cool tool at Google called BigQuery. I spoke about it at the Mountain West conference two years ago. It was created internally for processing large sets of logs. I like it because it allows you to query large datasets using SQL rather than a custom query language. It doesn’t require any indexing; it’s like magic. There’s a variety of public datasets to play with; I often use the GSOD dataset that contains global weather data.

00:14:43.210 So let's use BigQuery to figure out which location on Earth had the hottest temperature on May 4th, 2000. The dataset has temperature data identified by the location, year, month, day, and mean temperature for that day. Here’s an example query: I’m finding the maximum mean temperature for that day by table. It’s common for interview questions to expect a SQL query similar to this.

00:15:19.360 I won’t run the query yet, but it normally executes in under 20 seconds, scanning over six or seven gigabytes of data. After the first run, it’s cached, so future queries are instantaneous. The results showed that a specific place recorded 102 degrees on May 4th, 2000; the number indicates it’s in Africa. The data shows this weather station is located in Mali, which provides us another clue.

00:16:36.750 Now we’re seeking to locate Gorby again, and we meet some shady characters who give us a final clue. The final postcard looks like a landmark again! Someone gets the joke — congratulations! For the rest of you, it will become clear shortly. Gorby has been enjoying his world tour and interacting with lots of people. But since Gorby is internet famous, he’s been greeted with head rubs and belly rubs; he really loves those!

00:17:15.390 We now need to determine where Gorby is. To do that, we'll run the same code as we did for the first postcard. It turns out he’s at Kauffman Stadium right here in Kansas City. Congratulations, we’ve found GorbyPuff! Also, I need to poke fun at myself because my actual proposals for RailsConf were rejected, leading me to speak in the sponsored track.

00:18:00.500 This talk was inspired by children’s television from the 80s, and I’m affiliated with a company called Alphabet. So let’s just say this talk is brought to you by the letters G, C, and P — which stands for Google Cloud Platform. As part of the sponsored pitch, I assure you we can likely run your applications successfully. I haven’t run into anything we can’t run yet; it may not make sense for us, but that’s a decision for you!

00:19:14.950 We have virtual machines, storage, excellent tools for big data, and machine learning support coming out soon. All these tools are compatible with Ruby. If you’re among the few who actually use containers in production, let’s talk since we’ve got fantastic resources for that as well. I have some special news to share. We’re officially announcing the App Engine for Ruby is now in beta! Today is May 4th, and there’s no better time for this announcement.

00:20:10.170 App Engine is our platform-as-a-service product. Some of you may have tried it a few years back, and you would remember that we only supported App Engine for JRuby then. Well, the current product allows you to use MRI, the same Ruby you run on your laptop. We'll handle auto-scaling and log management. It’s fantastic that it works and runs everything you care about.

00:20:51.750 We have many developers who’ve worked with Ruby for years contributing to the project, and I'm thrilled to announce this. You can run Sinatra, too, if you don’t want to work with Rails. For anybody looking to explore our APIs, just run 'gem install g-cloud.' Mike and Chris have done a terrific job making this gem a better experience than any of the previous offerings, and it truly feels like Ruby.

00:21:38.080 I want to express my gratitude to Aaron Patterson because I needed help running the idea for this talk. Aaron quickly gave me the green light to use his cat pictures. Note that he is not in this talk, but he’s familiar with the slides. Big thanks also go to Mike Moore and Chris Smith for working on the gcloud gem alongside my other amazing colleagues.

00:22:10.230 I’ve been at Google for about 18 months, and I’m excited to share something practical for you that isn't Docker. As I wrap up, I’d love to take your questions! Also, I want to share something I started earlier this year: if you ask a question, I’ll show you a cat picture. Why should you use Google Cloud Platform over Docker? My answer: Why not both? You can use both, depending on what you prefer.

00:23:06.600 If you don’t want to handle operations yourself, consider App Engine. If you know how to set up a VM, try Compute Engine. If you’re interested in containers, check out the Container Engine. We support all three levels, and we love to find solutions for everyone.

See Slides on thagomizer.com