Ruby Video
Talks
A Magical Gathering
Aaron Patterson
@tenderlove
#ruby
#opencv
#image-processing
#computer-vision
#machine-learning

A Magical Gathering

by Aaron Patterson

In the video titled A Magical Gathering, Aaron Patterson, a member of the Ruby and Rails core teams, presents a whimsical yet informative talk at the MountainWest RubyConf 2014. The main theme of the presentation revolves around using programming and technology to create a system for identifying and managing a collection of Magic: The Gathering cards.

Key Points Discussed:
- Introduction and Context: Aaron shares his background, his connection to Utah, and his personal anecdotes about local cuisine and oddities, including the state snack of green jello and fry sauce. This light-hearted beginning sets a relaxed tone for the talk.
- Project Motivation: The inspiration for the project came from his vast collection of Magic cards, which he wanted to organize and identify digitally. Aaron humorously recounts his past experiences with the game and his decision to re-engage with it.
- Technical Overview:
- The system includes a webcam, a laptop, and a light box for capturing images of the cards. The process involves taking photos, extracting card images, identifying them, and saving the data.
- He describes using OpenCV for image recognition, explaining how he employs techniques like perceptual hashing and Hamming distance to compare and identify cards.
- Aaron implemented a promises system in Ruby to handle data downloads, improving efficiency with a thread pool.
- Data Handling and Challenges: Aaron discusses challenges with data organization and model adjustments, emphasizing the importance of structuring the data effectively in SQLite. He highlights issues like similar card artworks causing identification difficulties and needing to teach the system manually when it fails to identify properly.
- Conclusion: The system, while not perfect, allows him to effectively manage his card collection and predict card ratings. He stresses the importance of using technology creatively and encourages others to explore unique projects that leverage programming skills.

Overall, Aaron’s talk combines humor and technical depth, illustrating how programming can enhance personal hobbies in engaging and innovative ways.

00:00:24.860 Good morning! Whoo! Yes, good morning. I'm very excited to be here.
00:00:31.680 A couple weeks ago, I went to Ruby on Ales and caught a cold. I think I got it from Mike, and I feel like this conference is 'Ruby for what ails you,' which is a cold.
00:00:41.370 My name is Aaron Patterson, and you can find me on the Internet as 'tender love.' I am on the Ruby core team and the Rails core team. This doesn't mean that I know what I’m talking about; it just means that I'm really bad at saying no.
00:01:07.860 I’m actually from Utah, I grew up here in Salt Lake City. People always ask me the same question when they find out that I'm from Utah, and I want to preemptively answer it: the answer is no, I don’t know how to ski.
00:01:19.050 I grew up in Rose Park but moved to Seattle about 14 years ago, so don’t ask me for any food recommendations. I want to see how many of you live here in Utah now and how many of you are from out of town.
00:01:47.280 Okay, so about half and half. Half of you will already know some of the things I’ll talk about regarding Utah.
00:02:04.049 This is the Utah State cooking pot: a Dutch oven. I don't know why we did this, but I remember there was a hubbub back in 1997 when it was passed as a law. People were like, 'Why are we passing a law about cooking pots?'
00:02:14.579 Anyway, I love my Dutch oven! You should check out Stonecutter; they have scones that are different from anywhere else. I’ve never been to Stonecutter, but you should also go to Crown Burger because they have really good pastrami sandwiches.
00:02:27.090 Now, did you know that the state snack is green jello? This is totally true! So if you go somewhere in Utah and order food, try to get green jello. For authenticity points, it will probably have carrots in it.
00:02:36.180 Also, you should try fry sauce. It’s a Utah thing. You dip fries in it, and I guarantee that anywhere you go to eat where they have fries, you can ask for fry sauce, and they will have it.
00:02:42.840 The last thing I want to mention for those of you who are new here is that Utah has some weird stuff. We have a pyramid; look it up online, it’s quite interesting. Also, there’s a Sphinx of Joseph Smith. You should check it out at Gilgal Park, which is close by and within walking distance.
00:03:05.220 Gilgal Park has a lot of stonework and was once closed to the public as it was privately owned. Back when I was in high school, I would sneak into this park to take pictures. I’ve been kicked out of this park many times, but eventually, the city bought it and turned it into a public park.
00:03:20.340 So yes, you should check this park out. Today, I brought two very special guests with me—my parents. Nobody embarrass me, they are down here!
00:03:48.390 I invited them because I want them to see what I do. I’m not sure they believe that I actually earn a paycheck. But, to clarify, I’m not getting money from loan sharks or anything like that!
00:04:01.320 I thought it would be nice for them to see what I actually do for a living. Unfortunately, this talk has absolutely nothing to do with my job, but hopefully, they’ll learn something.
00:04:24.000 They taught me a very important lesson: if you learn things, you can do things. If you look at my grades from school, you’ll see that I didn’t learn this lesson for quite a while.
00:04:40.320 Unfortunately, the usefulness of this talk is about zero. We're going to discuss some related stuff, but let's get into it. This talk is called "Oh Ho Ho! It's Magic!" I could talk a lot about Magic: The Gathering.
00:04:53.360 Yes, I play Magic: The Gathering. Anyone else here play? A few people. Recently, I started playing again. For those who don’t know, Magic: The Gathering is a collectible card game.
00:05:09.240 You collect cards and play against people. I played this game about ten years ago and bought a whole bunch of cards. I was having a good time until I went to a tournament and got destroyed by a 13-year-old.
00:05:27.240 He insulted my parents and I quit. It felt like playing Call of Duty, except in real life. Recently, a friend of mine asked if I wanted to play Magic again, and I thought, 'Sure, I have thousands of cards in my closet!'
00:05:49.500 So I pulled the cards out, but I realized I had no idea what I was doing. I literally have thousands of cards—like five or six thousand—and didn’t know if any were good or how much they were worth.
00:06:12.540 It seemed like a job for computers. This task is repetitive and something computers should be able to handle. So, I put on my robe and wizard hat and put together a system to identify these Magic cards.
00:06:31.650 Today, I will talk about how I identify these cards using a computer. The primary idea is to lessen the work for myself, although you’ll see it actually entails more work. But it’s all about the journey!
00:07:25.380 So, let’s discuss my hardware. I have a laptop and a webcam, along with a light box that’s a bit washed out. The webcam points inside this box, and here's a demo of the final system.
00:07:51.500 The upper right is a live stream from my webcam, and in the upper left, you can see the program identifying the cards. It shows me what it thinks is the right card, and I verify if it’s correct. If so, it saves that card.
00:08:20.370 The high-level process includes taking a photo of the card, extracting the card from that photo, identifying it, and then saving the data. We repeat this loop continuously.
00:08:53.200 To identify cards, I start with a corpus of known images and their corresponding data. The program makes a guess based on this corpus. If it guesses correctly, I confirm, and if it’s wrong, I teach it the correct answer. Every time I teach it, it saves the data to improve future identification.
00:09:52.059 For the actual code involved, I utilize library functions for card matching, OpenCV for recognition and cropping, and SQLite for data storage. I’m going to go over how I gather the information.
00:10:13.360 I got it all from the Wizards of the Coast website. I need to gather information such as the card name, image, set, and rating, because a particular card can be reissued multiple times, affecting its price.
00:11:01.720 Thinking about this, I modeled the data. An image has one card, and a card has one image; this seemed straightforward. But I soon found that one image could correspond to two cards. Everyone who's written a web application knows how frustrating it is when your data changes!
00:11:42.430 I fixed that issue by adjusting my model, ending up with the realization that an image can have many cards. Next, I want to explain how I downloaded this data using promises in Ruby.
00:12:09.890 A promise in Ruby allows you to execute a chunk of code later and access the resulting value at that time. The API enables me to ask for the status of a promise and cancel it if needed.
00:12:55.600 That’s the entire implementation of the promise system in Ruby. The downside is you can create an unlimited number of promises, which can be inefficient, especially if your machine has limited CPUs.
00:13:57.560 Thus, I implemented an executor pool—basically a thread pool that lets me run a limited number of promises in parallel. This allows me to queue up a bunch of work, limiting the number of simultaneous connections.
00:14:35.800 I downloaded 1.6 gigabytes of data in about 40 minutes, which was awesome! However, I later discovered a website called MTG Jace Hong that has all the data I downloaded in JSON format. The moral of the story: Google first, as it can save a lot of time.
00:15:42.210 Next, I use perceptual hashing to identify images. For example, I have two images: one is my reference and the other is my scanned image. Each image has a key that we can compare using Hamming distance, which tells us how similar the images are. A lower Hamming distance means the images are more similar.
00:16:47.550 I needed to sort this data because Wizards has produced about 25,000 unique cards. To do this efficiently, I extended SQLite to add a Hamming distance function, allowing me to sort by distance in the database.
00:17:08.480 Then, I had to calculate the hashes for all the images stored in the database. This process took about an hour, and I had to get smarter about utilizing CPU resources effectively.
00:17:28.890 Now, let’s move on to recognition and cropping. What I am trying to solve here is how to crop images of Magic cards correctly so they are nicely formatted. I capture a photo with my webcam, and it needs to be processed accordingly.
00:17:54.640 The first step in pre-processing is to change the photo to grayscale, which allows us to detect edges better using the Canny edge detector. Once the edges are detected, I can find contours from the image.
00:18:21.720 After that, we loop through the contours to find the largest one. The largest contour should correspond to the card itself. Once we identify this contour, we convert it into a polygon for cropping.
00:18:41.900 I utilize OpenCV to obtain a polygon from the contour, which can then be warped into a rectangular shape. This enables accurate identification of the card.
00:19:03.120 After adjusting the image, we can use the Hamming distance to identify the card in the database, leading us to save this information. Once saved, the system becomes smarter and can easily identify cards based on previous inputs.
00:19:27.760 Now that I have implemented this system, I can finally answer questions like what cards I have and their respective ratings. Even though I haven’t integrated pricing systems yet, I can see how well each card performs.
00:19:49.660 Some challenges remain, such as the webcam's focus time. It takes forever to focus, which is not ideal when trying to identify thousands of cards. The other issue is total failure when the system cannot identify a card.
00:20:10.180 When the system fails, I need to teach it manually which card it is, but the good news is that once I've taught it, it remembers the information for the future. There are also problems with similar artwork where different cards look nearly identical.
00:20:54.010 In summary, this has been a glimpse into the system I built for identifying and managing my Magic card collection. I hope you’ve gained insights into using these technological methods in unique and interesting projects. Thank you very much!
00:28:31.990 Cheers!
00:28:45.180 You.
Edit