00:00:00.030
In between managing a team at Up and Ferocious, running a bank, and parenting two children, Michael finds time to play Mario Kart. This is relevant to our next talk where some people will come up on stage soon to play Mario Kart. It's cool that we’re going to watch them play games at a conference, and Michael's just going to stand there.
00:00:27.000
This taps into your enthusiasm for the game, would you say? Yes, indeed. This story was born out of banking. I work for a serious company that builds serious banking software in a serious industry. In case you hadn't gathered, it’s all very serious. So naturally, what do we do when we're looking to relax? We play Mario Kart, and we take that pretty seriously. You can tell it's serious when you see pictures like this where everyone is sitting about three or four feet from a ginormous television.
00:01:02.489
You really know you’re taking Mario Kart seriously when you start looking at the world a little differently. The software developers started with a simple question: who is the best at Mario Kart? Naturally, as software developers, we think, 'Well, we can solve this with software.' So we did. We built a little leaderboard application, and it looked something like this. I could have taken any race results, but let's pick one that's a bit flattering. It's a Slack integration with a standard Rails app.
00:01:47.460
The application posts back to Slack with the current leaderboard, showing who moved up in the rankings. You'll notice that there’s a score column there, and usually, this would elicit some sort of response. We needed a mechanism to track this leaderboard, so we started with an Elo rating algorithm, which is typically used for tracking chess games. This algorithm can track wins and losses between two players.
00:02:44.460
The problem with Mario Kart, however, is that we typically have four players in a game. To solve this, we would enter the results as a series of two-player games based on the results of one player beating another. The results indicated one person emerged as the 'king' of sorts, but clearly, that algorithm wasn't cutting it. So, we stumbled upon an algorithm called TrueSkill, developed at Microsoft Research as part of their work on Halo 2.
00:03:09.930
TrueSkill is designed to track arbitrary numbers of players and teams. One of the developers I work with, Tom Meyer, went out of his way to implement this, and he agreed that this new algorithm was indeed better. However, we wanted to go beyond simply knowing who was at the top of the leaderboard. We added functionality to inspect individual players to see how many times one player had beaten another.
00:04:32.639
We also wanted to plot a player's decline over time, so we developed the ability to generate a chart with the Google Chart Image Drawing API. The Rails app generates a chart and responds with a Google image URL, which gets displayed inline in Slack. Users can query this chart for any of these players over a specific date range. You might think this is a bit ridiculous, and it probably is, but it highlights the lengths we went to.
00:05:53.520
So let's demo it today. Jesse, please come up. We actually need a few people who can play Mario Kart. If anyone here has a Switch, come on up. Yes, you, come on up. We have a Switch set up here with four players.
00:06:16.260
So here's our system. You can see all the players as we let them play. It's exciting to see that we're detecting a match in progress. Here we go.
00:08:14.970
The system will track timings, indicating how long each player takes. This adds a Fitbit-like functionality, matching players against each other and providing statistical results, like the crown and race finish times. Mario Kart has certainly evolved into this functioning result-analysis system.
00:08:35.160
Now, how does this system work? It's broken down into three steps. First, we need to get an image off the Nintendo Switch. Second, we need to analyze the contents of that image. Third, we need to reconstruct the current state of a game from all the gathered data.
00:09:40.450
Getting an image off the Nintendo Switch involves using an HDMI splitter, an HDMI-to-Ethernet sender, and a receiving device. Normally, these devices would simply transmit video to a monitor; however, we sought to capture the network traffic instead. Using a TCP dump, we can grab the stream and write it to a series of files until disk space runs out.
00:10:48.350
Moving onto step two, extracting information from the images meant writing Ruby code to identify screens, such as the main menu. By focusing on unique identifiers and employing the pHash library to detect image similarity, I could accurately determine which screen we were encountering. This helped us extract essential game data from the images.
00:12:40.270
For the race screen, we identified player position and item information through pixel analysis and comparison against reference samples of the UI for accuracy. This method allowed us to track which player is in what position and what items they currently possess as part of their gameplay.
00:15:00.720
Additionally, I had to appropriately collect the race result screens to determine the points for each player and recognize who would wear the crown at the end of the match. This involved careful differentiation between frames to ensure that we collected the right information.
00:17:08.190
The information we gathered allowed us to efficiently reconstitute the game’s state. By establishing a data model that included matches, races, and race snapshots, we could build a coherent event stream and respond dynamically to game events.
00:19:01.960
Throughout this journey, I learned a great deal about the versatile applications of Ruby. The development process has motivated me to encourage others to look for innovative uses of Ruby and implement it on interesting projects. With the help of our outstanding open-source community, we made this entire project a reality.
00:21:05.750
In closing, I would like to express my gratitude to my workplace for supporting my attendance here. Working on building a digital bank aimed at helping people manage their money has been fulfilling. If you have ever wished for a bank that helps optimize the use of your finances, I encourage you to look into our services.
00:24:00.340
Lastly, I appreciate the opportunity to speak today and the great work put into organizing this conference. Thank you to everyone, and I'd be happy to answer any questions.
00:28:54.790
I'm open to any questions you may have about the demo or the development process. I had to pull this all together quite quickly, but it took a lot of collaboration with my colleagues to refine the code and improve the project’s scope. If you’re interested in building something similar, the code is straightforward, and I hope to see some creative implementations in the future.
