00:00:12.630
I have known our next speaker for nearly ten years. She's the welcoming, happy face when you attend Ruby conferences. We've attended multiple Ruby and Rails conferences together, and as the token Brits, we tend to hang out, sit at the front, and support each other as we talk nonsense into the microphone. Plus, we bicker like an old married couple when we're at conferences. We first met in 2013 when I was invited to drinks with a guy named JB, who was opening the Pivotal Labs office in London. There, I met this incredibly intelligent young woman who had just finished Makers Academy, one of the London-based boot camps. I slid up to JB and said, 'She's amazing! I would love to hire her! I think she'd be really great to work with.' JB smiled and said, 'I already have.' And so I never had a chance to work with her. Post-Pivotal Labs, she has been pursuing her own ventures, which you will hear about soon. So Nadia is a proud Rubyist, a podcaster, a dancer, and an ambitious entrepreneur taking on Amazon pretty much single-handedly. Most importantly, she is my friend and now your keynote speaker, Nadia Odunayo.
00:01:56.360
It's such an honor to be keynoting RubyConf. Thank you, Gemma, Emily, and Andy, for having me here, and thank you for that lovely introduction, Andy. I'm Nadia Odunayo, the founder and CEO of a company called The StoryGraph. We are a reading tracker and book recommendations app. We help you choose your next book based on your mood, your favorite topics, and themes. We've got a whole range of stats where you can learn about your reading habits and see how they develop over time.
00:02:21.480
We offer machine-learning-powered personalized recommendations. It's like having a trusted friend who knows your preferences but whose brain is also a book encyclopedia. You can find your next perfect read by filtering lists of books by mood, pace, fiction versus non-fiction, genre, book size, and more. You can also read alongside friends, adding live reactions to specific parts of the book. The comments remain locked until your friends reach those specific parts in their reading. There's a whole lot more on offer: we have three million unique monthly visitors, 36 million page views a month, and we serve 110 million requests monthly. People often ask what StoryGraph is built in, and I love saying good old Ruby on Rails.
00:03:37.019
With all the website and app activity, we get a lot of customer feedback, feature requests, and bug reports. Given that I run the company's Instagram and Twitter accounts, I often receive messages that include something like, 'If you could pass this onto your dev team, that would be great.' While I have a fantastic co-founder in Rob Freelo, the web and app dev team is made up of just me. In fact, I recently gave a talk at the inaugural Rails SaaS conference run by Andrew Culver in LA last month. My talk was titled 'Getting to 1 Million Users as a One-Woman Dev Team.' I spoke about the ups and downs of building StoryGraph, including some truly stressful moments. There were times when I felt stuck, questioning whether I could keep going. As the talk blurb says, it was a personal and intimate story.
00:05:02.940
But I saved the real intimate details for the RubyConf Mini audience. I have a secret—don't tell anyone. You see, despite all the focused hard work and stress that went into building StoryGraph into what it is today, I couldn't help but maintain a little side gig. I'm not just the founder, CEO, and sole developer of The StoryGraph; I'm also a private investigator. And what is it that I investigate, you ask? Ruby crimes—not jewel theft. I investigate crimes and mysteries related to our favorite programming language.
00:05:51.600
Why did I go into private investigating? Well, there had been rumors circulating of a shady mastermind causing havoc and confusion among Ruby developers worldwide with his constant meddling in the source code. I wanted to be the one to put a stop to him. Oh, and I’m sure you all know how lucrative the world of Ruby private investigating is. I decided that I couldn't do this side gig under my normal name; I needed to protect my identity. So, I decided to call my Ruby private investigating alter ego Deirdre Bug. D for sure! Why? Well, isn’t it rather obvious? I have several stories from my years of doing this work, but for today’s keynote, I thought I should tell you all about a case that happened within the very walls of one of our community’s most revered and beloved institutions. This is the case of the vanished variable.
00:06:56.760
Chapter One. It was June of last year. I'd had an incredibly stressful couple of days at StoryGraph. Rob and I had just finished dealing with our first experience of serious online controversy around our product. Our content warnings feature came under fire after a series of well-known authors brought to light that inaccurate warnings were being attached to their books. Things were going wild on Twitter.
00:07:04.440
Rob and I stayed calm and managed the Twitter situation before coming up with a proposal for adjusting our system. We sought feedback from the community, and after our ideas received positive responses, I coded day and night to implement the changes. It was a wild and stressful time, but we pulled through. Being the sole software developer really wiped me out. Despite knowing it wasn’t the case, I felt personally attacked throughout this ordeal. I needed a break from StoryGraph and was in the process of planning a relaxed evening when the phone rang. Immediately upon answering, I heard a breathless, anxious voice say, 'I know I’m the last person you want to hear from, but I need your help!' Oh, I recognized that voice—how could I forget? It was Jenny.
00:08:42.900
Some of you may be wondering, 'Who's Jenny?' She was 29, a seasoned Ruby developer obsessed with all things Rails. She betrayed her best friend during one of my earlier cases by giving him faulty information about how Ruby method lookup worked. I'd heard through the grapevine that she had learned the error of her ways and was now on a good path, but I was hesitant to trust her. Jenny worked for the world-renowned Ruby Institute of Professionals (RIP), the organization whose whole reason for existing was to ensure that the Ruby programming language and community stayed alive and kicking forever. Slightly unfortunate logo aside, it was a job-related issue that had Jenny calling me that day.
00:09:47.820
You see, the RIP had started a new internal incubator—a competition of sorts—all within the organization's aim of improving Ruby's image and usage worldwide. People could form teams and submit various startup ideas. The winning team would receive a $500,000 investment and the opportunity to work on their startup full-time, fully supported by the RIP. For the inaugural round, Jenny's team had made it into the final two. They were building a platform that assigns grants from a pool of money that RIP had allocated for this purpose, aimed at funding Ruby developers and organizations. There were three different types of awards on offer, varying in grant size for different kinds of projects.
00:10:40.380
For their demo, they simulated an automated draw for the grants, including doing multiple redraws until the maximum number of grants had been assigned. However, the number of awarded grants came out incorrectly in the semi-final today, which was embarrassing. They thought they'd be disqualified, but the judges advanced them due to the tight competition. The judges would be looking for any reason to disqualify one of the teams. At this point, Jenny was clearly very stressed; she explained that their lead developer, Alex, was also very stressed and wouldn’t admit that he needed a second pair of eyes. They were the only two developers on the team, and not only did Jenny have other jobs to handle, but she really hated working with him. The final was tomorrow, and they had only a few hours left to resolve this issue. 'We need you,' she urged.
00:11:52.680
I was quiet while I thought about it. Not only was I wary of helping Jenny, given her dishonest past, but I also wanted to enjoy my much-needed break after the stressful days I’d had at StoryGraph. I was meant to be relaxing. But, well, Deirdre Bug can never resist a challenge, and it’s never a good idea to pass up an opportunity to work with the RIP. After all, I was sure they’d pay me handsomely for my work. I agreed to take on the case.
00:12:45.540
Chapter Two. 'I don’t need help,' Alex said bluntly after listening to Jenny’s explanation of why I was there. He sat in his office chair with his arms crossed, a mix between sulking and glaring. Jenny started to protest, but I had an idea. I had dealt with this type several times before—self-proclaimed 10x hero developers who believed they knew everything and declared that they could solve all of the company's problems in an afternoon if only people would leave them alone. I knew what would loosen him up a little. 'I’ll tell you what, Alex,' I said. 'I won’t distract you, I won’t say much, and I won’t touch the keyboard. I’ll just sit next to you and be your rubber duck. You can talk me through the code and the bug, bounce ideas off me. Of course, you’ll find the solution yourself and get all the credit, but this way, you’ll find the bug quicker, and time is of the essence.' We were all silent while Alex thought it over, frowning before finally relenting.
00:13:57.060
Alex had already turned back to his keyboard by the time Jenny gave me a subtle triumphant fist pump and said she would leave us to it. However, as I went to take my seat next to Alex, he brusquely put an arm out to stop me. 'I just realized that I can't show you the codebase as it is,' he said. 'There’s confidential code in here from some of RIP’s clients and partners. Let me get rid of it.' I was wary that a couple of hours had already passed since Jenny’s call, but I couldn’t really argue with the need to avoid seeing potentially sensitive client information. So, I instead pulled up a chair a couple of seats down from Alex and decided to make the best use of my time by opening up Duolingo. After all, there was no way I was going to catch the shady mastermind if I didn’t stay on top of my Japanese.
00:14:49.740
Forty minutes later, Alex was finally ready to show me the code. Now, I’m only showing you a simplified version of what I saw that day. Each award type had its own file that looked like this. Here, Alex had defined a class named AwardA, which inherited from another class called Grant. Then there was a class instance variable called award_count that was set to zero. There was a class method also called award_count that returned the value of the class instance variable. Then there was a method called assign, which incremented the award_count class instance variable by one. Alex quickly showed me that the files for the other two award types were exactly the same except for the class name change.
00:15:48.900
Then there was the Grant class itself. It had a constant called MAX_GRANTS, which was set to 100. The Grant class had its own award_count class instance variable that didn’t seem to be used anywhere. The Grant class also had its own award_count class method, which added up the award_count for all three award types. There was also an awards_remaining class method, used in the redraw to calculate how many awards still needed to be assigned. It took the maximum number of grants permitted and subtracted the number of awards that had already been given out. Then there were scripts that assigned some awards: 50 lots of award A, 30 lots of award B, and 20 lots of award C. And finally, a script that checked how many prizes were left and performed a redraw if necessary.
00:16:39.900
So, in the code Alex showed me, there were a maximum of 100 grants that could be awarded, and 50 plus 30 plus 20 equals 100, which equals the maximum number of grants available. 'Are you following so far, or do you need me to go over it again?' Alex asked. 'No, I got it,' I told him. If anything, I’d been holding back from suggesting places where his code could have benefitted from some improvements. I mean, that wasn’t normally my style; usually, I just focus on the problem at hand and avoid making quick judgments about the quality of unfamiliar codebases. But Alex’s behavior was frustrating. Overall, his code looked straightforward; I couldn’t see what could go wrong.
00:17:32.020
Alex ran the scripts. 'What? 300 grants awarded? Maximum number of grants exceeded?' What a mystery.
00:18:11.300
Chapter Three. I spent a few moments feeling baffled before I remembered that I was the great debugger, a private investigator. I had to stay calm and focused, and I had to play my cards with Alex carefully. He seemed to be opening up to me, so my strategy was to let him keep driving and slowly nudge him in what would hopefully turn out to be the right direction. Something had struck me during the code demo—the class instance variables. Now, I thought that class instance variables belong to one specific class only, but all four classes had one with the same name. Given that the three award classes all inherited from Grant and we had three times as many awards being assigned, I wondered if some overwriting was occurring.
00:19:06.300
'What if,' I said cautiously to Alex, 'each class were to have a differently named class instance variable? Perhaps they’re all conflicting with one another?' Alex glared at his display as if he hadn’t heard me, but then he shrugged and said, 'Sure, we can try that.' So he took each award class and adapted the name of their class instance variable to include a reference to its type. He then ran the program again—same result.
00:19:57.300
So it wasn’t the case that the class instance variables with the same name in one inheritance chain were interfering with one another; they did, in fact, stick to their own classes. I was wondering what to suggest next when we were interrupted by one of Alex’s colleagues. 'Alex!' he said. 'You’ll never guess who’s in the office!' We both turned to look at him. 'Max!' Everybody loved him, but I knew differently; I knew he was not to be trusted. In that moment, his appearance worked in my favor, because Alex jumped up and sprinted away from the computer toward wherever Max was. You’d think he’d heard someone drop a billion dollars on the floor, and the first person to pick it up would get to keep it. But I didn’t care, because finally I could get my hands on the keyboard—Alex’s fancy, clacky keyboard with no labels on the key caps didn’t faze me. Now I could finally do what I did best: solve this mystery.
00:22:17.340
Out of habit, and thanks to my excellent Git hygiene—if I do say so myself—I typed 'git status' into the terminal. There were changes in the repo, so I typed 'git diff' and casually looked over the output before it hit me. 'Oops! I probably shouldn’t be looking at this; there might be client data in here that RIP doesn’t want me to see.' I quickly exited but not before something caught my eye: 151 lines were added to a simple script that Alex had created to demonstrate the bug, whereas the file we had been looking at had only been a few lines long. I quickly tabbed over to the file in question, hit Command + Down Arrow to jump to the bottom of the file, and this is what I saw: the Grant class had been reopened. The class instance variable award_count had been overwritten to be 300, and the class method now read from the class instance variable directly.
00:23:22.860
No wonder that–'What are you looking at?' Alex had returned and wasn’t too happy to see me sitting in his chair touching his keyboard. He looked over my shoulder at the screen and saw what I was looking at. He looked a little alarmed at first before confusion took over. 'Where did you get my old code from?' he asked suspiciously. 'That’s from an old demo,' I said. All of a sudden, Alex started to look incredibly embarrassed, as if he wished the ground would swallow him up. He explained how, when he was getting the repo ready for me, he’d been making quick edits, copying and pasting a bunch of code from one place to another by selecting the whole file and then deleting large swathes of it. He must not have noticed this bit of code being left behind in his haste.
00:24:32.520
But I said, 'This code must have been hanging around for a while. Surely you would have noticed it unless you never do partial commits.' Now, copy-and-paste errors can happen to the best of us; I've certainly been caught out by them before. I have to admit, though, after his behavior throughout the day, it was rather nice to see Alex humbled a bit. But however trivial the bug, a case isn't over until I see the code working, so I deleted the code from the bottom of the script and ran the program again. While it didn’t turn out to be a meaty problem to sink my teeth into and distract me from my stressful StoryGraph situation, the case was closed, and my work here was done.
00:25:25.380
Chapter Four. I was on my way out of the RIP building when I heard someone call my name. It was Jenny. 'I just heard from Alex that the two of you figured out what the issue is,' she said. 'He’s gone off to implement the changes now.' I'm not the 'Told You So' type, but I said, 'It was the class variables, wasn’t it? I knew that using them was a bad idea.' 'You mean class instance variables,' I said. 'No,' Jenny insisted, 'class variables!' 'You don't mean class instance variables,' I asked because there wasn't a single class variable in sight.
00:26:06.720
I didn’t like the expression that began to form on Jenny's face. 'Do you have time to come back upstairs, please?' she asked. I wanted to get to the bottom of this confusion. We headed to her desk, where she started to fetch the latest code from GitHub. I told her to be careful so I wouldn’t see any confidential code. She said, 'What? There’s nothing confidential in here! We started a whole new code base for this project.' And now it was my turn to raise an eyebrow. It wasn’t long before the expression on my face turned into one of shock because Jenny was now showing me the code, and it looked exactly the same as the code Alex showed me—except for the fact that all of the class instance variables in the award classes were, in fact, class variables, and the grant class no longer had its class instance variable. Apart from that, it looked the same.
00:27:38.160
'Let’s run the code,' I suggested. 'Okay, so it still works,' I said to Jenny, pointing to something in the web browser tab that was visible behind the terminal. Alex had made a commit 14 minutes ago that was a one-line change: he’d removed a line from the grant class. So before his last commit, the grant class instead of looking like this, looked like this: the grant class also had a class variable. 'Let’s run this code,' I said, my suspicions growing.
00:28:34.560
And my suspicions were proven correct; this code failed in exactly the same way as the earlier bug that Alex had presented to me. Now I was really confused! So there was, in fact, a bug, but Alex knew how to fix it all along and didn’t tell his teammates, then gave me a fake problem to solve when I came in! 'Let’s go find Alex and get to the bottom of this,' Jenny suggested. It was past 8 PM and hardly anybody was left in the office. Alex wasn’t at his desk, he wasn’t in the kitchen, and he wasn’t playing ping pong. Max was gone so we knew he wasn’t somewhere falling over him. I spotted the colleague who came over earlier and asked if he’d seen Alex anywhere.
00:29:51.540
'I just saw him; he was trying to find a quiet corner to take a personal call.' We hurried off in the direction the colleague gestured, planning to grab Alex as soon as he came off the phone. We were about to turn a corner when we heard snatches of hushed dialogue: 'Zoe, they almost caught me! I even brought in a private investigator. Faked some code for fun and to buy time. Jenny will be nosy, so I fixed the real code for now, but don’t worry, honey! I’ll put the bug back in before the final tomorrow.' My eyes widened; Jenny gaped at me before we rounded the corner to confront Alex, who looked absolutely mortified.
00:31:01.560
Chapter Five. It’s two hours later and I’m sitting with Jenny at her desk, reflecting on everything I pieced together after interrogating Alex. Alex is dating another of your colleagues, Zoe, and Zoe is in the team competing with yours for the final $500,000 prize. At some point throughout this whole situation, Alex became fed up with your team and your idea; he much prefers Zoe’s team idea, which he had a hand in coming up with. He wants them to win so he can work on that team alongside his girlfriend, and he set out to sabotage his own team by introducing a bug to the code. But anyone who had a chance to look at the code would’ve immediately suspected the class variables, and so he sent me on a wild goose chase in a bid to waste time.
00:32:51.600
Wow, so much Ruby drama! Here I was trying to escape drama in my day job, and instead, I ended up in yet another mess during my side gig. I said goodbye to Jenny and headed home, knowing that despite the hour, my night wasn’t over. As soon as I got in, I went straight to my computer and opened up a Pry session—I had things to investigate. First, I created a class called Grant, which had a class variable named type that returned 'Grant.' I also defined a class method named type, which returns the value of the class variable. Then, I created an AwardA class that inherited from Grant and looked exactly the same as its parent class, except for the fact that the class variable type was set to 'A.' Based on what I’d seen earlier, if my suspicions were correct, then not only would AwardA.type return 'A', but now Grant.type would return 'A' too.
00:34:37.560
And I was indeed correct. Unlike class instance variables, which have their own separate copy regardless of the inheritance chain, class variables seem to share a value among objects in a chain. It doesn't matter which direction I go in. I had created a subclass and set a class variable with the same name there. What if I reopened Grant's superclass and set the class variable there too? I confirmed that Grant's superclass was Object and then reopened the Object class, giving it the same class variables and methods as Grant and AwardA. So, Object.type returned 'object,' and Grant.type returned the wrong type error: class variable type of Grant is overtaken by Object.
00:35:44.640
What about AwardA.type? Exactly the same error, including the reference to Grant! So it looks like, with class variables of the same name, there is definitely only one value shared in the inheritance chain, which is referred to from only one class in the chain. But you can only set them in one direction. If you reopen a superclass and define a class variable that's already been defined in any subclass, it raises a runtime error in those subclasses. I wondered how far up the inheritance chain Ruby went looking for class variable definitions. I confirmed that the highest class in Grant’s inheritance chain was BasicObject before reopening the class and giving it the same class variable and method: its type being Basic, of course.
00:37:14.920
Yep, looks like Ruby goes all the way up to BasicObject when dealing with class variables. At this point, I had a lot in my head, and I don’t know about you, but I always find diagrams helpful. So, it was time to sketch out what I pieced together and see if it matched up with the behavior in Alex's code. First, I knew that in the C Ruby internals, for every Ruby class there’s a corresponding R-class structure that stores things like a table of the class's methods, its attribute names, and also a table of class instance variables. I assumed that there was another table for class variables. So, if I took the Grant class from Alex's code, we’d have an R-class structure that looked like this, with an entry for award_count in the class variables table.
00:38:09.480
When the award classes were first defined, including setting a class variable with the same name, there were no problems initially. My inquiry had indicated that Ruby journeyed up the entire inheritance chain looking for the highest superclass that had the same class variable set and then overwrote the value in Alex's code. All of the classes had the same initial value for award_count of zero. But then the scripts started assigning awards, and this was where the behavior of class variables started to cause issues.
00:38:30.990
Every time an award of one type was assigned, the value of the award_count class variable was incremented by one. After 50 lots of AwardA had been given out, it was the one and only stored value of the class variable on the Grant class that was being updated. So, after the first script had run, when the scripts had called award_count on method A, it had correctly returned 50. Calling the same method on AwardB and AwardC would have also yielded the same result, despite the fact that we hadn’t awarded any grants of type B or C yet. Then we had the script that awarded 30 lots of AwardB, and the single copy of the class variable again got incremented 30 more times. Finally, when we awarded the last 20 lots of AwardC, we had this: so, when the final redraw script was run, award_count was called on the Grant class, which added up the results of the same method on the three award types—three lots of 100 equaling 300—which matched the bug in Alex's code.
00:39:49.200
I felt rather pleased with myself for piecing this together, but I wanted someone even a little bit wiser than I was to verify that I had got it exactly right. Luckily, I knew just the person.
00:41:18.520
Chapter Six. Despite the hour, I wasn’t surprised that my contact replied right away, saying she’d be up for jumping on a video call with me to talk about class variables. Who was I speaking with? Ellen. Now, who on Earth is Ellen, some of you may be wondering? Ellen was 45, an experienced freelance developer who had seen many things and regularly shared what she learned on her adventures with the Ruby code base. I talked her through my diagrams, including the bug I had been dealing with and the general picture I pieced together about how class variables worked. 'Did I get it right?' I asked.
00:42:19.020
'Yes, except for one small thing,' Ellen said. 'Class instance variables and class variables are actually stored in the same table in the R-class structure. The extra @ symbol on class variables means that they don’t get mixed up.' 'Okay, cool,' I said. 'But there’s been something on my mind, Ellen. Class variables seem to only cause a lot of trouble. What are they good for?' 'Well,' said Ellen, 'they're really good for application configuration. In fact, your favorite framework, Rails, uses them extensively for that very thing.'
00:43:42.720
'Okay, fine, but that's a whole framework that the everyday developer doesn't have to care about then, right?' 'Well,' said Ellen, 'let me tell you a story. You discovered today that when accessing a class variable, Ruby will go up the whole inheritance chain, all the way to BasicObject, to check which class was the highest to set the variable. Eileen Uchitel of the Rails core team and Aaron Patterson, who is on both the Rails and Ruby core teams, wanted to dig into this fact some more. The bug you dealt with at the RIP today only featured an inheritance chain that was two levels deep. But Eileen and Aaron, with their expert knowledge of Rails internals, knew that Rails made extensive use of class variables, and the inheritance chain there went a lot deeper.
00:44:44.640
Surely, Rails’ extensive use of class variables was incredibly inefficient and might be having a performance impact on Rails apps everywhere! Much more efficient would be, instead of going up the entire inheritance chain each time to check for the highest superclass within which a class variable was set, there should be a cache. So, that’s what they built. With the cache, instead of Ruby checking every single class in the inheritance chain when trying to find the value of a class variable, the method points to a cache that tells your code exactly where to look for the class variable setup. After testing, they indeed observed a performance improvement with this cache. Eileen and Aaron submitted a pull request for Ruby that just got merged the other day. Because these two took the time to understand how things worked under the hood, they developed this change. With Ruby 3.1, Rails apps can see a 6 to 7 percent request throughput performance increase.
00:45:44.640
Now, why did I tell you this story? Ellen asks me. Well, people are always saying things like, 'Ruby is slow! Ruby doesn’t scale! Ruby is dying!' And I've had many conversations with Eileen about this; I know how she feels about it, and I feel the same way. You see, if only more of us took the time to learn Ruby—really learn Ruby—and understand how it works under the hood, and understand why it is the way it is, and get to grips with all of its quirks and gotchas, we’ll be able to spot pain points and opportunities to improve Ruby, make it faster, make it great for scaling apps, and so much more. If we see things we don't like about Ruby or think something could be better, instead of complaining, we should be more like Eileen and Aaron—investigating, experimenting, benchmarking, building, and fostering conversations in the community. We should be doing all of the things that’ll help keep Ruby alive and make it a no-brainer tool of choice for developers and organizations at all stages for decades to come.
00:46:44.760
As with any conversation with Ellen, I left it feeling super inspired. There and then, I vowed that whenever I got time away from StoryGraph—even though it wasn’t often—I would strive to understand how Ruby behaves under the hood and do what I could to contribute to the amazing language and community. Thank you.
