00:00:23.390
So, this is "Stupid Ideas for Many Computers." I am tagging "miser.rb" on Twitter, but my phone is nowhere near the podium, so you can't interrupt me if you tweet at me. However, I love tweets, so you should definitely do that! My first RubyConf was in San Francisco in 2009, and Aaron Patterson and Ryan Davis gave a talk. This is a recreation of the actual title slide from that talk, by the way. If you haven't watched it, go find it on Confreaks; it is still one of my absolute favorite conference talks ever.
00:00:35.250
The thesis of their talk was that Aaron and Ryan had been riding the bus together a lot and came up with a lot of really bad ideas that they then implemented for some reason that none of us have figured out yet. They shared perfectly good code, including tests, responsible network usage, and other things for these awful ideas. One classic example was "Foo Beyond Fails," which was running on PHP, if I remember correctly. It was awful, but it worked, which is the terrifying part. At the time, I was in QA, and my test lab looked a fair amount like that. It wasn't actually wrapped up; it was sitting on a rolling cart.
00:01:06.180
Now I look at Google, and this is what I have access to for testing my stupid ideas. So clearly, it's time for this talk to be redone as "Worst Ideas Ever at Scale." A little bit about me: I'm Aja Hammerly. You can find me on GitHub as stag_miser. All the code for this talk is in my examples repo, which I pushed up this morning. My blog is at [email protected], although I sometimes get paid to blog, which happens slightly more than three times a year, but not much. I also work on Google Cloud Platform, which I will mention one more time during this talk.
00:01:27.090
If you have questions about running your apps in the cloud that is not the one you are currently using, most likely, come talk to me. I have credits, I have stickers, and lots of other things. Whenever I go to conferences, I bring a bunch of plastic dinosaurs, so if you want a dinosaur, you can come get one from me later. Because I work at Google, I have to say this: unless otherwise stated, all code in this talk is published at Google and licensed under the Apache 2.0 license. Now that I've made that disclaimer, it's time to get to the stupid.
00:02:02.180
No, really! Now it’s going to get into load testing. So, this is the unless otherwise stated part: the code in this section predates my employment at Google quite a bit, and it was actually licensed MIT. It is not in that repo. If you want it, please don’t ask for it; it’s really bad. This is really stupid load testing. I started my career working in QA on very understaffed teams at very small companies. I was asked to load test applications, and I didn't quite know how to approach it. So I went to Seattle Ruby, and I wrote that the real problem with load testing is that I needed to have a bunch of computers and make them do things that looked like actual users.
00:02:48.300
In this case, I had a Rails log of the scenario I wanted to test. It had 112 rules in my toolbox at that point, mostly mechanized. To be clear, I am a wizard at mechanization. I have done some awful things with Mechanize, many of which I won't admit; however, those will be included in this talk. I was told that my approach might be a bad idea, but feel free to ask me more about that later if we are drinking. I decided to turn my Rails log into a mechanized script for two reasons.
00:03:21.320
First, it allowed me to parameterize it a bit so I could inject different user IDs, thereby avoiding replaying the same scenario as the same user a thousand times in parallel. It turns out databases do not like that very much in ways that are not actually helpful during load testing. Secondly, by taking the giant Rails log and pulling it down to just the bits I cared about and making them into a mechanized script, I had a much smaller amount of code to push out to a distributed computing environment. I needed to generate this load for a while.
00:03:59.650
This is a fantastic load testing technique that I do not recommend and have used in production systems more than once. The techniques I’ll show you should not be considered appropriate load testing techniques. So, what does the code to do this look like? Let's be clear: it’s not good. This is some of the very first Ruby code I ever wrote that I have not eradicated yet. It needs to die in a fire; however, it's still kind of useful, which is why I haven’t deleted it. To drive that point home, the first line of the code is literally as follows.
00:04:43.900
Here’s the code: this tiny bit that I pulled out focuses on a section at a time. This is a basic preamble where I require Mechanize, create a Mechanized agent, and load a login page. It’s really straightforward stuff, but I'm not actually doing those things. I’m metaprogramming: writing code that will do those things. I open up a Rails log file and look for lines that match the regex that’s quite horrible. The regex looks for lines with starting parameters, extracting the action, the controller, and the parameters sent to that controller.
00:05:06.560
I then perform some logic based on that and produce an output. It just so happened that for this particular app that I was testing, ninety-seven percent of the Rails requests were directed towards one of two actions. Therefore, I just decided to figure out which action it was and ignored everything else. After logging in, anything else was irrelevant to my use case because if you're going to load test, you should focus on the ninety-seven percent of your app, rather than the remaining three percent.
00:05:38.020
This technique is flexible. I found some code from earlier this morning, where I had used this in the last year. I don’t know what I was thinking then; I know better now and have better tools available. However, I'm pulling out the action, controller, and parameters so I can manipulate any sorts of request types that I need to. We even pull out the HTTP verb if necessary. I’ve just shown you how to do a basic record-playback test at this point, but at this moment, I haven’t shown you how to generate load.
00:06:06.230
That’s the key word you would think we should be saying. To put it simply: you just do it a bunch of times simultaneously, hence my friend the infinite loop. There are companies you have heard of where I’ve utilized this code and technique to test their code, which is a little terrifying. In another significant note, I just used Bash scripting. We touched upon Unix a couple of times this morning.
00:06:44.930
I recall a shell script I used that launched my mechanized scripts, and I ran it 30 to 50 times based on how large the machine I was on at that moment was, and just let it go. The first time I executed this, I was running on commodity hardware purchased off eBay: big wall pizza box servers or aluminum cases. The setups were warm enough that I could have made a grilled cheese sandwich on their casings. I was confined to a conference room with a rolling cart piled high and had to close the door because it became too loud with all the servers running while I was conducting my tests. I never made that sandwich, and I do regret it now.
00:07:38.960
During later tests, we moved to using VMs in the cloud. There are several cloud providers available. We provisioned a VM once, and probably installed the necessary gems onto it, then clicked the button that said 'Make more like this one' until we had enough load to do testing. I had initially intended to call this section 'Deployment,' but deployment implies some level of organization and intention that I simply did not possess at that time; this was just pure hackery.
00:08:14.100
We needed to deploy this next week to production with no knowledge of whether it would hold up under load, so we took five handlers and started hammering screws into the wall with those five hammers. I used VMs to SSH into these machines to initiate my Bash script, which allowed me to ramp up the load slowly as it was a manual process. I felt like I was living in the future!
00:08:54.439
To be clear, this is stupid. Why is this a stupid idea? You don't know if your server fails if it falls over; if half your recognized agents fail to start up, you might not realize if they timed out or crashed midway through. Another reason this is stupid is that off-the-shelf tools exist for free; Apache Bench exists for this purpose, and there are many expensive tools that do this far better. You could utilize VMs and Ruby with all these available tools, but please don't.
00:09:26.850
So, one stupid idea leads to another. This next stupid idea is courtesy of my boss. My boss and I chatted over coffee one day, and he said, 'I have this awesome interview question: I’m going to ask candidates if they can perform sentiment analysis on a series of tweets using emojis.' I thought, 'That is the dumbest idea I have ever heard,' but I loved the question, and I asked him if I could use it in a talk. He agreed, so thank you to my boss for allowing me to utilize his interview question.
00:10:08.500
If you are ever interviewed by my boss and receive this question, I want to make it clear: this is not the right answer. What I present is the stupidest answer I could come up with. Just to clarify: sentiment analysis is the process of identifying and categorizing opinions expressed in a piece of text. It is challenging; true sentiment analysis is incredibly difficult because humans are not very precise with language. If someone says, "Sure, I'd love to," with a smile, you would think they're sincere. However, if your teenage son says it with an eye roll, they're probably not sincere, and the sentiment of that second statement could be vastly different from the first.
00:10:47.940
As we don’t just live in a world of words anymore, we also use emojis and emoticons, which are not necessarily any easier to detect sentiment for. They're new, so maybe we haven't developed as much subtlety in how they’re used. So, let’s consider how we might categorize these five emojis into positive or negative: a heart, a thumbs up, and a smiley face as positives, while a pile of poop and a grimace as negatives. I have to admit part of my motivation for this is to use the pile of poop emoji on my slides again.
00:11:30.420
However, if we consider, a thumbs up could arguably be a little less positive than a heart, or maybe the grimace is a bit more negative than the pile of poop. So, something like the following representation could be more accurate. This would involve creating a continuum from positive to negative with numeric values assigned to each emoji. If I'm going to do sentiment analysis, I also need a corpus of text.
00:12:08.100
However, I don’t have access to people's SMS messages where they share a load of context through emojis, primarily because I am not the U.S. government. Therefore, we can use Twitter to say... Here’s the code I needed. I utilized the Twitter gem to pull tweets, and if you’ve used the web service, you’ll find code to grab a thousand tweets at a time. I used keys and secrets to connect to the Twitter service and searched for a certain query. In this case, I searched for '#happy,' believing it likely to contain a good percentage of emojis.
00:12:43.490
Then, after retrieving a bunch of tweets, I’m just throwing them out to standard output and using Unix redirection to save them to a file for further analysis. I then analyze a tweet by going through a predefined sentiments collection, adding the value corresponding to any identified emoji or emoticon to the sentiment score of the tweet. The sentiments constant comes from a separate file because encoding is a pain, and I only wanted one file encoded in UTF-8. Hence, I have these strangely spaced white blocks in my editor that do not show the emojis.
00:13:13.120
So, this is a wonderful and nice approach, but this is a talk about doing stupid ideas at scale. So far, we’ve primarily focused on one stupid idea. Now, how do we scale this up to many computers? Ruby’s Rinda library allows for distributed coordination, and it establishes a shared tuple space, which is essentially a server holding a collection of tuples. If you’re familiar with Python, you’ve heard of tuples. I'll represent my tuples where the first element of each array is a symbol indicating the type of tuple, with the rest being the payload.
00:14:01.350
For all the examples I’ll share, the payload is typically a single entry: it may be a string or numeric, and in later examples, it could act as another object. We will have worker processes taking items out of this tuple space and placing items back in. If they retrieve something, they can process it, modify it, and then put a new value into that tuple space. This coordination allows me to have as many workers as desired, performing similar or dissimilar jobs in a very efficient manner. The beauty is how straightforward the code is to implement this in Ruby.
00:14:50.870
You will require the Rinda server to maintain the tuple space. This is all the code needed to set up a Rinda server on your machine. In Ruby 2.2, this works wonderfully. You can create a shared tuple space to connect many different agents. Then, it allows you to execute your tasks in a distributed fashion, and you only need to pass the URL of the server as a command-line argument. You wouldn’t even need to do that for the tweets.
00:15:38.800
I set up three types of workers: the fetcher, which records tweets into the tuple space. It feeds tuples looking like this: the first symbol is tweet, followed by the tweet's body. Here’s the code that accomplishes that—by the way, that line I’m highlighting is where the tweet is written into the tuple space. The first five lines are setting up a DRb connection and connecting to the tuple space.
00:16:19.830
In this example, I am not connecting to Twitter directly; instead, I’m reading tweets from that file created earlier, as Twitter is adept at rate-limiting. I lack the time before this talk to work on a means to increase my tweet rate without troubling Twitter. This is a straightforward method to write an extensive number of tweets into the tuple space.
00:16:56.940
The next stage in this processing pipeline is the analyzer. If you're familiar with MapReduce, this functions as the mapper. The analyzer will take tweets out of the tuple space and write the sentiment of each tweet back into the same space. Here’s the code: The first five lines mimic the earlier sections with DRb connections and are almost identical to the fetcher.
00:17:25.200
Then I initiate an infinite loop. Inside that loop, I employ the command to remove anything aligning with the pattern in tuple space: an array of two elements, with the symbol tweet as the first element and a string as the second element. I’m serving the string as the variable tweet, then I apply my sentiment analysis code as detailed earlier. Finally, the last line writes into the tuple space an array, where the initial symbol is sentiment, and the second value represents the numeric sentiment derived from that tweet.
00:18:03.870
This structure might look as follows: the sentiment may have a numeric value of five, and next, I require a reducer. In the MapReduce paradigm, reducers consolidate a large collection into a smaller one or a single element. In Ruby, we refer to this as ‘inject,’ which is an alternative for reduce. Using my reducer, I will collect all tuples starting with sentiment, ultimately attaining the overall sentiment across all tweets.
00:18:48.820
Here is the coding logic for that: I’m running a variable for the total sentiment while collecting everything that starts with sentiment from the tuple space, incrementing a total sentiment value, and then outputting it to standard out. That’s how simple it is. For those who prefer visual formats, here’s how it looks: I begin with a tweet, the fetcher pulls it in and deposits it into the tuple space, the analyzer takes it, and then the sentiment is placed back into the tuple space.
00:19:34.210
The reducer retrieves the sentiment, updating the cumulative sentiment value. I haven't yet discussed how to do this at scale. The answer is containers. If you’ve attended any conferences this year, I’m confident you've heard discussions regarding containers, especially Docker. I implemented this with eight containers because I was working with a relatively modest number of tweets—a thousand. Once you have a framework established, increasing that to one hundred or even a thousand containers isn’t particularly difficult.
00:20:30.500
That’s one of the perks of computer science: there are primarily three numbers involved: zero, one, and many. You can manage many different aspects of a scenario with relative similarity in terms of difficulty. I distributed those containers across five virtual machines, allowing for flexible scaling.
00:21:03.580
The following architecture shows a simplified overview: a single Rinda server and multiple analyzers built into the system. Using Google Kubernetes—as it’s a fantastic orchestration tool—I can tell it I have these VMs, and I require these N containers running. It figures out the best method for placement, without my needing to fuss over the details, which is fantastic!
00:21:38.640
I executed this setup and discovered that for the tweets I downloaded, the overall sentiment was 202, which is quite positive. This makes sense, as I was going through tweets tagged #happy. So, this presents a reasonable approach to conducting sentiment analysis on Twitter without being entirely ridiculous.
00:22:31.940
Now, shifting focus to combinatorics, I’ll discuss Latin squares. The earlier squares can be used for various mathematical subjects, and Latin squares filled with different symbols are a classic example. For those unfamiliar, a Latin square is an N x N array with exactly one of each symbol per row and column.
00:23:13.450
How many distinct Latin squares are there? Anyone want to venture a guess on how many exist for a four by four matrix? Or for a value below a thousand? Raise your hand if you believe it’s more than a thousand. In reality, there are 576. If you escalate this to a six by six grid, the number increases. Testing my code for six by six, there are substantially more.
00:23:58.980
While one can't even conceive how many Latin squares of size 9 exist, the estimation goes up to a whopping 6.67 billion! Just to provide perspective, there are more Latin squares than there are stars in our galaxy, and slightly fewer than the known universe's total stars.
00:24:48.890
Now, how do I generate these Latin squares? I’ll utilize a sieve approach. Those unfamiliar with the concept can relate it to the Sieve of Eratosthenes, where we produce various options up front and systematically eliminate those that don’t qualify. My programming stands because I reject the extreme computational time of these checks.
00:25:39.210
I created the algorithms for both Twitter and Latin squares in less than two hours in total. Given that I developed this code during a flight, I may have inadvertently made mistakes or oversights. But, these permutations bear vast consequences for evaluating possibilities amid complexity. Generating Latin squares yields an extensive possibility space to evaluate.
00:26:19.930
To check validity, I’ll extract options from the tuple space and identify rows' lengths. If each row matches a sequence—1 to N—and when I rotate (transpose) through 90 degrees, I verify if all columns equally conform to the sequence just as rows previously did.
00:27:10.620
Unfortunately, I can't feasibly conduct these tests on ordinary machines. I attempted this and it did not go well, necessitating scaling strategies. This time, windowing will manage my possibilities while placing solutions back into the tuple space.
00:27:57.610
This simple code still demonstrates how I can structure a comprehensive solution using the tuple space. Deploying is executed as before, following an architecture diagram where this time five VMs may prove considerably more useful if adequately investigated.
00:28:38.130
In conclusion, I want to express the absurdity of a lot of ideas discussed today: none were useful or built correctly; however, they all were rather fun. Numerous off-the-shelf tools and algorithms exist that are far superior to these half-baked attempts.
00:29:05.320
Before summing up, I’ll highlight that these 'stupid ideas' demonstrate that it’s entirely valid to pursue playful avenues, even within technical domains. Don't hesitate to explore concepts that may seem frivolous or incorrect; these explorations are learning opportunities.
00:29:48.320
Thank you for your attention. As a last note, I’ll be presenting this talk once more next month to a larger audience. Should anything I discussed seem unclear or if you have feedback, please share so that I can optimize or refine it appropriately. And with that, I’ll take questions and offer stickers and dinosaurs!
