Ruby Video | How Much Pressure Can Your App Handle

RubyConf AU 2020

How Much Pressure Can Your App Handle - Performance Testing With Ruby

by Nancy Cai

In the talk titled 'How Much Pressure Can Your App Handle - Performance Testing With Ruby', Nancy Cai, a software engineer from Airtasker, addresses the critical need for performance testing in applications to handle varying spikes in traffic. Cai emphasizes the importance of understanding the application's limitations before significant marketing events or business expansions. The presentation elaborates on the process of performance testing, which involves several key phases: determining performance criteria, configuring a test environment, planning and designing tests, implementing a testing framework, executing and monitoring tests, and analyzing results.

Key Points Discussed:
- Need for Performance Testing: Cai highlights real-world scenarios showcasing the importance of being proactive in performance testing to prevent system downtimes that can damage the company's reputation.
- Types of Testing: Various types of performance tests are explained, including load testing, stress testing, and spike testing, each serving different purposes in measuring system resilience under pressure.
- Formulating Right Questions: Instead of generic questions, more specific and context-driven inquiries are necessary to really gauge an application’s performance and readiness for increased traffic.
- Establishing a Test Environment: It is vital to create an environment that closely mimics the production environment to ensure the tests yield relevant results.
- Selection of Tools: Nancy lists criteria for selecting performance testing tools, including the ability to simulate production-like traffic and the ease of using these tools for team members.
- Implementation and Monitoring: The process of creating and executing test cases using tools like JMeter is discussed, along with methods to visualize results during tests.
- Infrastructure Scaling: Nancy introduces a three-dimensional scaling model for better resource management and optimization during peak loads.

Examples and Illustrations:
- Nancy shares a case study from Airtasker, where a marketing campaign led to an unexpected surge of users that strained their system, highlighting the real impact of performance issues.
- Tools such as Ruby JMeter and Terraform are discussed as effective solutions used in building and managing the testing framework.

Conclusions and Takeaways:
- Performance testing is an essential component for any application facing increased user traffic.
- It is crucial to conduct thorough testing while being mindful of real-world conditions.
- Companies should continually monitor performance post-tests to ensure that applications can withstand future demands.
Cai encourages further exploration of performance test strategies through resources like the Ruby JMeter GitHub repository.

00:00:00.000 I studied Economics, Interpreting, and Translation in university. However, after graduation, I worked as a dental nurse for two and a half years before realizing that it wasn't the career I wanted.

00:00:05.730 I decided to change my career path, so I taught myself the necessary coding skills to become a QA engineer. Later, with the mentorship of other software engineers at Airtasker, I gained the skills to join their ranks.

00:00:19.859 In addition to Ruby, I have a deep love for the Lord of the Rings; it’s my favorite movie! I also enjoy making fondant cakes, which my coworkers say are amazing.

00:00:45.440 Hello, everyone! I’m Nancy, a software engineer from Airtasker. Today, I'm going to walk you through our journey of performance testing using Ruby.

00:00:51.290 First, I want to start with a question. Imagine one day your CEO is on a call discussing your application doing some public relations for your company. How many of you would be confident to say that you know whether our platform can handle a huge spike in traffic? If it can't, do you know where the breaking point would be? Raise your hand if you know.

00:01:13.460 No one? Let's move to a more realistic question. When your business thinks we’ve done well enough in Australia, how many of you really know whether your current infrastructure can support expansion into three more countries over the next 12 months? Still no? That’s exactly the point.

00:01:35.780 For most of those who don’t know the answer and are looking for some direction, I’m here today to illuminate the dark! I’ll walk you through the journey of our performance testing, from design to implementation, and finally share our learnings. Let’s start with the ‘whys’—why do we want to conduct performance testing? We aim to find the invisible boundary lines: there is a point we know when we cross it leads to system downtime, but we often don’t know exactly where that point is.

00:02:15.319 Engineers may understand the limitations of their specific areas, but they might not have the complete picture. We want to grasp the limitations of our current infrastructure to facilitate expansion, and we also need to understand how our platform behaves under extreme circumstances. It’s crucial to note that during a high sales period, performance degradation may not be gradual—it often leads to unforeseen behaviors. We want to be proactive rather than reactive.

00:03:08.549 Let me share an example. The company I work for, Airtasker, is a marketplace where people can complete various everyday tasks. Two years ago, we launched a marketing campaign, offering $500 for people to play with puppies. Naturally, this led to an influx of users, and we had 1,700 people flood onto our platform for that one task. When we started celebrating the campaign, things began to break down.

00:03:46.799 Alerts started to scream, and we were at an off-site company gathering. People were saying, 'Put down your beers; let’s put out this fire!' This situation happens more frequently, and it can damage our brand reputation. Therefore, we aim to prevent such incidents by being proactive.

00:04:18.989 So, what's the process for performance testing? The first step is to determine performance criteria. Here, we gather enough information to ask the right questions and identify what types of performance testing we need to carry out.

00:04:40.050 The second step is to configure a test environment. We must create a test environment that simulates our production environment, ensuring we have all the necessary tools in place. The third step is to plan and design, where we analyze how real users behave and design our test scenarios and cases accordingly.

00:05:26.320 In the implementation phase, we build the testing framework, set up monitoring tools, create test cases, and prepare test data. The fifth step involves executing and monitoring the tests, after which we analyze the results, fine-tune our application, and rerun tests for comparison.

00:06:04.069 Let’s discuss the first phase: determining criteria. We need to understand the types of performance testing available so we can select the appropriate ones based on our criteria.

00:06:17.360 Load testing, for instance, tests the system’s behavior under anticipated load. Stress testing helps to find the breaking point by gradually increasing the number of users until the system fails. Spike testing analyzes system behavior during sudden increases in traffic.

00:06:44.810 Volume testing boosts your dataset to observe its impact on response time, similar to endurance testing, which examines memory consumption over a longer time. For a clearer understanding, we can compare testing time and the number of virtual users.

00:07:10.469 As you can see, load testing considers a specified number of users, while endurance testing involves a longer duration to surface issues like memory leaks. Stress testing is when we keep increasing the user count until we reach our limits.

00:07:47.660 Next, we’ll discuss how to formulate the right questions for establishing performance test criteria. Often, people ask generic questions like: 'What’s our average response time?' or 'How much traffic can we handle?' Unfortunately, these questions typically lack business value.

00:08:26.350 Instead, we should ask detailed questions tailored to our context. For instance, given regular traffic patterns, at what point does our performance become unacceptable? How much headroom does our infrastructure offer? If we double or triple our traffic tomorrow, can our application manage it?

00:09:02.190 To answer such questions, we would conduct stress tests while monitoring key performance indicators, such as ensuring the error rate does not exceed 0.5% at a given level of traffic.

00:09:46.640 On a personal note, I’ve been stress-testing my public speaking skills—from presenting five minutes in front of 15 people, to 10-minute talks for 40 engineers, to speaking today before an audience of 400 to 500 people for 35 minutes. Will I pass out from nervousness? Let’s find out in 30 minutes!

00:10:11.860 Returning to performance testing: if a major marketing campaign is on the horizon, how quickly can we scale to meet the anticipated demand? This is a perfect example of a situation requiring spike testing.

00:10:38.580 Drawing from historical data allows us to forecast potential traffic increases. For example, if we expect 300% more users during a marketing event, we can set criteria like ensuring that node and deployment replicas can scale to meet this surge within a few minutes.

00:11:02.360 Now, let's talk about the test environment. We want it to be as close as possible to production so that test results are reflective of reality. It should be isolated as we intentionally put it under pressure.

00:11:35.290 To achieve this, we run our tests using a duplicate image of our production instance on a different Kubernetes cluster. We replicate configurations and utilize tools like Terraform for resource management.

00:12:11.940 In terms of data, using a smaller test dataset can sometimes hide slow queries, so we create a snapshot of our production database and anonymize it for testing.

00:12:44.400 Keeping our testing environment cost-effective is also crucial. We have scripts that enable us to spin up and shut down our testing environment efficiently.

00:13:05.340 Regarding our selection criteria for performance testing tools, we want them to properly simulate production-like traffic. We prefer industry-standard tools that are well-maintained and possess strong community support.

00:13:36.950 We aim for tools that are easy to create and maintain tests with minimal learning curve. It's vital that all team members can operate these tools, and keeping a history of test runs for comparison is essential.

00:14:08.260 Visualizing results greatly aids in analysis. We also intend to run our performance tests regularly, so integration with CI/CD tools is critical.

00:14:38.560 Various performance testing tools are available in the market today, with some being open-source and others commercial. For smaller tests, tools like JMeter may suffice, but they come with limitations, such as the maximum number of virtual users.

00:15:11.690 To run larger-scale tests, we often select commercial tools built on top of open-source ones, which can support a greater number of virtual users while offering detailed reporting dashboards.

00:15:51.870 In our case, we chose JMeter due to its large user base and excellent dashboard capabilities. We also selected Flatirons, which uses Ruby JMeter—an open-source DSL for building JMeter test plans, enabling us to structure our tests in Ruby rather than XML.

00:16:21.370 Moving on to the planning and design stage, we have several best practices to follow. Firstly, it’s important to ramp up users gradually rather than launching everyone at once, as this can create artificial bottlenecks in the system.

00:16:52.460 Similarly, we avoid starting from a zero-load situation, as this doesn’t reflect real-world conditions. We want to consider different clients and browsers, as traffic will come from various sources.

00:17:24.280 Additionally, we should distribute user load to simulate a more realistic traffic profile. Not every user will interact with our application the same way at any given time.

00:17:57.480 We can analyze traffic profiles by measuring HTTP requests and focusing on endpoints that are vital to our application.

00:18:23.660 Next, I'll illustrate how this implementation allows our various teams to leverage performance testing. I’ll show you a script that spins up the test environment.

00:18:51.020 First, if the command is 'up', we navigate to the Terraform repository and provision the resources according to our Terraform plans.

00:19:18.890 We create our deployment by applying all Kubernetes configurations and, once done, we destruct our deployment to clean up all resources.

00:19:45.080 Now, I want to provide an example of what test data looks like using Swagger. For test cases, we define attributes such as name, category, status, and selfies.

00:20:13.600 To keep our reporting organized, we create helper methods used across our test cases. For instance, we set default hosts and request headers, including content types and authentication tokens.

00:20:42.660 Next, I’ll showcase how we create test cases using Ruby JMeter. To begin, we specify the number of virtual users, ramp-up time, and duration of the test.

00:21:12.460 We also define the HTTP method and endpoint for our requests. If necessary, we may extract data from responses for further use.

00:21:42.130 In this setup, we can control how traffic is divided. For example, we can allocate 50% of traffic to one endpoint and later use the ID extracted from responses in subsequent requests.

00:22:11.700 In our custom defined controller, we can standardize data import using the Ruby gem. This also allows for diverse assertions, such as checking response codes and durations.

00:22:50.900 Next, we’ll explore how to run commands to execute tests using command-line options. We leverage the OptionParser class to set parameters like test duration.

00:23:22.730 The command processes inputs and executes tests accordingly, either creating test plans or running tests based on conditions.

00:24:10.630 Now, I'll present a demo of how the command functions. Although I won’t conduct a live demo due to concerns over potential issues, I can show you the expected outcomes.

00:24:48.270 Upon running the command, you can see generated test plans that include user ramp-up times and configurations. This gives you an idea of how the tests will execute.

00:25:29.420 We can visualize results as the test runs in non-GUI mode, with logs indicating success and failure rates.

00:25:53.590 To see real-time data, we can use an open-source tool called Teraform created by BlazeMeter, which provides live reports and metrics during the tests.

00:26:40.940 After running tests, we can view results through dashboards showing success rates and response times.

00:27:12.600 Next, I’ll show you how to run the test plan using the JMeter dashboard. We’ll import the test plan and see how all the configurations take shape.

00:27:47.670 As the tests run, we retrieve request and response data for various endpoints, alongside a statistical overview of testing metrics.

00:28:15.720 After all tests, we analyze our Application Performance Monitoring results to validate whether our performance criteria match the set benchmarks.

00:28:47.590 We examine results, particularly spikes in errors and response times, to identify problematic endpoints with longer queries.

00:29:13.860 After identifying issues, we address them promptly. Now, I’ll introduce our three-dimensional scaling model.

00:29:43.740 The y-axis represents splitting application components for independent management. The x-axis signifies scaling by increasing database replicas. The z-axis relates to categorizing traffic effectively.

00:30:28.850 We strive to distribute traffic strategically and fine-tune resource allocations—considering AWS instance types and memory. Additionally, we enhance our codebase to better handle load.

00:31:07.330 To summarize our learnings, baseline tests should be conducted first to determine benchmarks, and we must recognize test differences between production and testing environments.

00:31:45.420 We also need to understand how to ask pertinent questions to ensure performance tests meet stakeholder needs and to remain aware of external dependencies that might affect outcomes.

00:32:10.030 Expect to face bottlenecks in unexpected areas, and for those interested in further information, I invite you to visit the Ruby JMeter GitHub repository and my performance test generator repository.

00:32:46.640 Reflecting on our tests with NASA's public API, we observed that while our error rate and memory usage remained normal, CPU usage spiked occasionally.

00:32:56.460 Fortunately, all key functionalities operated as expected, leading to a positive overall assessment. Thank you!

RubyConf AU 2020