Enough Statistics So That Zed Won't Yell At You

Performance Testing

Devlin Daley

2 talks

Enough Statistics So That Zed Won't Yell At You

by Devlin Daley

The video titled "Enough Statistics So That Zed Won't Yell At You" features Devlin Daley at the MountainWest RubyConf 2008, where he aims to introduce viewers to basic statistical concepts necessary for software development and analysis.

Key Points Discussed:

Introduction to Statistics: Daley emphasizes the importance of statistics as a tool for creating models that represent data accurately, using well-known examples like planetary motion and human body temperature.
Understanding Distributions: He describes the normal distribution and how it affects data analysis, noting that the average temperature (98.6°F) varies and showcases this variability using the bell curve.
Sampling Techniques: Daley explains the significance of representative sampling to draw valid conclusions about a population, exemplified with personal anecdotes about sampling family height.
Statistical Testing: The presentation delves into statistical tests, particularly the z-test and t-test, explaining how they are used to determine the significance of observed differences, such as distinguishing between Diet Coke and Coca-Cola Zero.
Application in Web Performance Measurement: He discusses the practical aspects of statistical testing, especially in evaluating web application performance, and how to utilize tools like HTTP Perf for this purpose.
Common Pitfalls: Daley warns about the dangers of confounding variables and the necessity of controlling for these in any experimental design.
Importance of Sample Size: He underlines the role of sample size in testing, detailing how the Central Limit Theorem supports the assumption of normal distribution for larger samples.
Use of Statistical Software: The application of statistical software, especially R, is highlighted as a valuable resource for conducting statistical analysis.
Conclusions on Methodology: He concludes that understanding and correctly applying statistical principles is essential for accurate analysis, urging the audience to maintain clarity in their definitions and methodologies to avoid misinterpretation of data.

Takeaways:

Proper statistical understanding is critical in software performance analysis and data-driven decision-making.
Sampling methods and statistical tests provide frameworks for objectively evaluating hypotheses.
Misinterpretation can arise from inadequate sampling or unrecognized confounding variables, emphasizing the need for careful experimental design and continuous learning from community resources.

00:00:17.840 All right, so I'm presenting today on 'Enough Statistics So That Zed Won't Yell At You.' My name is Devlin Daly.

00:00:24.660 I actually contacted Zed about this talk, and I asked him if I would be able to present enough information to prevent him from getting upset. He said probably not, but we could try, so that's what we're going to do today.

00:00:42.480 Some caveats: I am not a statistician. I'm actually into digital identity. If you want to talk about OpenID, OAuth, SRP, or permissions later on, I’d be happy to discuss that. I have taken a few classes in statistics, and Pat sort of roped me into this presentation; it's all Pat's fault.

00:01:02.399 Today, my goal is to familiarize you with some vocabulary and basic concepts related to statistics.

00:01:09.479 The basic concept behind statistics is that it emerged due to the need for models. Models should be familiar to us because software itself is a model. There are different mathematical models describing how planets orbit the sun in elliptical patterns.

00:01:21.780 While these equations can describe planetary motion accurately, the data does not always fit the model perfectly; there is often an error function involved. This error is what statistics studies.

00:01:41.280 For example, consider the temperature of the human body. This isn't just one fixed number; it varies. We typically say it is about 98.6 degrees Fahrenheit, but in reality, it is distributed in a manner described by the normal distribution, commonly represented by the bell curve.

00:02:03.299 In this distribution, 98.6 degrees would represent the average. When measured, temperatures will always be slightly above or below this average due to random fluctuations. In a normal distribution, we can determine how much data falls close to the mean.

00:02:31.020 For instance, when considering standard deviations around the mean, we expect about 68 percent of the data to fall within one standard deviation, and about 95 percent to fall within two standard deviations. This means not every data point will fit neatly within these ranges, but on average, we see these distributions hold.

00:02:57.360 When measuring something, we often encounter distributions that aren't perfect normal distributions, but they are still distributions. Distributions are characterized by their center—usually the mean—and by their spread, which is indicated by the standard deviation.

00:03:12.360 To further illustrate the relationship between the mean and the spread, consider two samples represented by blue and orange distributions. Both have the same average, but the orange distribution has greater spread in its data values. If we are looking at web server performance, we might misleadingly conclude that performance is the same because the averages are identical, yet the larger spread indicates potential problems.

00:03:39.060 Returning to the example concerning human body temperature, finding the true average requires measuring everyone in the world, which is practically impossible. Therefore, we must sample the population. A sample is a subset used to estimate the characteristics of the entire population. It's crucial that the sample is representative, or the data we gather can lead to false conclusions.

00:04:49.979 For example, if I were to sample only my family to estimate the average height of people, I'd misleadingly conclude that everyone is around six feet tall, since I'm taller than my family. To infer correctly about a population, we need a good representation of it, and that typically requires us to collect multiple samples.

00:05:21.900 One critical aspect of statistics is sampling—it’s essential. Single instances of tests or benchmarks yield singular data points; we need multiple samples to detect variance and establish a distribution. I work for Phil Windley, who is known for his love of Diet Coke.

00:06:02.039 He runs a conference where he ensures there’s plenty of Diet Coke available, much more than any other beverage. For example, if we wanted to find out if Phil could distinguish between Diet Coke and Coca-Cola Zero, we could design a simple experiment.

00:06:51.840 He could taste two cups without labels, and if he guesses right, there is a 50/50 chance of success. This method, however, is not accurate because it doesn't account for chance alone. To ensure validity, we need a larger sample size and randomization. If Phil guesses correctly more than 50% of the time, we'd determine he can indeed tell the difference.

00:07:50.400 Statistical testing, specifically the z-test, compares the proportion of correct guesses to what would happen if he were just guessing. This is how we apply logic and structure in statistical tests. We start with a null hypothesis, for instance, that Phil cannot tell the difference. The null hypothesis assumes a 50% probability of guessing correctly if he cannot tell the difference.

00:09:01.680 With statistical tests, we either reject or fail to reject the null hypothesis. If the tests show enough evidence, then we can say he might be able to distinguish between the two drinks.

00:09:59.880 An example of an application of statistical testing is A/B testing in conversion rates. If I run a website and change its design, I need to test if the changes actually improve customer conversion. We would need a representative sample of visitors to compare their conversion rates.

00:10:40.680 Issues might arise if the new site is underperforming due to a technical error, such as being incompatible with a specific browser. Statistical analysis would allow us to isolate these factors, ensuring that performance issues are accurately attributed to the web design and not user errors.

00:12:11.220 Let's switch to performance measurement for web applications. For instance, using a tool like HTTP Perf, we can measure how many requests our application can handle per second. We need to sample this population over time, pulling out multiple measurements to analyze average performance accurately.

00:13:12.540 When using a web server like Mongrel, we can start by measuring the performance per second. After making optimizations or changes to our application, we can establish if those changes improve performance using statistical methods like the t-test, which assesses whether observed changes in application performance are statistically significant.

00:14:35.640 The t-test evaluates two sample estimates of the same population, allowing us to determine if there is a statistically significant difference between them. When running performance benchmarks on different hardware configurations, we can identify if there is a true performance difference or just random variation.

00:15:38.640 This significance is expressed through a p-value, which indicates the probability that observed effects are due to randomness alone. Essentially, the t-test is an approximation of a permutation test, comparing the observed results to the entirety of randomized possibilities.

00:16:23.040 In practical scenarios, we can use statistical software, such as R, to perform these tests easily. For example, given samples from two groups, we can determine if they come from the same distribution using t-tests to inform our decisions.

00:17:31.500 While overlapping distributions may give us p-values indicating similarity, we need to assess whether this closeness is simply due to chance or captures genuine differences.

00:18:32.050 As we've discussed, confounding variables can obscure the results of statistical testing. To manage these, one should control experimental conditions by only changing one variable at a time.

00:19:49.200 It's critical that measurements are not affected by external factors. For performance tests, using separate machines for testing and production can minimize this risk.

00:20:34.320 Testing on a virtualized environment can also introduce variance due to shared resources. When using services like Amazon EC2, you must ensure adequate sampling to account for potential inconsistencies.

00:21:59.960 In summary, we have several statistical tools available to us, such as the z-test and t-test. To avoid mistakes in measurement interpretations, peer reviews and cross-checks are essential.

00:23:36.240 Resources like R can assist in statistical analysis, while good laboratory practices enhance the reliability and reproducibility of our tests. Engaging with the community and learning from others can also promote better understanding and application of statistical methods.

00:25:00.480 Lastly, discussions around the nature of measurements, like what defines a user in performance assessments, remind us of the importance of maintaining clarity about what we measure.

00:26:23.940 Sampling is essential; random sampling increases the likelihood of representative samples over repeated experiments. Additionally, the central limit theorem suggests that sample means tend toward a normal distribution with enough samples.

00:28:25.260 Determining the appropriate size for a sample relies on understanding the population dynamics and the required precision of results. Ensuring your statistical efforts are appropriately structured enhances their credibility.

00:29:24.540 In conclusion, understanding and correctly applying statistical principles is crucial in analyses, especially when working with complex systems. Causation cannot be assumed, and careful parameter management is needed for efficacious analysis.

00:30:38.340 The nuances of statistical methods, like Bayesian approaches, add depth to our understanding of data; they help define the origins of our observations within established probabilistic frameworks.

MountainWest RubyConf 2008