Metric Driven Development with Ruby on Rails

Talks

Metric Driven Development with Ruby on Rails

Jeff Casimir

#database-performance

#user-experience-ux

Metric Driven Development with Ruby on Rails

by Jeff Casimir

The video titled "Metric Driven Development with Ruby on Rails," presented by Jeff Casimir at LA RubyConf 2012, focuses on enhancing performance in Ruby on Rails applications through metrics-driven approaches. Casimir emphasizes that scaling applications successfully involves careful design and optimization rather than just adding hardware. He outlines essential methods for identifying performance issues, including:

Performance Metrics: Understanding that performance is critical beyond functionality. Testing is not enough; developers should be aware of how quickly applications operate.
Runtime Performance: Stressing the difference between development and production performance, where real-world user demands can reveal slowdowns not visible in smaller tests.
User Experience: Highlighting that developers should see themselves as contributors to user experience, where fast applications lead to happier users and improved business outcomes.
Goal Setting with Metrics: Using tools like New Relic for performance tracking helps gauge the impact of application changes.
Profiling Tools: Introducing tools like PerfTools.rb to analyze CPU usage and identify bottlenecks. Casimir uses practical examples, such as profiling a sample blog application to reveal inefficiencies in database queries and memory usage.
Optimization Techniques: Discussing specific techniques, such as improving database queries using the Bullet gem to identify N+1 queries and suggesting practices like includes or eager_load.
Awareness of Garbage Collection: Addressing how Ruby's garbage collector can impact performance and exploring techniques to monitor and manage its effects.

To conclude, Casimir encourages developers to adopt a mindset of measurement and analysis over assumption, pressing that achieving performance is a continual process informing development decisions. The session underscores the importance of aligning performance goals with user satisfaction and overall application success.

00:00:00.080 foreign

00:00:23.920 Jeff is one of those presenters I've seen on the conference circuit. The one advantage of doing conferences regularly is that I get to meet great speakers.

00:00:30.560 You did present in Austin, right? I've seen him present at RubyConf, and one interesting part of his early presentations was that he has been doing software development full-time for zero years.

00:00:36.320 So, zero years! Awesome! But he came to software development after being a high school teacher. He now runs a company called Jumpstart Lab.

00:00:42.399 I'm going to let him talk a little more about himself, but we're glad to have him here. He did a great job with the workshops for us over the last two days as well.

00:01:17.360 Good morning! I'll try to move quickly to get you to lunch and so forth. The presentation I have for you today is about metrics-driven Ruby on Rails performance.

00:01:23.119 It's a topic I've become increasingly interested in over the last two years. I think it's one of the pieces that we're currently missing in our community.

00:01:28.560 Ruby has been famous, or perhaps infamous, for emphasizing testing over the past several years. However, performance considerations are the next step beyond testing. Testing proves that your application works, but that's not enough—it also needs to work quickly.

00:01:35.119 We're developing a growing consciousness around this need. Let's talk about how to get started with performance metrics.

00:01:41.439 As Kobe mentioned, my name is Jeff Casimir. Jumpstart Lab is my company, and with me today is Steve, who will be speaking later.

00:01:48.159 In terms of the percentage of Jumpstart Lab employees speaking here, it's 100%. We're taking over—watch out, L.A.!

00:01:54.320 In March, I was contractually engaged with the board associated with LivingSocial, where I'll be teaching a group of 24 people who know somewhere between little and nothing about programming.

00:02:04.880 After five months, they will join the LivingSocial engineering team. It's an experiment in long-term, intense developer training.

00:02:11.280 If you’re interested in this, we're planning another session in September, and I can tell you more about it during the day.

00:02:20.080 Today, we're going to discuss runtime performance—how does your app perform in production? This can be very different than during development.

00:02:29.360 One key mistake people make when interested in performance is to create sample data in development, run their code, and everything seems fine.

00:02:36.959 However, in production, having a thousand users is often a lot slower than just five users or whatever small number you used for testing.

00:02:45.760 We rarely automate the process of creating large amounts of test data. How often in your app do you think, 'This works well enough, so let me write a rake task to generate 10,000 objects and see how it feels?'

00:02:53.440 Examining runtime performance is a fascinating issue, and what you'll often find is that your database is the problem.

00:03:06.159 Ruby community members often quote, 'Optimization is unimportant,' emphasizing that if you worry about performance, you're not using Ruby correctly.

00:03:12.959 However, the full quote warns against premature optimization, which raises the question of when optimization becomes necessary.

00:03:20.080 What point is your software considered mature enough that it must perform well? It's only when a poor user is trying to use it that you need to care about performance.

00:03:30.080 The Ruby movement started around the concept of joy. You hear speakers often discuss the joy of Ruby programming. The joy of using software is when it operates as expected and does so quickly.

00:03:43.440 If software functions as intended but is slow, that does not bring joy. Similarly, if it runs quickly but does not meet your needs, it also leads to dissatisfaction.

00:03:51.200 Mature software is only a concept when people are actively using it. Users really are the ones who care about performance, while developers often overlook this aspect. However, they should prioritize user experience.

00:04:04.360 The greater your application's performance, the happier your users will be. An interesting article from either Fred Wilson or Y Combinator highlighted how the performance of applications impacts user enjoyment very early in the interaction.

00:04:15.120 When applications respond quickly, users enjoy the experience more. The more responsive your application, the more users you will gain. More users also lead to increased revenue.

00:04:26.560 Ultimately, performance translates into user experience. If I conducted a survey asking how many of you consider yourselves experienced in user experience, I bet only about ten of you would raise your hands.

00:04:40.160 This lack of recognition is a shame. We need to take the stigma off of UX; it isn’t just about being a social media expert. UX is a critical aspect, and as developers, you are among the most important contributors to user experience.

00:04:52.640 Every line of code you write directly influences user experience. You need to perceive yourself as a user experience developer or expert, acknowledging that it's the only thing that truly matters.

00:05:06.720 If your users dislike your application, no amount of beautiful patterns or plugins, or sophisticated testing frameworks will matter. They will simply represent a waste of time.

00:05:17.920 Let's discuss goal-setting. When you work with larger companies, like Etsy, they employ impressive metrics. They can chart everything from deploys to sales increases, helping analyze the impact of changes.

00:05:29.440 Etsy's scale makes it easier for them to test these ideas. For example, with a million transactions a day, even a small change can yield significant results.

00:05:38.240 However, if you have just three transactions a day, it’s challenging to simply obtain that fourth one. Etsy has advantages in scale, allowing them to directly correlate all aspects of their work with customer results.

00:05:49.760 When it comes to online shopping, like with Etsy or eBay, if a page takes more than half a second to load, users often navigate away, forgetting about the initial tab in the process.

00:06:02.560 This situation can lead to lost sales, where simply improving performance just a little bit would have made a tangible difference.

00:06:14.640 When you begin your performance metrics, aim for a threshold of 500 milliseconds: if a server-side response takes longer than this, it is considerably slow.

00:06:21.920 You may not notice the delay immediately during development since hitting refresh appears almost instant. But in production, the delays can feel painfully significant.

00:06:30.560 Matt Ayman, who shared my room, pointed out that server responses should ideally be under 100 milliseconds. Once you transmit over 500 milliseconds in user-side response, you're likely to experience noticeable delays.

00:06:45.440 If the delay is over 100 milliseconds, users will perceive your application as sluggish. It's essential to measure and focus on these metrics right from the beginning.

00:06:52.560 There's a common misconception that performance problems can always be fixed later. While this is true to a certain extent, if you wait too long, it can become incredibly challenging.

00:07:04.360 This sparked my interest in tracking performance more seriously. One excellent resource for this is New Relic.

00:07:11.280 I didn't use New Relic for a while, but after returning to it, I found it incredibly useful. I'm not affiliated with them, but I sincerely recommend their tool.

00:07:17.920 If your application generates revenue and you're not using New Relic, you're likely missing out on a valuable performance analysis tool.

00:07:26.720 New Relic provides a transparent way to dive into your code, allowing you to see issues regarding database load directly. This visibility can lead you to identify problem areas quickly.

00:07:36.160 Another vital aspect we need to discuss is performance analysis more in-depth. When looking for performance metrics, you may take a more scientific approach.

00:07:44.960 For effective performance tracking, we need to analyze CPU, memory, and database usage.

00:07:52.000 Let’s get started with CPU analysis by using PerfTools.rb. This tool, developed by Google, serves as a CPU monitoring utility that runs alongside your executed code, gathering information over time.

00:08:01.360 If someone mentions C programming, I tend to feel a bit anxious about having to write in it, but thankfully, Aman writes the C code so we don't have to.

00:08:08.960 Aman is one of my favorite personalities in the Ruby community. If you're interested in performance, his contributions are invaluable, and you will find them throughout various tools.

00:08:16.560 He has wrapped PerfTools with Ruby, enabling us to continue working in our comfort zone while still taking advantage of performance insights.

00:08:30.240 PerfTools measures every 10 milliseconds, checking what method is being executed and retrieving its stack trace.

00:08:38.320 This tool answers two burning questions: which methods are consuming time, and it's intriguing to check who calls these methods.

00:08:46.080 While the functions you write might not be CPU-intensive, they could be calling other methods that are either very slow or repeatedly executed, impacting overall performance.

00:08:56.320 PerfTools operates as a command-line program and can also be integrated into a Rack middleware. Thus, you have a convenient option, and you don’t even need to access the terminal to use it.

00:09:06.320 To get started, you just need to install the middleware and set it up in your Bundler, configuring an initializer with whatever output format you'd prefer.

00:09:12.440 If you append 'profile=true' to your URL, you’ll see performance data instead of the page content.

00:09:18.560 This data will often reveal concerning statistics, for example, that a significant portion of execution time is consumed by the garbage collector or ActiveRecord.

00:09:25.840 To accurately interpret the output produced, it’s helpful to understand that PerfTools doesn't categorize information efficiently, often presenting raw numbers without clear label headings.

00:09:32.640 You'll see on the left the sample count, but what often matters more is the rightmost column, indicating the total time percentage attributed to a method and those it calls.

00:09:39.360 This illustrates the overall responsibility for CPU cycles taken by a given method. It shows the impact of not only its cycles but also those of methods it invokes.

00:09:46.480 If you produce a PDF output, it will richly visualize the information, displaying complex graphs that illuminate your performance metrics.

00:09:54.080 If you apply it while using Rails in Rack middleware, you'll observe countless middleware dependencies and processes being executed.

00:10:01.600 These stacks can provide insightful visualizations that help you pinpoint which methods are taking excessive time and where potential bottlenecks reside.

00:10:08.320 The text outputs can be beneficial for identifying CPU-heavy methods, while the PDF format excels at displaying call stacks in a clear manner.

00:10:14.960 Let me show you an example from my sample project. This is a simple blog, which I built during a workshop, where I established a thousand articles.

00:10:21.680 The front page features a dashboard displaying five recent articles and comments, alongside some statistics.

00:10:28.160 I intentionally wrote this application to be inefficient, so I would have relevant metrics to analyze performance.

00:10:35.280 Interestingly, I found that the methods I designed to be inefficient weren't actually as slow as I anticipated.

00:10:42.960 One method computes word count by querying all articles from the database, splitting down their words, and summing them up. I thought this would be slow, considering I had a thousand articles.

00:10:50.080 Surprisingly, it executed in just a quarter of a second; however, the dashboard still took nearly two and a half seconds, which is excessive.

00:10:56.760 When running the text output, I discovered that SQL Lite and garbage collection were the primary contributors to that delay.

00:11:03.520 This insight indicated that I was likely executing many small queries, not just a few large ones, causing significant object creation.

00:11:10.480 It’s common to find scenarios where hundreds of queries are needed, creating numerous temporary objects that quickly become garbage.

00:11:17.920 As I analyzed the data, my goal was to identify what I authored and trace back through the stack to locate potential performance issues.

00:11:25.680 Among the most active methods implemented was one intended to find the most popular articles based on comments.

00:11:32.320 Initially, I aimed for a counter cache to demonstrate its significance, but I soon discovered it consumed an astonishing 60% of the runtime.

00:11:38.880 That revelation shocked me! I had no idea it would be that significant. After identifying the issue, my first step was to avoid trying to fix it immediately.

00:11:45.920 Instead, I attempted to validate that I had correctly pinpointed the source of the issue.

00:11:52.960 To simplify, I replaced the operation designed to find the most popular articles with the straightforward command to grab just one article using 'article.first'.

00:11:59.520 As expected, the execution time dropped drastically from 2300 milliseconds to 600 milliseconds. While 600 milliseconds is still not ideal, that's a significant improvement.

00:12:05.600 This highlighted a 70% reduction in runtime. My next plan was to tackle the cause of the performance issue.

00:12:12.240 PerfTools provides a quick way to profile the CPU usage of specific methods. You can supply it with a block to focus on particular elements within your application's performance.

00:12:19.440 Running it as a standalone can streamline the process; you can apply it directly to the method, excluding all the middleware and other unnecessary executions.

00:12:25.200 Evaluating the CPU time spent revealed visibly that the bottleneck resided within the database interactions.

00:12:32.000 This finding implied I was making excessive database queries, and optimizing this interaction could lead to further performance gains.

00:12:39.200 The insights from profiling the CPU performance as well as memory and databases can guide you on fixing inefficiencies.

00:12:46.640 If you take away one concept today, let it be that PerfTools.rb is incredibly powerful and you should try it out.

00:12:54.440 Many of the performance tracking tools can feel intimidating, but PerfTools allows you to stay within Ruby while still utilizing these advanced techniques.

00:13:01.520 Next, let’s discuss memory profiling. One of the first gems that comes to mind is the library 'memprof', which tracks the use of objects.

00:13:08.560 Although it’s a fantastic tool for version 1.8.7 of Ruby, it may need to be left behind as newer versions are adopted.

00:13:15.680 The latest capabilities around profiling object allocations can still be enjoyed through PerfTools.

00:13:22.560 By appending ‘mode=objects’ in your URL, you can focus on tracking object allocation instead.

00:13:31.680 The output provided can show you which methods allocate the greatest number of objects, although analyzing this data can be complex.

00:13:39.920 Finding actionable improvements can sometimes feel more challenging compared to the clarity provided with CPU profiling.

00:13:45.440 When you encounter results of allocating an extraordinarily high number of objects, it can lead to thoughtful optimization strategies.

00:13:52.520 For example, if you notice that you're creating many hashes, each with multiple string keys, consider switching to symbols instead.

00:13:58.560 This simple change can dramatically reduce the number of objects created and stored in memory.

00:14:05.040 Another tool available for memory analysis is the middleware designed by Matt Ayman, which tracks garbage collection activity.

00:14:11.440 As you run your application, you gain real-time insights into how many requests have passed since the garbage collector last ran.

00:14:18.640 For those unfamiliar with Ruby’s garbage collection, be aware that MRI utilizes a simplistic method. When it’s time to collect garbage, execution pauses while the collector examines each object to determine if it’s still needed.

00:14:26.640 This can severely impact your application’s performance, as all execution halts until garbage collection completes.

00:14:32.840 If you are seeking to improve Ruby performance but dislike the current garbage collection method, consider exploring options with languages like C++.

00:14:39.440 In Ruby 1.9.3, a new feature allows manipulation of the garbage collector directly through its module, leading to various optimizations.

00:14:47.840 You can inquire about the garbage collector's activity or the number of objects being cleaned up, which helps assess efficiency during runtime.

00:14:55.040 Next, let's address database performance. This is a common area where most applications can improve.

00:15:01.680 One of my personal favorite tools for database optimization is the gem 'Bullet'. It's a handy tool that helps identify N+1 query issues.

00:15:08.640 In most cases, you may encounter situations where it pulls all parent objects and their children instead of querying the database efficiently.

00:15:15.920 It tracks these problems quietly in the background, sending alerts when it detects when your application is performing multiple queries that could be optimized.

00:15:25.840 Bullet identifies those occurrences, and the recommendation to remedy the situation can often be to use includes or eager_load to prevent what are essentially tiny, repeatable queries.

00:15:32.320 That being said, always analyze the recommendations Bullet provides to discern whether they indeed lead to performance improvements.

00:15:39.920 For example, when you utilize the includes method, you may end up running significantly slower queries instead of fixing the overall query performance.

00:15:48.560 These changes depend critically on the data structure, so gauge the size of your queried data, the frequency of access, and monitor your results consciously.

00:15:55.840 A common beginner mistake is misusing ActiveRecord methods where equivalent operations yield different results, such as length versus count.

00:16:05.280 Utilizing length retrieves all result objects, thus calling multiple database queries, while count applies a simple 'COUNT(*)', returning just an integer.

00:16:12.960 This simple change can significantly enhance performance, showcasing how precise adaptations can yield considerable improvements.

00:16:21.440 Additionally, you can implement the 'slow jam' gem, which monitors your log file and alerts you to slow queries.

00:16:28.960 This gem complements the insight gained from the Bullet gem by helping you find which actions are causing high latency.

00:16:35.520 While it may not explain why the queries are slow, knowing which queries run slow is the first step to addressing these issues.

00:16:42.560 The Rails 3.2 release introduced auto-explain features, which could also assist you in investigating slow queries, although it requires further debugging.

00:16:50.720 Unfortunately, some features released may need additional tweaks before fully operational.

00:16:57.520 My biggest takeaway today is the distinction between guessing and precise analysis of performance. One major mistake developers make is to neglect performance entirely.

00:17:05.840 The second mistake is to presume knowledge of what operations are efficient or inefficient without fitting it into measurable results.

00:17:12.080 Ultimately, avoid guessing—measure and know.

00:17:16.960 If you're interested in the slides, tools, and sample projects, be sure to visit the following GitHub repository for all related materials.

00:17:29.520 Our team at Jumpstart Lab offers top-notch Ruby and Rails training programs, and if you're interested in improving your company's skill set, please get in touch.

00:17:35.680 Thank you!

00:33:02.240 no

00:33:29.200 you

LA RubyConf 2012