Double Dream Hands: So Intense!

by Aaron Patterson

In his keynote presentation titled 'Double Dream Hands: So Intense!' at RailsConf 2011, Aaron Patterson shares his insights and experiences as a prominent figure in the Ruby community. He begins by expressing gratitude to notable peers, including John and Jose Valim, before diving into technical discussions related to the latest Rails 3.1 features.

Key points from the talk include:
- Introduction to Aaron Patterson: He identifies as an open-source developer at AT&T, illustrating his personal enjoyment of working within a large corporation due to its innovative processes, such as an efficient mail-to-fax system.
- Rails 3.1 Features: Patterson presents several significant features added to Rails, focusing on their functionalities and improvements:
- New Generators: He humorously discusses a fictional generator that provides money during financial downturns, emphasizing the creative nature of Rails development.
- Prepared Statement Caching: He details how this feature speeds up database interactions by preparing and caching SQL statements, resulting in a substantial increase in performance metrics, particularly for SQLite and PostgreSQL, while critiquing MySQL's performance.
- Serialized Attributes: Patterson introduces the new ability to save arbitrary Ruby data structures in the database. He explains the transition from a YAML-dependent system to a flexible serialization format allowing various strategies.
- Streaming Responses: This feature allows Rails to start sending data to users before all data is processed, enhancing user experience by utilizing the Rack API.
- Reflection on Rails Performance: He critically evaluates Rails' performance over time, noting an increase in average request rates from Rails 2.3 to 3.1, pointing out the implications of background processing on performance due to increased stack depth and garbage collection issues.
- Community Engagement: Throughout his presentation, Patterson emphasizes the importance of community involvement in the Ruby ecosystem and the future trajectory of Rails, encouraging new contributors to participate in the framework's evolution.
- Interactive Dance Segment: To lighten the mood, he engages the audience in a fun dance routine, creating an atmosphere of camaraderie before wrapping up.

Concluding his talk, Patterson underscores the importance of refactoring code for reusability and efficiency while urging the community to embrace best practices to secure the future of the Ruby programming language. His presentation is not only informative but also entertaining, leaving the audience with an invigorating message about collaboration and innovation in the tech industry.

00:00:02.440 Before I begin, I want to express my love for the Ruby community and say thanks to a few people.

00:00:08.639 First off, I want to thank John for his excellent work on Active Record relation codes. You should all tweet at him and say thank you.

00:00:16.520 Next, I want to show some love to Michael Feathers. He told me he included my picture in one of his presentations, so I reciprocated by including his picture in mine. Thank you, Michael Feathers!

00:00:32.040 I also have great admiration for Jose Valim. I tried to kiss him the other night, but he wasn't too excited about it.

00:00:38.680 So, this talk is titled 'Double Dream Hands: So Intense!' What does it mean? Honestly, I don't know. Hopefully, you'll find out by the end.

00:00:44.720 Now I can begin. You can start the clock. Oh my God, happy RailsConf! In case you're lost and don't know where you are, my name is Aaron Patterson. You can find me online as Tender Love, and I work for a giant corporation called AT&T.

00:01:05.040 I know most of you are probably working for startups, but I happen to be an open-source developer at AT&T. My job is to work on tools that improve the development process and, hopefully, our bottom line.

00:01:19.040 I actually enjoy working for a large corporation, and I want to share why. One impressive aspect is how I file expense reports. It's a fascinating technology.

00:01:31.000 The process starts with me needing to VPN into our network since I work from home full-time. I fill out a form, detailing all the expenditures, and then I have to fax my receipts. It's a little cumbersome because I don't own a fax machine, so I usually pay a couple of bucks to upload a PDF so it can be faxed off.

00:01:51.960 One day, I reached out to our accounts payable person over email, explaining that I didn't have a fax machine and asking how I should send receipts. She advised me to compile everything into a PDF and send it to a specific email address.

00:02:04.000 I followed her instructions, and to my amazement, three minutes later, I received a response that said 'fax sent.' I couldn’t believe it—I emailed a PDF to a mail-to-fax system that faxed it for me. This technology is incredible!

00:02:23.679 This is why I enjoy working at such a large company. It’s also worth noting that I get quite nervous when speaking on stage. Jose gave me great advice: whenever you're on stage, ask yourself, 'What would Freddie Mercury do?'

00:02:35.120 I always include this little reminder in my presentations to help manage my nerves. However, it's even more daunting for me because I'm delivering a keynote address, and I just looked at the list of other keynote speakers—individuals who have changed the world, like Dick Gabriel, Guy Steele, and DHH.

00:02:59.239 We even have some incredible CTOs keynoting, like Dr. Nick, who is an actual doctor, and Corey Haines, whose title is literally his name. I find that amazing! Meanwhile, I'm just a senior software engineer—how can I possibly compare?

00:03:17.679 As I pondered this, I received an IM from Chad, who asked if I wanted to hear a story about sex in the tech industry. Naturally, I was intrigued. He referred to an email that O'Reilly had sent about Rails, which included click tracking on the keynote speakers' pictures.

00:03:30.680 Chad let me know that my picture was clicked more than any other speaker's. I was shocked! I started to wonder why that was. What did you click on and what is it about me?

00:03:44.640 I began reflecting on what sets me apart from the other keynote speakers. Well, I have expertise in Ruby and Rails, and I'm the only one on both the Ruby core team and the Rails core team. Additionally, I must admit, I look a bit ridiculous up here.

00:04:04.000 With that in mind, we have three items on our agenda today. First, I'll be discussing the new features in Rails 3.1. Next, I will have some straightforward conversations. Finally, we will look at some development pro tips.

00:04:17.960 I must warn you, this talk will be very technical. Prepare yourselves, as I'll also be critical of our framework. However, I want to clarify the difference between being critical and being negative. I will be pointing out some issues, but we'll also discuss improvement strategies to make our already awesome framework even better.

00:04:35.160 Let's proceed with the new features. We'll examine what has been added to Rails 3.1 over the past year. Before diving into that, however, I want to share my strategy for modifying APIs.

00:04:56.719 My approach to modifying APIs revolves around two core principles: not changing them, thereby retaining the same interface you use today, and enhancing them for better performance, while always ensuring backward compatibility.

00:05:12.680 Among the new features in Rails, we added several new generators. Since I can't see anyone here, I won't ask for a show of hands, but I'll just assume that most of you work at startup companies, probably around 90%.

00:05:29.679 One new generator included is `rails generate vc`. So, when you're running low on funds, just run this, and boom—money! Apparently, it only works in San Francisco.

00:05:45.679 Another interesting addition by DHH is called `halt?`. You give it a function, and it informs you whether that function completes successfully. It might not seem impressive, but this could change the face of computer science.

00:06:05.759 Today, I want to elaborate on prepared statements, particularly the prepared statement caching support added to Rails 3.1. Typically, we send SQL statements to our database that look like this, but now we want to send statements formatted with placeholders for the values.

00:06:19.480 In our standard Rails application, we generate a SQL statement, send it to the database, and get records back, only to repeat that process over and over again. In the new world, we’ll send a statement off to the database and tell it we plan to execute that statement in the future, regardless of the values.

00:06:37.040 The database responds with a token, processes the SQL statement, prepares a query plan, and caches that plan. For subsequent queries, we simply send the token and the actual values we want to publish, which means that the database needs to execute the query plan and return the results.

00:06:56.159 These four steps are reduced to two: we prepare the statement once and reuse that token for multiple queries. Normally, when we execute queries directly, it takes the database a lot of time, so this new approach streamlines everything.

00:07:13.560 To give you tangible insight, let's look at the impact on the three different database adapters we have: SQLite, PostgreSQL, and MySQL. With SQLite, we can observe a clear performance increase.

00:07:31.180 Before implementing prepared statement caching, our average execution rate was around 8,600 queries per second. However, after the implementation, we increased that number to almost 13,000 queries per second. This is especially significant with simple statements.

00:07:51.439 I also examined more complex SQL statements to see how performance changed. It’s great to see a linear performance graph; as complexity increases, we're witnessing considerable improvement. For the most straightforward query imaginable, like 'SELECT * FROM wherever WHERE ID equals something,' we’ve enhanced performance drastically.

00:08:09.360 PostgreSQL showed us solid linear performance too, increasing from 4,600 to 5,500 queries per second. I was very satisfied with these results. As we ramp up complexity, we still see remarkable performance improvements.

00:08:27.160 On the other hand, MySQL didn’t show promising results. Our performance remained linear, but we experienced degradation in the number of queries per second. When I discussed the work necessary to reduce steps back on the database side, MySQL did not deliver as expected.

00:08:45.440 When sending a prepared statement to MySQL, it doesn't perform query planning in advance. It only begins the planning process upon executing the statement, which results in two network round trips. This is a very different path than performing normal, straightforward queries.

00:09:02.160 The performance results differ drastically; with prepared statements, we only save on SQL parsing time, leading you to evaluate whether the time saved compensates for the two additional round trips.

00:09:20.080 Additionally, benchmarks indicate that using the prepared statement API returns parsed dates as dates; thus, it's challenging to determine if prepared statements are genuinely slower across the board.

00:09:38.000 Now, how can we utilize prepared statements in Rails? The changes made to the API allow you to search for a user like this: `User.find(...)` and, I'm pleased to share, you'll experience significant speed improvements—all for free!

00:10:04.800 The next topic I want to discuss is serialized attributes. This feature allows you to save arbitrary Ruby data structures into your database. For instance, you could save a user's preferences as a hash without problems.

00:10:20.680 Previously, this data was stored in YAML format, which left me wondering why we decided on YAML instead of JSON or other formats. Upon inspecting the Active Record source code, I found that it was tightly coupled with YAML.

00:10:39.680 So, I refactored it to decouple YAML from Active Record, allowing any storage strategy we want. We can now supply a second parameter to the `serialize` method that indicates the chosen strategy.

00:10:56.160 To implement a custom strategy, you only need to provide a loader and a dumper to the `serialize` method. This new API opens up exciting possibilities for storage options: Base64, JSON, Marshal, XML, and beyond.

00:11:14.120 Let me announce that Rails now supports NoSQL. Everyone should be excited about this! However, I must caution that some of the things I say on stage may not be true willingly.

00:11:30.320 To show off the feature, I wanted to create 26 users with preference hashes—storing unstructured data is now a breeze! It's straightforward: we'll save these preferences to the database, retrieve them, modify them, and save them back.

00:11:46.760 We can also find users who have their favorite color set to green. Given that I'm the only one with my favorite color set to green, we should only find one record in the database.

00:12:02.360 While it would be amazing to store unstructured data in a database and run queries on it, we must remember that there are limitations. This feature only works on PostgreSQL, and you might encounter some complexities while using it.

00:12:20.840 For those who want to try implementing this, I'll share some example code that you can use to experiment. But I must emphasize that you should exercise caution, especially since there is some eval involved.

00:12:36.840 It's essential to think about various serialization strategies, including encryption methods. Implementing a coder to encrypt data before saving to the database is quite feasible.

00:12:54.360 This coder can encrypt sensitive information like passwords, and you can utilize a method to store these securely in the database. However, we need to be mindful of a key issue: the API must remain consistent for better usability.

00:13:12.560 When working on this API, it was crucial for me to maintain consistency with existing libraries, such as YAML, JSON, and Marshal. This consistency helps make the API intuitive and easier for developers to adopt.

00:13:28.960 An important lesson learned here is that strong abstractions lead to useful features. I didn’t set out with the specific aim of implementing JSON storage or bcrypt; instead, I focused on removing unnecessary coupling to YAML.

00:13:45.360 By decoupling, I made the codebase flexible and extensible, leading to this feature's emergence. Working with good abstractions opens up unexpected results in our code.

00:14:03.360 On that note, I want to contrast this with a feature that has been somewhat controversial: has_secure_password. When you declare has_secure_password in your model, it automatically creates a password field and encrypts it before storing it.

00:14:20.560 Its advantages, like using bcrypt for security, are compelling. However, I am concerned because it hides essential details and relies on implicit assumptions that might not be clear to developers.

00:14:34.880 Due to this lack of transparency, it becomes challenging to reuse this feature in existing applications. Many of my applications don't use a column for passwords that aligns with this method, nor do we necessarily use bcrypt.

00:14:47.840 As I design APIs, I strive to promote reusability. The inability to utilize has_secure_password in various contexts limits its effectiveness.

00:15:02.720 In a similar context, the serialization strategies I discussed earlier demonstrate a more flexible design. So, anyone interested in contributing to Rails Core could refactor has_secure_password to be more aligned with these strategies.

00:15:16.360 Next, let's talk about streaming responses, a new feature in Rails 3.1. The idea is to enable immediate data transmission to users before the complete data is assembled.

00:15:29.760 In previous versions of Rails (like 3.0), we processed all ERB buffers in memory first, then sent the output. With Rails 3.1, we aim to spit out data as soon as it's ready—this way, we're not holding up client requests while processing on the server.

00:15:46.760 To understand how this is implemented, we need to examine the Rack API. In a simple Rack application, you must implement a method called 'call' that receives an environment and returns a triple. Note that the third element returned represents the body, which must be iterable.

00:16:02.360 In Rails 3.0, we append content to the body as we render it. The challenge with streaming is that we need to delay evaluations until the socket is fully established.

00:16:17.760 We need to postpone evaluations until we have a connection, encountering a need to properly handle socket operations while processing takes place.

00:16:32.360 One potential issue we encountered during implementation focuses on how middleware functions in chains. Middleware forms a linked list, where request timers, connection managers, and applications continually call one another.

00:16:53.160 If the connection manager opens a database connection and then delegates to the next middleware before closing its connection, it results in exceptions. We need a solution for this to enable seamless streaming in conjunction with database connections.

00:17:09.760 To manage this, we leverage the fact that Rack calls 'close' on the body after iterating through it. Thus, we implement a body proxy that delegates to the actual body and ensures that the database connection closes appropriately at the end of the process.

00:17:25.000 Our connection manager opens and delegates to the app, creating our proxy body without invalidating any connections. This proxy allows us to handle closures efficiently.

00:17:43.360 Though it’s complex, reshaping the middleware stack requires managing multiple linked structures carefully while ensuring that interactions do not negatively impact performance.

00:18:00.520 We can't jam all types of content processing into the same middleground. Instead, we should embrace diversity and separate the generators, filters, and connection management processes.

00:18:14.680 Now, it's time for some real talk. Rails is getting slower because more work is being done in the background. My benchmarks of Rails 2.3 versus Rails 3.0 revealed that our current average request rates are lower.

00:18:33.760 We've introduced many background processes, such as body proxies in our middleware, resulting in increased stack sizes. Although the increases in request-response handling occurred, we've seen the downside in performance.

00:18:52.960 The main bottleneck can be attributed to garbage collection pressure. As stack depths increase, Ruby's garbage collector has to handle more objects, leading to longer processing times.

00:19:09.760 With Rails 2.3, the stack was around 51 deep; Rails 3.0 now measures 60 deep, and Rails 3.1 will reach 67. If we could lower this stack size, we might boost performance while doing the same amount of work.

00:19:27.360 To address this issue, we might need to adapt our Rack API to eliminate dependencies on return values. We should explore more event-driven architectures that support streaming more efficiently.

00:19:44.920 I have a prototype of such an approach that maintains a linear growth pattern for requests and responses. It has shown significant promise, demonstrating that we could improve request processing metrics if we adapt our practices.

00:20:03.360 As we look forward to the coming year, let's focus on enhancing speed, memory consumption, and optimizing stack depth while ensuring backward compatibility remains strong.

00:20:19.840 In the spirit of openness, I consulted many colleagues and gathered their insights that morphed into development pro tips. Some of these recommendations would be great to discuss now.

00:20:34.080 This leads me to share a fun activity with everyone here! Everyone should bring their heads down, look up slowly, and follow some fun dance moves with me. This will serve to lighten the mood!

00:20:49.600 Now let's get our arms involved! Raise both arms up and bring them down. Get ready—things are about to get lively!

00:21:05.500 You will be punching across while cranking up your energy and moving your shoulders to the rhythm! Double dream hands to the audience now—let's get it moving!

00:21:19.500 Slide into a fun dance sequence! Crank is an essential part of this. We've got a rhythmic vibe going, and each punch counts—stay in sync with everyone!

00:21:35.300 Now, let's get those jazz hands going—feel the energy all around you! Thumbs up as we tap into our fun moves with enthusiasm.

00:21:51.700 Two step claps and knee bumps! Make sure we’re jazzing it up, and let’s keep that energy flowing as we reach out for the audience together, having a great time!

00:22:06.400 Feel free to explode with double dream hands—everyone’s in a tight group, and it’s a moment to celebrate what we've achieved together!

00:22:24.000 Before we wrap things up, I want everyone to come together for a grub photo, capturing this exciting journey we shared, so let's get back up here as we celebrate!

00:22:40.000 Thank you for being a fantastic audience! The excitement and energy really brought the stage alive! Remember, there's an important message to ponder on moving forward.

00:22:55.400 The Rails core team, like any team, evolves over time. DHH won't be developing on Rails forever, but I believe this framework has a bright future ahead.

00:23:11.200 It's crucial that we refactor our work to ensure that new contributors can enter this space and join the core team. The openness to collaboration is what will keep Ruby thriving.

00:23:27.400 A final word of wisdom—remember that not all features are tangible. Emphasizing speed, memory efficiency, and good abstractions encourages reusable code.

00:23:44.000 The more we write reusable code, the more flexibility we’ll enjoy. Keep this in mind as your homework moving forward: always think about how you can refactor.

00:24:01.400 Thank you for being such an amazing audience! I really appreciate your energy and engagement today.

00:24:11.200 Feel free to connect with me and share your thoughts or experiences within the Ruby community. Together, we can continue to grow and evolve!