00:00:02.440
Before I begin, I want to express my love for the Ruby community and say thanks to a few people.
00:00:08.639
First off, I want to thank John for his excellent work on Active Record relation codes. You should all tweet at him and say thank you.
00:00:16.520
Next, I want to show some love to Michael Feathers. He told me he included my picture in one of his presentations, so I reciprocated by including his picture in mine. Thank you, Michael Feathers!
00:00:32.040
I also have great admiration for Jose Valim. I tried to kiss him the other night, but he wasn't too excited about it.
00:00:38.680
So, this talk is titled 'Double Dream Hands: So Intense!' What does it mean? Honestly, I don't know. Hopefully, you'll find out by the end.
00:00:44.720
Now I can begin. You can start the clock. Oh my God, happy RailsConf! In case you're lost and don't know where you are, my name is Aaron Patterson. You can find me online as Tender Love, and I work for a giant corporation called AT&T.
00:01:05.040
I know most of you are probably working for startups, but I happen to be an open-source developer at AT&T. My job is to work on tools that improve the development process and, hopefully, our bottom line.
00:01:19.040
I actually enjoy working for a large corporation, and I want to share why. One impressive aspect is how I file expense reports. It's a fascinating technology.
00:01:31.000
The process starts with me needing to VPN into our network since I work from home full-time. I fill out a form, detailing all the expenditures, and then I have to fax my receipts. It's a little cumbersome because I don't own a fax machine, so I usually pay a couple of bucks to upload a PDF so it can be faxed off.
00:01:51.960
One day, I reached out to our accounts payable person over email, explaining that I didn't have a fax machine and asking how I should send receipts. She advised me to compile everything into a PDF and send it to a specific email address.
00:02:04.000
I followed her instructions, and to my amazement, three minutes later, I received a response that said 'fax sent.' I couldn’t believe it—I emailed a PDF to a mail-to-fax system that faxed it for me. This technology is incredible!
00:02:23.679
This is why I enjoy working at such a large company. It’s also worth noting that I get quite nervous when speaking on stage. Jose gave me great advice: whenever you're on stage, ask yourself, 'What would Freddie Mercury do?'
00:02:35.120
I always include this little reminder in my presentations to help manage my nerves. However, it's even more daunting for me because I'm delivering a keynote address, and I just looked at the list of other keynote speakers—individuals who have changed the world, like Dick Gabriel, Guy Steele, and DHH.
00:02:59.239
We even have some incredible CTOs keynoting, like Dr. Nick, who is an actual doctor, and Corey Haines, whose title is literally his name. I find that amazing! Meanwhile, I'm just a senior software engineer—how can I possibly compare?
00:03:17.679
As I pondered this, I received an IM from Chad, who asked if I wanted to hear a story about sex in the tech industry. Naturally, I was intrigued. He referred to an email that O'Reilly had sent about Rails, which included click tracking on the keynote speakers' pictures.
00:03:30.680
Chad let me know that my picture was clicked more than any other speaker's. I was shocked! I started to wonder why that was. What did you click on and what is it about me?
00:03:44.640
I began reflecting on what sets me apart from the other keynote speakers. Well, I have expertise in Ruby and Rails, and I'm the only one on both the Ruby core team and the Rails core team. Additionally, I must admit, I look a bit ridiculous up here.
00:04:04.000
With that in mind, we have three items on our agenda today. First, I'll be discussing the new features in Rails 3.1. Next, I will have some straightforward conversations. Finally, we will look at some development pro tips.
00:04:17.960
I must warn you, this talk will be very technical. Prepare yourselves, as I'll also be critical of our framework. However, I want to clarify the difference between being critical and being negative. I will be pointing out some issues, but we'll also discuss improvement strategies to make our already awesome framework even better.
00:04:35.160
Let's proceed with the new features. We'll examine what has been added to Rails 3.1 over the past year. Before diving into that, however, I want to share my strategy for modifying APIs.
00:04:56.719
My approach to modifying APIs revolves around two core principles: not changing them, thereby retaining the same interface you use today, and enhancing them for better performance, while always ensuring backward compatibility.
00:05:12.680
Among the new features in Rails, we added several new generators. Since I can't see anyone here, I won't ask for a show of hands, but I'll just assume that most of you work at startup companies, probably around 90%.
00:05:29.679
One new generator included is `rails generate vc`. So, when you're running low on funds, just run this, and boom—money! Apparently, it only works in San Francisco.
00:05:45.679
Another interesting addition by DHH is called `halt?`. You give it a function, and it informs you whether that function completes successfully. It might not seem impressive, but this could change the face of computer science.
00:06:05.759
Today, I want to elaborate on prepared statements, particularly the prepared statement caching support added to Rails 3.1. Typically, we send SQL statements to our database that look like this, but now we want to send statements formatted with placeholders for the values.
00:06:19.480
In our standard Rails application, we generate a SQL statement, send it to the database, and get records back, only to repeat that process over and over again. In the new world, we’ll send a statement off to the database and tell it we plan to execute that statement in the future, regardless of the values.
00:06:37.040
The database responds with a token, processes the SQL statement, prepares a query plan, and caches that plan. For subsequent queries, we simply send the token and the actual values we want to publish, which means that the database needs to execute the query plan and return the results.
00:06:56.159
These four steps are reduced to two: we prepare the statement once and reuse that token for multiple queries. Normally, when we execute queries directly, it takes the database a lot of time, so this new approach streamlines everything.
00:07:13.560
To give you tangible insight, let's look at the impact on the three different database adapters we have: SQLite, PostgreSQL, and MySQL. With SQLite, we can observe a clear performance increase.
00:07:31.180
Before implementing prepared statement caching, our average execution rate was around 8,600 queries per second. However, after the implementation, we increased that number to almost 13,000 queries per second. This is especially significant with simple statements.
00:07:51.439
I also examined more complex SQL statements to see how performance changed. It’s great to see a linear performance graph; as complexity increases, we're witnessing considerable improvement. For the most straightforward query imaginable, like 'SELECT * FROM wherever WHERE ID equals something,' we’ve enhanced performance drastically.
00:08:09.360
PostgreSQL showed us solid linear performance too, increasing from 4,600 to 5,500 queries per second. I was very satisfied with these results. As we ramp up complexity, we still see remarkable performance improvements.
00:08:27.160
On the other hand, MySQL didn’t show promising results. Our performance remained linear, but we experienced degradation in the number of queries per second. When I discussed the work necessary to reduce steps back on the database side, MySQL did not deliver as expected.
00:08:45.440
When sending a prepared statement to MySQL, it doesn't perform query planning in advance. It only begins the planning process upon executing the statement, which results in two network round trips. This is a very different path than performing normal, straightforward queries.
00:09:02.160
The performance results differ drastically; with prepared statements, we only save on SQL parsing time, leading you to evaluate whether the time saved compensates for the two additional round trips.
00:09:20.080
Additionally, benchmarks indicate that using the prepared statement API returns parsed dates as dates; thus, it's challenging to determine if prepared statements are genuinely slower across the board.
00:09:38.000
Now, how can we utilize prepared statements in Rails? The changes made to the API allow you to search for a user like this: `User.find(...)` and, I'm pleased to share, you'll experience significant speed improvements—all for free!
00:10:04.800
The next topic I want to discuss is serialized attributes. This feature allows you to save arbitrary Ruby data structures into your database. For instance, you could save a user's preferences as a hash without problems.
00:10:20.680
Previously, this data was stored in YAML format, which left me wondering why we decided on YAML instead of JSON or other formats. Upon inspecting the Active Record source code, I found that it was tightly coupled with YAML.
00:10:39.680
So, I refactored it to decouple YAML from Active Record, allowing any storage strategy we want. We can now supply a second parameter to the `serialize` method that indicates the chosen strategy.
00:10:56.160
To implement a custom strategy, you only need to provide a loader and a dumper to the `serialize` method. This new API opens up exciting possibilities for storage options: Base64, JSON, Marshal, XML, and beyond.
00:11:14.120
Let me announce that Rails now supports NoSQL. Everyone should be excited about this! However, I must caution that some of the things I say on stage may not be true willingly.
00:11:30.320
To show off the feature, I wanted to create 26 users with preference hashes—storing unstructured data is now a breeze! It's straightforward: we'll save these preferences to the database, retrieve them, modify them, and save them back.
00:11:46.760
We can also find users who have their favorite color set to green. Given that I'm the only one with my favorite color set to green, we should only find one record in the database.
00:12:02.360
While it would be amazing to store unstructured data in a database and run queries on it, we must remember that there are limitations. This feature only works on PostgreSQL, and you might encounter some complexities while using it.
00:12:20.840
For those who want to try implementing this, I'll share some example code that you can use to experiment. But I must emphasize that you should exercise caution, especially since there is some eval involved.
00:12:36.840
It's essential to think about various serialization strategies, including encryption methods. Implementing a coder to encrypt data before saving to the database is quite feasible.
00:12:54.360
This coder can encrypt sensitive information like passwords, and you can utilize a method to store these securely in the database. However, we need to be mindful of a key issue: the API must remain consistent for better usability.
00:13:12.560
When working on this API, it was crucial for me to maintain consistency with existing libraries, such as YAML, JSON, and Marshal. This consistency helps make the API intuitive and easier for developers to adopt.
00:13:28.960
An important lesson learned here is that strong abstractions lead to useful features. I didn’t set out with the specific aim of implementing JSON storage or bcrypt; instead, I focused on removing unnecessary coupling to YAML.
00:13:45.360
By decoupling, I made the codebase flexible and extensible, leading to this feature's emergence. Working with good abstractions opens up unexpected results in our code.
00:14:03.360
On that note, I want to contrast this with a feature that has been somewhat controversial: has_secure_password. When you declare has_secure_password in your model, it automatically creates a password field and encrypts it before storing it.
00:14:20.560
Its advantages, like using bcrypt for security, are compelling. However, I am concerned because it hides essential details and relies on implicit assumptions that might not be clear to developers.
00:14:34.880
Due to this lack of transparency, it becomes challenging to reuse this feature in existing applications. Many of my applications don't use a column for passwords that aligns with this method, nor do we necessarily use bcrypt.
00:14:47.840
As I design APIs, I strive to promote reusability. The inability to utilize has_secure_password in various contexts limits its effectiveness.
00:15:02.720
In a similar context, the serialization strategies I discussed earlier demonstrate a more flexible design. So, anyone interested in contributing to Rails Core could refactor has_secure_password to be more aligned with these strategies.
00:15:16.360
Next, let's talk about streaming responses, a new feature in Rails 3.1. The idea is to enable immediate data transmission to users before the complete data is assembled.
00:15:29.760
In previous versions of Rails (like 3.0), we processed all ERB buffers in memory first, then sent the output. With Rails 3.1, we aim to spit out data as soon as it's ready—this way, we're not holding up client requests while processing on the server.
00:15:46.760
To understand how this is implemented, we need to examine the Rack API. In a simple Rack application, you must implement a method called 'call' that receives an environment and returns a triple. Note that the third element returned represents the body, which must be iterable.
00:16:02.360
In Rails 3.0, we append content to the body as we render it. The challenge with streaming is that we need to delay evaluations until the socket is fully established.
00:16:17.760
We need to postpone evaluations until we have a connection, encountering a need to properly handle socket operations while processing takes place.
00:16:32.360
One potential issue we encountered during implementation focuses on how middleware functions in chains. Middleware forms a linked list, where request timers, connection managers, and applications continually call one another.
00:16:53.160
If the connection manager opens a database connection and then delegates to the next middleware before closing its connection, it results in exceptions. We need a solution for this to enable seamless streaming in conjunction with database connections.
00:17:09.760
To manage this, we leverage the fact that Rack calls 'close' on the body after iterating through it. Thus, we implement a body proxy that delegates to the actual body and ensures that the database connection closes appropriately at the end of the process.
00:17:25.000
Our connection manager opens and delegates to the app, creating our proxy body without invalidating any connections. This proxy allows us to handle closures efficiently.
00:17:43.360
Though it’s complex, reshaping the middleware stack requires managing multiple linked structures carefully while ensuring that interactions do not negatively impact performance.
00:18:00.520
We can't jam all types of content processing into the same middleground. Instead, we should embrace diversity and separate the generators, filters, and connection management processes.
00:18:14.680
Now, it's time for some real talk. Rails is getting slower because more work is being done in the background. My benchmarks of Rails 2.3 versus Rails 3.0 revealed that our current average request rates are lower.
00:18:33.760
We've introduced many background processes, such as body proxies in our middleware, resulting in increased stack sizes. Although the increases in request-response handling occurred, we've seen the downside in performance.
00:18:52.960
The main bottleneck can be attributed to garbage collection pressure. As stack depths increase, Ruby's garbage collector has to handle more objects, leading to longer processing times.
00:19:09.760
With Rails 2.3, the stack was around 51 deep; Rails 3.0 now measures 60 deep, and Rails 3.1 will reach 67. If we could lower this stack size, we might boost performance while doing the same amount of work.
00:19:27.360
To address this issue, we might need to adapt our Rack API to eliminate dependencies on return values. We should explore more event-driven architectures that support streaming more efficiently.
00:19:44.920
I have a prototype of such an approach that maintains a linear growth pattern for requests and responses. It has shown significant promise, demonstrating that we could improve request processing metrics if we adapt our practices.
00:20:03.360
As we look forward to the coming year, let's focus on enhancing speed, memory consumption, and optimizing stack depth while ensuring backward compatibility remains strong.
00:20:19.840
In the spirit of openness, I consulted many colleagues and gathered their insights that morphed into development pro tips. Some of these recommendations would be great to discuss now.
00:20:34.080
This leads me to share a fun activity with everyone here! Everyone should bring their heads down, look up slowly, and follow some fun dance moves with me. This will serve to lighten the mood!
00:20:49.600
Now let's get our arms involved! Raise both arms up and bring them down. Get ready—things are about to get lively!
00:21:05.500
You will be punching across while cranking up your energy and moving your shoulders to the rhythm! Double dream hands to the audience now—let's get it moving!
00:21:19.500
Slide into a fun dance sequence! Crank is an essential part of this. We've got a rhythmic vibe going, and each punch counts—stay in sync with everyone!
00:21:35.300
Now, let's get those jazz hands going—feel the energy all around you! Thumbs up as we tap into our fun moves with enthusiasm.
00:21:51.700
Two step claps and knee bumps! Make sure we’re jazzing it up, and let’s keep that energy flowing as we reach out for the audience together, having a great time!
00:22:06.400
Feel free to explode with double dream hands—everyone’s in a tight group, and it’s a moment to celebrate what we've achieved together!
00:22:24.000
Before we wrap things up, I want everyone to come together for a grub photo, capturing this exciting journey we shared, so let's get back up here as we celebrate!
00:22:40.000
Thank you for being a fantastic audience! The excitement and energy really brought the stage alive! Remember, there's an important message to ponder on moving forward.
00:22:55.400
The Rails core team, like any team, evolves over time. DHH won't be developing on Rails forever, but I believe this framework has a bright future ahead.
00:23:11.200
It's crucial that we refactor our work to ensure that new contributors can enter this space and join the core team. The openness to collaboration is what will keep Ruby thriving.
00:23:27.400
A final word of wisdom—remember that not all features are tangible. Emphasizing speed, memory efficiency, and good abstractions encourages reusable code.
00:23:44.000
The more we write reusable code, the more flexibility we’ll enjoy. Keep this in mind as your homework moving forward: always think about how you can refactor.
00:24:01.400
Thank you for being such an amazing audience! I really appreciate your energy and engagement today.
00:24:11.200
Feel free to connect with me and share your thoughts or experiences within the Ruby community. Together, we can continue to grow and evolve!
