Ruby Video
Talks
Code spelunking in the All New Basecamp
Nick Quaranto
@qrush
#ruby-on-rails
#javascript
#elasticsearch
#redis
#performance
#code-quality
Summarized using AI

Code spelunking in the All New Basecamp

by Nick Quaranto

In the video titled "Code spelunking in the All New Basecamp," Nick Quaranto discusses the significant features and technical innovations of the recently launched Basecamp Next, which is built on Ruby on Rails. The presentation aims to share various coding practices, patterns, and techniques that were leveraged during the development of Basecamp Next, providing insights on how these can enhance Rails applications.

Key points discussed include:

- Code Reusability with Concerns: Quaranto emphasizes using Ruby on Rails' concerns to share code across models and controllers, which helps maintain cleaner and more manageable codebases.

- Stacker and CoffeeScript: The CoffeeScript component behind the page layout, referred to as Stacker, allows for dynamic content updates through long polling, improving user experience.

- Client-side Testing: The importance of client-side testing is highlighted, with tools like Capybara and custom testing frameworks discussed as avenues to streamline front-end testing processes.

- Separation of Presentation Logic with Jbuilder: Jbuilder is utilized to maintain view data in views rather than models, improving the organization of the code.

- Simplified Development Setup: A single setup script is proposed for easing the onboarding of new developers and maintaining sanity in the setup process of the application.

- Debugging Techniques: Methods for debugging JavaScript performance issues are examined, stressing the importance of logging and performance tracking in live applications.

- Strategies for Managing Dependencies: Suggestions on keeping applications running smoothly even with potential external service failures (like Redis) are addressed.

- Enhanced Logging Practices: Logging techniques, including tagged logs and SQL query logging, are discussed as essential tools for troubleshooting and maintaining the health of the application.

- API Documentation with Jbuilder: Quaranto explains using Jbuilder for JSON API responses, enhancing the documentation process within GitHub for better manageability.

- Instrumentation for Metrics: The use of StatsD for performance metrics tracking is introduced, showcasing how it helps in analyzing application performance and usage patterns.

Quaranto concludes that continuous learning and evolving practices have been key to Basecamp Next's success. His presentation encourages developers to explore these patterns to improve their own applications, making the development process more efficient and effective. Overall, this session provides valuable insights into modern web application development, specifically within the Ruby on Rails ecosystem.

00:00:24.660 Hey everyone! There we go. Hi! I'm Nick Quaranto on Twitter, @nickq, if you want to bug me. I currently live in Buffalo, New York. I was in Boston for a while, but now I'm back in my hometown of Buffalo. There are a few guys from the Western New York Ruby group here, so say hi if you see them! I work at 37signals, where we're known for having a lot of loud opinions, including mine.
00:00:41.800 Today, I'm hoping to share with you a lot of old and new patterns that I've found and learned while looking through the Basecamp Next codebase, which I'm going to refer to as BCX on and off. That's kind of our internal codename for it. I'll also share some neat gems that we've used and how they work. That's why we're here at RailsConf—to see how other people do this stuff. It's really exciting to learn from one another.
00:01:01.269 You may hear conflicting opinions depending on which talks you’ve just attended, and that’s totally fine. We’re all just going to see how things work. Here’s a little graph I whipped up with a cool program called Gorse. It tracks Git repositories and apparently fires little lasers out from people's avatars. I think this is fun! I have some snapshots to share.
00:01:19.390 Basecamp Next has been launched for about 50 days now, and it’s been a crazy ride for us. It’s been really intense and active. This has been under development for 12 months, starting back in April. There was a little SWAT team that dropped in to see if it was worth rewriting. As things kept rolling along, more team members came on. When I joined in December, we basically worked full-on until the launch in March.
00:01:58.450 There have been around 14,000 commits, give or take a few due to branches being thrown about. Interestingly, the code-to-test ratio is one-to-one, which is controversial. I’m sorry, but that’s just how the ratio is, and it’s almost precisely on the dot, which is kind of scary.
00:02:23.340 Now, I want to go over a little bit of what the app looks like—not just from a production perspective, but also how it appears to developers getting started with it. Of course, it's all based around a big Rails app, and we use a few databases.
00:02:47.879 We still use MySQL and Memcache a ton for caching. DHH wrote a great blog post about how we have a terabyte Memcache cluster, which I think just started getting filled up about a week or two ago, almost 50 days post-launch. That’s pretty crazy! My suggestion was to buy another terabyte, but our Ops team didn’t like that idea.
00:03:10.980 We also use Redis a bunch for Resque. Mostly, we utilize Redis for live updates on pages, and MySQL is still a major player. I would say Memcache is used more than Redis in the app. Additionally, we use Elasticsearch for our search functionality, which is pretty cool.
00:03:34.829 There are several other services that the Rails app depends on, which we haven't discussed much. Depot is one of those; it’s where we store files. I think it’s like an old Rails pun, 'the depot.' There’s another service called Portfolio that handles avatars for all our apps.
00:04:01.260 Another service we use is Launchpad. If you’ve ever gone to the single sign-on section where you select which products to sign into, that’s Launchpad—it manages all the sign-in authentication for Basecamp. Finally, Cream Bee, or Queen Bee as we call it, handles billing and communicates with all the apps.
00:04:18.780 You might think, 'Is this all just a blob? Is it a monolith?' One thing I've learned from having all these services intertwined is that I often feel like I have no idea what I'm doing. I think everyone at the company is learning how to grow the architecture, and we’re all trying to make it better.
00:04:49.460 Over the years, they have been improving it, and I think we’ll continue to enhance all the components I just mentioned.
00:05:06.200 Now, let’s talk about how to set up Basecamp Next on a local machine. It’s pretty simple—all you need is MySQL and those other dependencies like Redis and Memcache. You don’t need them to get started or to run the tests. I think that’s really neat because it allows everyone to get started on the app much quicker. If you actually do need to use them later, you can just drop them in.
00:05:40.850 All of this is fronted by POW. POW was written by my coworker Sam, and it’s a really nice OSX-enabled server for Rails and Ruby apps. It’s written in Node, which is kind of weird, but it’s truly an awesome piece of software. It’s obscenely simple; you basically just symlink in the directory where your Rails app is, and POW takes care of the rest. If you haven’t checked this out for local development, please do—it’s fantastic! I haven't typed 'rails server' in months thanks to POW.
00:06:43.610 To work with Basecamp Next locally, you also need Launchpad for signing in and Portfolio for handling avatars. Setting up these two additional services is something you only need to do once. The way POW works is pretty neat, too. It registers a whole top-level domain on your local machine at .dev, which amazes me. Portfolio lives at 37image.dev, along with BCX and Launchpad, so most of the time you’ll just be accessing bcx.dev. POW makes service development easy for us.
00:07:58.000 Now, regarding testing, there are some opinions circulating that all we do at 37signals is programming, but I can assure you that we have an extensive test suite. We primarily use Test Unit and Mocha for stubbing and mocking, and I’ve introduced a few things that no one has yelled at me about yet.
00:08:55.270 I’ve introduced Capybara, which we mostly use for acceptance tests, and we also leverage Capybara for the selector API that we use in Action Controller tests. This selector API is much nicer than using search selectors, and I honestly don’t know why those are still around.
00:09:17.870 I also wrote a little gem called 'n' that allows you to run a test by line number if you're using Test Unit or MiniTest. This little gem saves me a few minutes here and there, which is pretty neat.
00:09:51.590 So what does the actual test suite look like? This isn’t discussed enough, but I find it quite interesting. We have the three typical test suites in our app. We also have a separate test suite for handling JSON, which Mike Morris talked about yesterday regarding presenters. We have an entire test suite for posting JSON with the correct headers and parsing it back properly.
00:10:25.000 We also have tests for the Polar service, as it needs to be tested, and there’s a Rails engine shared across all our apps called 37 ID that we also run tests for to ensure functionalities, like email validation, work correctly in this app along with the others.
00:11:02.170 There are also two additional test suites on the fritz that I want to get working again. We have an acceptance test suite that primarily uses Test Unit and Capybara workflow. While it has been operational, we’re in the process of getting it working effectively again. The integration tests are going well, so far.
00:11:43.400 We also have client tests—a CoffeeScript test runner to assess some of the complex interactions on the front end hasn’t been fully baked yet. We’ve been discussing looking into PhantomJS and some other test runners, as we do have an aversion to Selenium. Hopefully, once it's fully matured and working within our build process, we can start using it.
00:12:24.050 One cool thing I learned about all these different test suites is that we can chain Rake tasks. I hadn’t seen this kind of setup before. I’ll show you how we define the API tests in Basecamp Next.
00:12:41.580 You can do a Rails subtest task, or you could do a Rake test task. This may do some setup, but this just highlights where my tests live, which libraries I need, and then we invoke this nifty Rake API. We can load a specific Rake task, and instead of overriding an existing one, we can hook behavior after it runs.
00:13:19.439 The cool part is that this behavior will also fail if the original test fails. So, if our unit, functional, or integration tests don’t pass, the API tests won’t run either, which is crucial.
00:14:01.110 So, how are we doing so far? I’m glad to hear that! Excuse me, I also need some water!
00:14:43.560 Now, let’s discuss development setup. Typically, we don’t talk about this kind of thing, but I think it’s vital. Not only is getting new developers set up important, but it’s essential for everyone's sanity. How many times have you opened a Rails app and wanted to blow on the cartridge to fix bugs?
00:15:08.750 When things aren’t working, it can be frustrating, and wasting hours trying to get it running is not ideal. I really believe that all apps should have a reset button. Instead of banging on it, everything should just reset, and then we can continue. At 37signals, we have one bash script that does this, and I wish this were more conventional.
00:15:43.210 Typically, I execute 'scripts/setup' and then 'rake.' So, if I’ve just woken up and pulled from Git, I set everything up and get a clean build—ready to roll for the day.
00:16:00.500 Generally, the 'script/setup' typically does a few things: it sets up the app, which usually involves bundling gems, resetting the database, loading the schema, migrating data as needed, restarting the server, and potentially performing any custom wrangling you may need.
00:16:35.170 This is a reset button that I highly recommend having. Additionally, I suggest keeping it quiet. There's a neat little bash trick where you can wrap a bunch of commands in brackets and redirect their output somewhere.
00:17:23.360 This helps us when we say, 'installing libraries' because we let bundler throw a fit about what’s wrong while we do our magic—and if it fails, we can redirect that output to a log file to see what happened.
00:17:45.580 Another piece of advice is to use flags to speed up the process. Some setup steps were notably slow for us, so someone introduced a flag to keep the database alive during resets, skipping the time-consuming steps.
00:18:06.110 Now, let's shift gears and talk about concerns in Rails. Concerns stem from a widespread sentiment that sharing code in Rails can be challenging. It's a recurring theme in our work, as we need to refrain from repeating ourselves. However, sharing code isn’t as straightforward as one might hope.
00:18:54.460 The way we started tackling this is through using concerns, which is an ActiveSupport feature. With ActiveSupport::Concern, you can create modules that expose instance methods and include class methods with ease.
00:19:27.180 This pattern is widely utilized in our application. For instance, with our Ajax functionality, we have an Ajax concern that’s mixed into controllers. In the included block, we can create before filters and other methods.
00:19:50.450 This is great because it allows for much cleaner organization of methods. For shared behaviors like trashing items, searching across models exist as concerns, housing the necessary logic for controllers while keeping the application controller cleaner.
00:20:35.130 Modules aren’t classes, and if you add a bunch of methods, the main class can still become cluttered. However, by using concerns, we're enabling one of the simplest refactoring methods, which is to extract methods and share code among classes without causing a mess.
00:21:03.700 Assets are another crucial topic. Last week, I did a long count and found that we have around an equal number of lines between Ruby and CoffeeScript, with more tests than either. This ratio emphasizes the balance we strive for in our coding practices. We also have a fair amount of SCSS through SASS and various mix-ins, but I'm not the best person to discuss that.
00:21:42.170 In terms of JavaScript, we primarily use jQuery and Backbone, steering clear of Prototype. Recently, we brought over Node.js’s event emitter, which has proved invaluable. Of note is a library created by my coworker Sam called Echo, which serves as a really nice JavaScript templating framework.
00:22:46.290 For WYSIWYG needs, we use a well-made editor that has shown exceptional support from its author. If you’re in the realm of needing WYSIWYG functionality in your Rails app, this new editor will serve you well! We also use several other jQuery plugins that address various needs in the application.
00:23:37.910 Now, let’s transition to the subject of JavaScript patterns. Despite being Rails developers, understanding JavaScript best practices and patterns is essential, as it complements the Rails framework. Extending jQuery is commonplace for us, and thus we’ve created a variety of custom plugins.
00:24:08.090 One of the custom plugins helps highlight elements, giving a quick yellow flash, while another resets forms utilizing jQuery's native methods. It's important to remember that returning `this` in CoffeeScript can lead to strange syntax that may feel inconsistent.
00:24:38.890 We also encapsulate our JavaScript inside a global object, ensuring we avoid scope leaks. We utilize CoffeeScript's existential operator to create a global namespace for our application, preventing pollution of the global scope.
00:25:48.410 Data behaviors derived from the Rails UJS framework help us define custom JavaScript functionality through data attributes. This keeps our JavaScript logic separate from the views and allows more flexibility for our designers to modify styles without breaking functionality.
00:26:40.520 Here's how we establish custom behaviors with data attributes, applying various functionalities depending on the data assigned to each HTML element. It’s important to ensure consistency in the naming convention for these behaviors to avoid confusion in our code.
00:27:28.820 The event system we're utilizing in BCX allows us to intercept clicks and handle content updates dynamically, which enhances user experience remarkably. We are smart about managing errors in context, knowing that the clear feedback provided can save significant debugging time—this is especially important during local development.
00:28:33.400 Additionally, we have a design called Stacker that essentially layers pages on top of one another, providing a seamless interaction for users. When the page is updated, we intelligently render the new HTML and display it as part of the current view rather than performing a full page reload.
00:29:53.260 This method also ensures that any errors encountered during the loading process are displayed clearly, allowing rapid identification of issues without digging through the console.
00:30:34.680 Console logging is another essential aspect of our workflow. We use a variety of console methods that provide insight into what’s happening in real-time. Simple methods like `console.count` help us keep track of how many times specific actions occur, which can be instrumental in debugging.
00:31:53.280 Additionally, enhanced methods for logging warnings and errors can help to identify critical issues quickly, while grouping logs can give a clearer picture of events in context.
00:32:05.750 The browser's profiling tools are incredibly useful, allowing you to analyze the performance of your JavaScript execution, which is invaluable for optimizing load times. By understanding which functions slow down the user experience, we can target our optimizations effectively.
00:32:46.860 Moreover, we’ve embraced the use of HTTP status codes correctly, particularly the 204 No Content response for JSON APIs, ensuring we don’t send unnecessary response bodies that could lead to errors in JSON parsing.
00:33:30.580 Error handling in our application is treated seriously; we extensively log operations while capturing the context around requests through tagging, which aids in tracing issues back to their source. We've developed ways to verify unique identifiers across our logs, making troubleshooting much more efficient.
00:34:25.220 Lastly, our approach to logging doesn't just stop at tracking errors and operations. We use relationships between different parts of our system to enhance understanding. We've adopted StatsD for tracking metrics like successful logins and emails sent to optimize our user experience better.
00:35:56.150 That's about all I had to share! We're still learning and trying out new things with Basecamp Next; I don't think we'll ever stop evolving this system. I hope I've taught you something today, and I appreciate your attention. Thank you!
Edit
Explore all talks recorded at RailsConf 2012
+61