Patterns of Basecamp's Application Architecture

Talks

David Heinemeier Hansson

#software-architecture

#community

Patterns of Basecamp's Application Architecture

by David Heinemeier Hansson

The video titled "Patterns of Basecamp's Application Architecture" features David Heinemeier Hansson, the creator of Ruby on Rails and co-founder of Basecamp, discussing the intertwined evolution of Rails and Basecamp's application architecture. In his talk at Rails Conf 2013, Hansson emphasizes the importance of real-world application as the basis for developing frameworks, arguing that frameworks are derived from actual applications rather than abstract features. He showcases how features and improvements in Rails were often inspired by challenges faced while building Basecamp.

Key points discussed include:

- Rails Development: Hansson reflects on his decade-long experience with Rails, noting its evolution and the community's pioneering spirit that fosters innovation.

- Historical Context: He contrasts the technological landscape of 2003 (the era of flip phones and Java) with today, showing how dramatically technology has evolved while also questioning why Rails remains relevant.

- Community Involvement: The growth of the Rails community is highlighted, with increasing contributions leading to new features and improvements, underscoring the notion of collaborative development.

- Application Architecture: Hansson delves into Basecamp’s architecture, advocating for a document-oriented approach where software behaves more like hypertext documents rather than traditional GUI applications.

- Caching Strategies: He presents various caching strategies employed in Basecamp to optimize performance, including key-based cache expiration and Russian doll nested caching, to enhance speed while minimizing complexity.

- Dynamic Elements: The addition of dynamic elements to Basecamp through JavaScript, enabling updates without the need for page reloads, was also discussed as a way to enrich user experience while maintaining document-like structures.

- Future of Rails: Concluding, Hansson emphasizes the need for ongoing innovation and adaptation, urging developers to embrace change and keep progressing rather than settling for comfort.

In conclusion, Hansson's insights promote a philosophy of practical application-driven development, collaboration, and the pursuit of speed through smart architectural patterns, which continue to guide the evolution of Basecamp and Rails. His reflections encapsulate a belief in the potency of simplicity and the value of embracing the document-oriented nature of web applications.

00:00:16.720 Welcome, everyone. Before I dive into my talk about the patterns of Basecamp's application architecture, I have a little announcement. About an hour and 20 minutes ago, we released release candidate one.

00:00:30.970 We've also branched for Rails 4.0 stable, so we're very close to completing and shipping 4.0. It would be fantastic if everyone here who plans to work on Rails code over the next four days could try it out. That is the best way for us to identify what might still be missing or to catch any bugs. We have a history of releasing betas and release candidates, and then receiving fifty tickets filled with obvious bugs or broken features once the final version ships.

00:00:58.420 It would be great if we could launch 4.0 and for the first week or two at least not have to issue an update. So, if you can help us out with that, it would be amazing. Now, the story of Rails is intertwined with the story of Basecamp. This is a screenshot of the very first version of Basecamp released in 2004. I started working on Basecamp back in 2003, before I had any ambitions to work on Rails, and before I had any idea that what I was building would evolve into a framework.

00:01:30.970 My interest was in developing an application, and the need to create my own tools was a secondary concern. This led to a unique style of working which, I believe, we've fully embraced as a community within Rails core. Frameworks are not inventions; they are extractions. We don’t sit down and brainstorm what features we could want; instead, we build actual applications that must function in the real world—things that matter to people. From there, we identify the components that we repeatedly use.

00:02:09.790 This is why Basecamp is incredibly important to me. I can say with confidence that nearly every positive feature I’ve incorporated into Rails has originated from Basecamp or a derivative of it. A few bad features emerged when I mistakenly thought I could invent good frameworks, which never turned out well. The question we should ask ourselves is not, 'What would someone, somewhere maybe want to do with Rails?' That speculative approach has never proved successful.

00:02:56.900 During my journey with various tools designed to 'help' developers, I've often seen that they don't meet the expectations I have for real-world development. As such, my goal became to focus solely on creating the perfect framework for building Basecamp. This inadvertently led to creating something that resonates with many other applications, despite the fact that Basecamp is not unique in its functionalities. By tackling problems from this concrete angle, we’ve managed to extract an experience that just feels right—something that’s genuinely been used under pressure.

00:03:49.940 I've had my fair share of frustrations with Rails at times, especially when something takes four lines of code when it really should only take one. I find that our best work surfaces when I’m irritated enough about a situation that pushes me to innovate. This contrasts sharply with the example-driven development approach that many employ in their day-to-day work because it’s easier to generate pretty examples.

00:04:36.900 In Rails, we had this standard scaffold that for a long time appeared verbose and redundant, leading many newcomers to say that the code generated was overcomplicated. A few have attempted to optimize it through various gems, only to find that making example code visually appealing sacrifices practicality for real-world applications.

00:05:13.320 Since I began working on Ruby on Rails and Basecamp back in 2003, it has now been ten years—a significant portion of my life. I don't think it’s an exaggeration to say that this journey represents my life's work up till this point. It’s amusing because when I look at Rails code today, it still resembles the Rails code I wrote back then. Of course, it has evolved in numerous ways and is vastly improved, but the core principles remain unchanged.

00:05:58.580 Reflecting on the changes since 2003, I recall that flip phones were once considered state-of-the-art technology. It’s funny to think how far we have come. Many aspects of technology advance quickly, yet here we are, still discussing, debating, and enhancing a framework that originated in the era of flip phones. It’s also interesting to note that many users interacting with our applications today primarily do so through mobile devices.

00:06:41.250 When examining the state of hardware, I found that at one time, half a gigabyte of RAM cost around $50; now, you can get sixteen times as much for nearly half the price. This radical change poses interesting questions about how we rethink our approach to development. Often, people lock their understanding of hardware costs to the time they started programming, thinking that if it was scarce and expensive when they began, it would always be so.

00:07:16.730 For instance, in 1993, when Matt first started working on Ruby, he had to be concerned with the costs of RAM, which was indeed quite high. Yet, with the foresight that resources would improve over time, he understood that the bottleneck really wasn't hardware but instead the limitations of human cognition. The mental processes behind programming are the same as they were in the early '90s, so we need to design languages and environments that enhance our productivity under those constraints.

00:08:23.590 It’s notable that back in 2003, Java was still a significant contender, and many serious developers worked in it. Reflecting on the early challenges of Rails, we were often trying to convince Java developers that there were better ways to achieve what they were trying to do. The underlying design philosophies of creating well-structured web applications had indeed derived from the Java world, but many were constrained by the limitations of the technology at that time.

00:09:08.150 Since then, whilst there are still people programming in Java, and the JVM continues to thrive, Rails has allowed developers to get freedom from templated solutions that were previously in use, like Struts. This is intriguing to contemplate in light of how much technology has shifted over the past decade and why Rails has managed to stay relevant when so many other frameworks have fallen by the wayside.

00:10:00.500 It challenges my initial assumptions about my work. When I started developing Rails, I could never have imagined standing here now, speaking to a sold-out audience ten years later—a much different world than the one in which flip phones were considered cutting edge. As I reflect upon this journey, I realize our spirit of innovation and willingness to adapt is what defines the Ruby and Rails communities.

00:10:41.910 We don’t simply settle for comfort; we continue to push forward. Many communities place a high value on stability, which can hinder progress, whereas the Ruby community has taken bold steps to prioritize innovation, even when it involves pain. Transitioning from Ruby 1.8 to 1.9 or from Rails 1 to 2 to 3 and now 4 wasn't easy, highlighting that any real progress requires sacrifice.

00:11:14.580 In contrast, other communities, like Python, have struggled for years to adopt Python 3. It’s a fascinating illustration of how the Ruby community, while not immune to challenges, has been willing to embrace change, making them unique within the programming world.

00:11:46.910 As we look toward the future, it’s essential for our community to remain ambitious. If we had simply parked our ambitions, believing that Rails 3 or Ruby 1.8 was sufficient, then we wouldn’t be experiencing the advancements we are now achieving. Pioneering efforts, like those of Joe Spolsky in 2001, remind us that creating meaningful software takes time. We have gradually built upon our investment, both in Rails and in Basecamp, leading to the fruitful progress we’re witnessing today.

00:12:20.670 The journey we've taken is not solely about features, but about the people who have engaged with Rails, contributing once they learned about its potential. Back in 2004, the first contributors alongside myself offered patches and wrote the initial commits. In 2005, that number increased significantly, but since then, there has been a steady influx of new contributors. Following the economic crisis in 2009, many developers resonated with what Rails offered, and interest spiked.

00:13:06.740 This was key to advancing the community; it’s intriguing to see how we moved past a comfortable plateau into fresh engagement and enthusiasm for contribution. Moving forward, we encourage responses from those moving through our ecosystem—many of whom may outgrow us, ultimately seeking other options, representing a natural progression in technology.

00:13:40.350 However, I think it’s essential to remain open as we consider how to foster developer satisfaction instead of locking ourselves into a secluded comfort zone. The important aspect is ensuring that we balance growth while embracing change so as to best serve our user base. These patterns of engagement and evolution are integral—we should recognize this company as a dynamic entity, continuing to adapt as our users’ needs evolve.

00:14:23.290 Returning back to the patterns of Basecamp’s application architecture, we know patterns emerge from solutions to problems in a specific context. I believe the context for Basecamp lies in creating dynamic hypertext documents. Although it seems simple today, I find that the architecture emphasizes the continuing relevance of basic principles surrounding web documents.

00:15:02.230 While coding frameworks may seem minimal compared to so-called complex systems, I argue that embracing simplicity is crucial. Recognizing the web for what it is—an interconnected set of documents, is profound in guiding software development approaches. Dividing applications into the two categories of document-based and GUI-based brings our development back to their roots in a way that enhances user experience.

00:15:48.430 This delineation is relevant for various applications—from something strictly simple like Wikipedia to those with greater dynamic functionality like Google Maps, which does not feel document-oriented. Striving to enhance how we perceive documents in developing applications is fundamental. Rails feels natural when we approach software design as a document-centric framework, allowing smoother interactions and steering clear of complexity.

00:16:20.790 Designing for the document-based web means focusing on HTML and avoiding unnecessary GUIs that can detract from the user experience. Embracing a document-centric structure facilitates user understanding of how our application works, making interfaces clearer and more intuitive, further emphasizing the profound connection between user experience and design choices.

00:17:10.260 The evolution of Basecamp observed a rethinking of its interface, leading to its material design being that of a stack of pages, promoting the notion of documents. Additionally, the new version aims to embrace this philosophy while managing its intricate layers without needing a complex GUI.

00:18:02.540 Fulfilling that commitment to a document-based approach means recognizing the significance of HTML’s semantics, to avoid variables that may create confusion. In an era of quick consumption and static interfaces, it becomes crucial not to overcomplicate solutions while still enhancing performance.

00:18:44.600 All of this leads us toward a richer, more coherent use of both HTML and Rails, where applications can evolve to meet the real-world requirements of our users while still channeling the original design ethos. Overall, by planning through documenting and breaking components into structured pieces, we find fluid constructions that allow us to maintain usability.

00:19:34.830 Furthermore, this document-oriented framework enables us to automate many of the processes that would typically require substantial manual intervention. Applauding this revolutionary understanding of application development conveys an ethos that base our development on existing functionalities, while driving innovation to enhance energy within the community. Our practices have demonstrated that while knowledge of our foundational principles is critical, implementation is equally significant.

00:20:20.290 So, as we navigate this document-focused world, it’s imperative to appreciate how we continuously shift our views back toward what our work means. Ultimately, the user interface's simplicity remains key as we focus on accessing, delivering, and serving information with improved technology in the most valuable way.

00:21:01.360 In conclusion, I encourage all of you to explore patterns that unpack the opportunities available with Rails that line up with your goals and aspirations. Embracing simplicity doesn’t mean sacrificing functionality or innovation, but rather facilitating a methodology that meets our users where they are, whether they’re in a position to innovate continually or merely seeking clarity in their work.

00:21:38.060 As we continue on this shared journey to innovate and improve upon what we’ve already built, always acknowledging the foundational work that serves as our launching pad, we can go so much further together. Thank you for your attention.

RailsConf 2013