00:00:00
I am Nathan Walls, and I am a software developer at VitalSource Technologies. The primary office I work out of is in Raleigh, North Carolina, but we've got offices here and there. We're owned by the Encripto Contact Group, headquartered in Luverne, Tennessee.
00:00:16
Today, I'm going to be discussing the history of a suite of our applications organized around a common core. This core dates back to the very early days of Ruby on Rails. To start, I'll provide a little stage setting.
00:00:39
On this chart, you will see a rolled-up aggregation of seven different repositories that we have, dating from August 2, 2005, all the way to this past week. This chart shows the number of commits per week—it's not broken up by repository; it's just an aggregate number of commits. It provides a substantial, yet fundamentally incomplete view of what one segment of our company does. As you look from the left to the right, you can observe a general trend line that goes ever so slightly upward. Look just past this area around 2015 into 2016, and you will see a giant spike on the right side of the graph. We'll delve deeper into that area later in the talk.
00:01:20
The nice thing is that with these seven repositories, we have a lot of history, starting from Subversion and going all the way through our transition to Git. Now, let me start with a little prologue.
00:01:50
VitalSource does a number of things around eBook publishing. We allow publishers to send us books to build, set prices, determine availability, and manage other aspects. We then sell these books to end users, particularly students in learning applications. This began in October 2015 when we were in the middle of an upgrade from Rails 2.3 and Ruby 1.8.7. I was gaining experience across the app suite by working on this project.
00:02:21
One major task was to upgrade a lot of JavaScript. The application relied heavily on Prototype, which was being phased out in favor of jQuery. We had many old `rjs` style link-to-remote and other remote method calls that required updates.
00:02:51
The magical pieces of code weren't really JavaScript; they were just magical Ruby incantations that turned into JavaScript in the browser. This led to the need for upgrading that non-Ruby-flavored JavaScript, essentially consolidating everything on one version of jQuery.
00:03:15
During this process, I encountered some very early code and ran `git blame`, leading to a significant discovery. This was code from the original maintainer of the projects I was working on. This motivated me to reach out on Twitter and share that I looked at very old code today, which was originally started by Dan Benjamin, known from podcasting fame.
00:03:56
To my surprise, he responded, astonished that the code he had written back in 2005 was still in use and largely unchanged. This sparked my curiosity regarding the history of this application. I began looking at similar code bases connected to this one while preparing for this talk.
00:04:30
Earlier, I spoke with several members of both the development team, past and present, as well as members of the business team who were part of the inception and growth of these applications. That's what we'll discuss today—team growth, changes at a high level, interactions between the business and IT, code growth, and how this code has evolved.
00:05:02
We'll go into some code examples—not many, but enough to illustrate our points—see how developers and business interact, reflect on the available lessons, and review some tools folks can use to extract insights from their own code bases.
00:05:32
While preparing this talk, two quotes stood out to me: 'In the absence of logic, there's history,' and a repeated phrase from various people stating, 'It seemed like a good idea at the time.' In a twelve-year-old codebase, many lessons have been learned, and best practices evolve.
00:05:59
Keyules have displayed that you cannot start out with all best practices in hand. The positive takeaway is that this code has endured for twelve years because it has been successful for the business, allowing us to continue working with it.
00:06:29
Let's explore the history in four acts. The first act is the introduction of Rails. David touched on some of this earlier today in his opening keynote, mentioning the original announcement to the Ruby mailing list about Rails 0.50.
00:06:53
At that time, in July 2004 at VitalSource, there was a series of cobbled-together automation steps around constructing and assembling eBooks. It was primarily Java-based and quite limited. The team working on it was undoubtedly overwhelmed, seeking a more visual process. Thus, Rails entered the scene and made it feasible to create a robust web application.
00:07:25
Dan Benjamin was experimenting with Rails and found it more enjoyable than the combination of PHP for the front end and Java for the back end. Along with another developer, Damon Clinkscale, he started building this application at the Apple Worldwide Developer Conference in 2005.
00:07:54
They skipped most of the sessions to focus on writing the new Rails application, with Dan focused on the front-end and Damon on the back end. Looking back, we now rely on many features in Rails that didn't exist back then—Ajax was new, and Prototype and jQuery weren't available.
00:08:23
The Rails conventions we know today were not yet integrated into Rails. Users were accustomed to XML-RPC or SOAP styles for APIs, combining flavors among them. The original Rails codebase reflected this context, and you would find a `whizzle` file in your routes.
00:08:54
Additionally, at that time, Capistrano wasn't available—deployments did not have a robust path. Many things we consider standard today had to be figured out from the ground up, but they managed to establish working methods.
00:09:32
The first commit message reflects about four hundred files committed back in August 2005 using Rails 0.11.1 from March 27, 2005. One of the controllers from this commit was the Libraries controller, which still exists in our application today. As a result, it retains some recognizability as Ruby code, but it differs noticeably from how we would write a Rails controller today.
00:09:54
The application started as Phoenix, which later became Connect and most recently evolved into an application called Manage. Another application, dubbed P2 Services, essentially created a services layer.
00:10:24
Around 2006, Rails hit version 1.0 in December, and version 1.1 introduced rjs, enabling the writing of those magical Ruby scripts that generated JavaScript for Ajax and remote calls. Version 1.2 incorporated REST.
00:10:50
As Rails continued to advance towards version 2.0 in 2007 and 2.3 in 2009, the growth of the business alongside the code brought various frustrations. During this time, Heroku was founded, GitHub emerged, and Passenger improved deployment, which many Rails applications started adopting.
00:11:32
VitalSource found that the growth of these applications started outgrowing the development organization. A tension developed between building features and taking time to research business problems and answer customer questions. This environment became very business-driven and interrupt-driven.
00:12:02
Some developers presented during this time found it frustrating to complete work. Although developers had managers, their direct interactions were minimal. As applications grew, developers typically had areas of specialization, leading to silos.
00:12:37
As Phoenix developed, the database layer began in 2005, and this led to a shared approach among models and services. This sharing continued as T2 Services was established for the necessary data pooling.
00:13:13
With more applications being built, the focus shifted towards maintaining consistency, leading to the migrations being moved into their own application named Goose for central management of modifications. This resulted in a set of reporting applications capable of providing business data back, mainly Reporter and Uber Reporter.
00:13:59
And eventually, by 2010, P2 Services evolved to accommodate various versions reflecting business needs. Unfortunately, the suite of applications, along with others running on Rails, essentially remained locked to outdated versions.
00:14:31
The shared approach amongst the applications demanded a consistency that locked them into using Ruby 1.8.7 and Rails 2.3. Their intention was to maintain uniform machines to ensure operability. However, this restriction began to limit flexibility, especially as Rails upgraded to 3.0.
00:15:10
In 2010, Rails experienced a significant community merge with Merb, prompting major changes in Active Record and elevating Ruby to version 1.9.3. As the perceived performance detriment of Ruby 1.8.7 lingered, Ruby Enterprise Edition also started gaining traction.
00:15:48
By 2013, a new CTO stepped in, followed closely by a development director, leading to significant process and organizational changes.
00:16:20
The focus began shifting from rapidly shipping everything to a more structured and testable approach. Technical management started playing a crucial role, providing an umbrella that shielded development teams from constant requests.
00:16:55
This cultural change allowed for product managers to take input from the business on how to steer feature work, rather than relying on direct business requests.
00:17:30
As work continued, the teams were in consensus about the necessity to catch up with community standards. VitalSource's acquisition of its primary competitor in 2014 prompted extensive work to digest and integrate various applications, a transition that took about 18 months.
00:18:27
This period consisted of migrating users, evaluating functionality across platforms, and beginning work on a combined storefront with features that the newly acquired company had but VitalSource lacked. Throughout all this change, Rails and Ruby continued to evolve.
00:19:09
By the time we reached Rails 3, we were finally ready for the significant upgrade that had been a long time coming. By fall 2015, the integrating work with CoreSmart reached completion, which prompted the decision to upgrade our applications.
00:19:57
Led by our CTO, a focus on minimizing operational impact initiated this upgrade process. Gaining continued support for Ruby 1.8.7 and Rails 2.3 had become increasingly difficult, necessitating the move to updated frameworks.
00:20:40
The upgrade efforts occurred in parallel across applications, completing the necessary steps to transition from Rails 2.3 to 3.2 and on to newer versions. This stepwise upgrade approach took about seven months of meticulous testing and adjustments.
00:21:16
Despite being challenging, these upgrades posed numerous complications when mixing urgent work with required updates. Maintaining divergent codebases and ensuring their shippability needed careful synchronization.
00:21:50
Especially since this suite of applications shared a database, coordinating cross-application dependencies was vital to ensuring successful deployments.
00:22:27
Numerous functionalities in upgrades triggered the need for some hand-holding to ensure proper transition. For instance, transitioning away from RJS templates to native JavaScript required significant changes, as did the restructure of ActiveRecord queries in line with capabilities introduced in later versions.
00:23:15
Team structures started evolving too. What had been isolated teams focused on various back-end segments saw an influx of new hires and leadership adjusting to the larger development process.
00:23:53
The advancements in our API structure embraced JSON as a primary preference, supporting XML for external APIs. Aiming for more meaningful error handling proved an invaluable lesson from prior integrations.
00:24:36
As the development team began to adopt a review culture, the focus shifted to integrating logic into distinct classes and crafting cleaner controllers that handle business interactions more effectively.
00:25:15
This change emphasized treating the API as the central interaction point with the core database. Adopting more collaborative work practices began to lead towards the perspective that teams had ownership over when they upgraded Rails and Ruby.
00:26:05
The implementation of new development practices fueled UX refresh efforts, particularly since the core application hadn't changed significantly since 2005. We focused on increasing automated test coverage, aiming to prioritize test quality from the outset.
00:27:00
Transitioning from Scrum to Kanban approaches allowed each team to iterate their methods and processes based on their business interactions, smoothed by their retrospective insights.
00:27:43
As we mature, we recognize the need for some applications to retire gracefully. We plan to replace our early service layer with more up-to-date functionality that includes robust authentication and additional RESTful APIs.
00:28:33
Both reporting applications developed from this codebase are set to be replaced with a more comprehensive ETL-based reporting solution from another development team. The valuable insight we've gained allows us to let go of legacy applications without sacrificing value.
00:29:10
For development teams tackling older codebases, tools like Code Climate can provide complexity analysis to identify difficult areas. Code frequency graphs from GitHub reveal spikes in activity during significant shifts, informative for underlying changes.
00:30:03
Project history release records offered insight into how changes in dependencies could shape direction. Git logs become invaluable for digging into commit details and trends, providing a rich source to understand evolution across time.
00:30:55
Using Git as a time machine offers tremendous potential for developers wanting to understand historical decisions. Spinning up a project from a specific moment allows teams to contextualize why changes occurred.
00:31:38
Invaluable discussions with long-standing colleagues helped provide historical context for code structures and decision-making processes. I endeavored to maintain curiosity rather than an accusatory approach to these inquiries.
00:32:31
Lessons learned underscore that code should be judged based on the context and circumstances that shaped it during its development. There are often very good reasons for decisions made in the past, influenced by deadlines, knowledge gaps, and inherent limitations.
00:33:07
Getting to a Minimum Viable Product (MVP) was pivotal amidst extensive workloads. This shift encouraged external access while acknowledging that upgrades invariably consumed considerable time, stalling innovation.
00:34:23
We've identified clear opportunities to refactor business logic, particularly in avoiding the pitfalls of shared migrations that produce cross-application headaches. Despite challenges, coding in Rails continues to be a joy for many developers.
00:35:12
Reflecting on how different development paths may have materialized can provide insight into decisions made. To ensure clarity and intentionality, documentation and error handling must improve in guiding external interactions.
00:35:50
Ultimately, I carry forward the belief that my colleagues did their best work given their knowledge context and circumstances at the time. My approach is met with love, seeking to understand how perspectives evolve over project lifecycles.
00:36:35
The Rails application has proven successful over 12 years and maintains the potential for growth. Our upcoming UX facelift signals a commitment to evolving with user needs.
00:37:10
It's surprising that a code base remains stable and functional over such a time, showcasing the business’s faith in Rails when it wasn’t fully matured. Along with thanks to the staff and team members who contributed, I look forward to seeing how our code continues to develop as we face the future.
00:37:53
Thank you for being here. Your participation in this conference is invaluable. I appreciate the time you spent engaging with these ideas today. Enjoy the rest of your conference.