Build and maintain large Ruby applications

Talks

Enrico Teotti

@agenteo

#ruby

#modular-design

#dependency-management

#automated-testing

#domain-driven-design

#software-architecture

#legacy-applications

Build and maintain large Ruby applications

by Enrico Teotti

In the presentation "Build and Maintain Large Ruby Applications" at RubyConf AU 2016, Enrico Teotti discusses the complexities and challenges of developing substantial Ruby web applications. He emphasizes that there is no single tool or silver bullet for success; rather, effective application development results from a combination of team diligence, the use of local Ruby gems, and automated testing. Teotti illustrates his points with insight from his decade-long experience in Ruby development, focusing on managing complexity within applications that evolve over time.

Key points from the presentation include:

- The notion that while Ruby can be challenging for large applications, with the right practices, it is possible to successfully build and maintain them.

- The importance of clear organization within Ruby projects, comparing Ruby files to ingredients in a recipe, which should be easy to identify for both current and future developers.

- The case of Kadeem, an Italian street snack, serves as a metaphor for clarity in the development process. Teotti discusses how poor organization (grouping classes by design pattern) can hinder navigation and understanding within larger applications.

- He advocates for the use of namespacing to maintain clear boundaries and reduce confusion among different components within an application.
- Local Ruby gems are positioned as valuable building blocks that enhance organization and clarity, reducing cognitive overload and improving developer onboarding.
- Teotti shares his experience of a real application where a lack of proper dependencies and organization led to confusion and fear among developers, ultimately resulting in changes to improve clarity.
- He suggests implementing a structured dependency flow to prevent circular dependencies and to allow teams to operate more efficiently as they scale.
- The talk concludes with the significance of automated testing and a culture of continuous improvement, highlighting the necessity of adopting local gems to foster productivity and learning.

This presentation ultimately stresses that while Ruby has limitations, strong organizational structures, thoughtful dependency management, and a dedicated team can result in effective and maintainable large applications.

00:00:00.120 Welcome everyone! My name is Enrico. You can get in touch with me on my Twitter account.

00:00:04.319 That's my blog. Thanks for the introduction! I don’t know if I am that absolutely technical, but you can read my blog, and the slides are already on my /r account, so you can look them up there. I'm a Principal Software Engineer for Pivotal Labs, a software consultancy based in Boulder, Colorado.

00:00:19.500 Let me start by saying that there is no special tool or silver bullet that will allow you to build and maintain large Ruby applications. So, I'm not going to present a gem that fixes all your problems. This discussion is not about applications with large loads but rather about those containing large business rules and complex domains that evolve as the application development continues.

00:00:37.350 You might have heard that Ruby shouldn't be used for large applications. In my experience, large Ruby applications do pose challenges, but they are possible with a combination of three factors: team diligence, the use of local Ruby gems, and automated testing. However, relying solely on any of these elements was not sufficient. Local gems alone were not enough, nor was team diligence or automated testing alone. The goal is to create an application you want to work on, one that you can understand from day one rather than feeling overwhelmed by cognitive overload.

00:01:10.290 Now, if you are wondering where my accent is from, I’m Italian. I learned to speak English by watching Family Guy. I lived in Australia in Sydney for about six years and now in the United States, so my accent will fluctuate among those regions. During the talk, consider Ruby files in a project as ingredients in a recipe. Think about the project that you're working on right now. Can you tell what recipe it’s cooking? Would you, as a developer, be able to do that? Now imagine your project six months from now: would you still be able to tell what recipe is cooking? Would a new developer be able to do that?

00:01:58.549 Let’s take an example: those are the ingredients for Kadeem. Kadeem is a popular street snack in Italy, priced between three to five euros, but in Sydney, Australia, it can only be found in a small cafe run by two young Italians near Bondi Beach, priced around 10 to 12 euros. One simple dish, one simple recipe can be extremely profitable. This isn't a real fact, so I’m going to make up names: let’s call the two owners Mario and Luigi, with Mario from Brescia, a province in Italy that is not particularly famous for Kadeem. But it's the only place in Sydney that serves it. When my friends and I wanted to have it, we had to go there. However, every time we went there, we never saw Mario and Luigi making Kadeems.

00:02:53.160 They had a couple of students, Stevo and Anna, who had no cooking experience but learned the recipe and familiarized themselves with the ingredients and started making the profitable Kadeem for Mario and Luigi. Everything was working out fine until three months later when I spoke with Mario, and Anna went on a gap year, so they had to ask someone else, Diego, another student with no cooking experience. As he looked at the ingredients, he had no idea what was going on and needed stable help. That reflects some intrinsic difficulty in learning something new. Maria was annoyed and muttering and shouting at him in Italian. So he said, 'Diego, salento his man tirra cycle cosa,' which is quite common in stressful situations. After a few days, everything seemed to go back to normal until six months later when Mario and Luigi realized selling Kadeem wasn’t as profitable as it once was, and they started bringing new recipes onto the menu: pizza, tiramisu, carrot cake, and new ingredients.

00:04:31.350 Now, would you be able to tell the recipes for the new ingredients? You might easily find one for carrot cake, but you could be struggling to identify the rest. Diego's reaction to this was a mix of confusion and frustration. He wasn't just affected by the intrinsic difficulty of learning a new recipe; he was also burdened by extraneous difficulties surrounding the kitchen worktop. Being a novice, he might think there’s something wrong with him since to Mario and Luigi, it’s clear there are four recipes on the kitchen worktop. Diego starts to wonder if perhaps the kitchen worktop is not organized well, so he speaks with Mario, suggesting they organize things. Mario replied, 'Everyone in Italy groups ingredients by color.' So, the following ingredients were grouped by color.

00:05:53.420 This isn't helping at all when Mario insists Stevo thinks which ingredients are for pizza, which are for tiramisu. When he complains to Mario, he replies, 'This is the convention in Italy.' Have you ever seen something similar happening in a Ruby application? Classes grouped by design patterns might be okay in a small app, but in a larger application, they can hinder your ability to navigate through the code. Classes grouped this way describe a trait about that class but say nothing about its operational context.

00:06:57.300 Eventually, Stevo, with the help of Mario, divided the kitchen worktop into four areas for the four recipes. There are some shared ingredients, but it’s clear which ones they are, and this division helps avoid confusion. Everything was finally running smoothly until one day when Steve was preparing Kadeem, and Diego, sitting next to him preparing pizza, was rushing because he wanted to go to Bondi to play volleyball with friends. As he reached for the milk jar, it slipped out of his hands and cracked open on the kitchen worktop, ruining the pizza dough. Mario exclaimed, 'Mahna Mahna!' which translates to 'Oh no, oh no.' This situation is similar to using namespaces in Ruby. They provide a simple form of separation for your application and create small boundaries.

00:08:19.840 Classes defined within a namespace are specific to that context and force all their external clients to use the complete namespace to access them. In this example, the submission class defined within promotions depends on a class defined within membership, which represents a dependency you will only discover if you read the entire codebase. This may be manageable in a smaller application, but how does it work in a larger app?

00:10:00.000 I worked on an application for about three years in a team of five. We started with blank spaces. We divided the application into vertical slices and created silos so developers could work in parallel, initially without issues. However, after about one year, we realized we had created a tangled mess of cross-namespace dependencies. These silos were leaking into each other, making it difficult to onboard new developers. At the center of those tangled files, any changes caused unforeseen side effects, which were met with plenty of laughter from our peers. This resulted in an application nobody dared to touch, despite the desire to fix the issue. We struggled to commit to a team-wide approach due to some developers claiming that this was the Ruby way.

00:12:14.290 I don't believe that's true. Ruby provides us the tools to build maintainable applications. Stevo, Mario, Luigi, and Diego finally agreed to separate each recipe onto its own kitchen worktop. Now, any leakage or issues will be contained within that space, creating a shared area for the shared ingredients.

00:13:12.520 What prevents the shared workspace from becoming a cluttered mess? It is team diligence, which is similar to utilizing local Ruby gems. You might be familiar with RubyGems, the tool to pull and push open-source libraries. However, it's under the misconception that that's all gems can do. You can have gems local to your repository, local to your application. Use those as building blocks for a solid dependency structure for your app, and it reveals the intention behind your dependency structure.

00:14:19.600 Every dependency starts from one small gem that contains the core functionality of your first feature. As you add new functionality, you can extract functionalities into separate gems. Each gem has a specification file that acts as a map for navigating through your large application. This approach reduces cognitive overload, helping you think of your application in smaller, more manageable parts—like deciding if tiramisu and carrot cake need separate gems or one.

00:15:51.720 As features grow—like Mario requesting a calzone, a folded pizza—you can extract the pizza dough and share it with the calzone so that each feature maintains intentional dependency structures that map your application boundaries clearly. It’s crucial that these structures don’t become purely technical as they may hinder communication. A general rule I follow is to have between two to three concepts within each gem. If I exceed that, I discuss with my product owner whether that signifies a different context or team.

00:17:19.790 The dependencies should always flow from top to bottom—your main application depends on those gems rather than the reverse to avoid circular dependencies. For the next few minutes, we will delve into the technical details of building a local dependency structure. If you lose track at any point, please catch me after the talk, and I’ll clarify. The main Ruby application can be any framework; its primary purpose is to act as a delivery mechanism for your business logic encapsulated within the gems.

00:19:00.410 In this application we worked on, we had multiple files, each responsible for a certain task, allowing each class to do its job effectively. For example, the main application served merely as a connector, aggregating classes while connecting the health plan gem to a drug information gem. The main application only calls health and requires that gem.

00:20:23.020 The Gemfile contains a path option indicating that these gems are local to your repository. This allows your main application to worry solely about its entry point, with transient dependencies automatically loaded from that directory.

00:21:24.830 These local gems could have meaningful names for your team. I often refer to them as 'components.' You can create a gem with 'bundle' command, and it builds a repository that should be removed, since the gem will reside within your application’s repository. Once the health gem is worked out, it requires a specification that maintains dependencies, ensuring all dependencies have a spec lock to prevent multiple gems from installing different versions.

00:22:13.320 Next, you can write your specs to ensure functionality. For example, a sample spec confirming if the detail method returns a hash will now pass because the aggregation structure is properly defined. The main Ruby app connects to the first gem. Let's connect this first gem to the other dependency,

00:23:03.790 creating a seamless flow of information being fetched from services, ensuring all data is organized well. Attention to detail at this level allows for better feature assignment and expansion without redundancy. Finally, as teams grow larger—more members means more process boundaries to maintain—you may consider migrating slices of your monolith into services, waiting until the last responsible moment. The saga of this migration deserves its own talk.

00:24:17.790 By utilizing local gems, we could deploy parts of a monolithic application with ease. This application began with a team of four, featuring two sections—admin and public portions. Three months into development, we learned that the admin portion must go live within a company VPN, prompting suggestions to use services.

00:25:02.090 Instead of complicating matters with a service architecture, we decided to divide boundaries within the application. Shared behavior existed between the editorial and public portions, routing through a single persistent layer connecting to the database with a clear interface.

00:25:08.590 We encapsulated the behavior of those components with the overlapping dependencies, maintaining a solid structure that would track all changes efficiently.

00:25:18.350 Another intriguing aspect of our project included planning for legacy migration. We encapsulated all logic related to extracting legacy content from a cumbersome legacy system into a gem. When we were finally ready to go live, we simply removed that gem, leaving a clean transition.

00:25:55.430 Within a Rails application, those high-level entry points can serve as engines that plug-in routes and controllers. Your application might resemble Kadeem today, but how likely is it to remain that way? Instead of piling all ingredients into one kitchen worktop, consider separate contexts as your application evolves.

00:26:21.840 When returning to your work, think about local gems and be aware of fixed mindsets in your team. Fixed mindset implies abilities are stagnant, either you have the skill or you don't. In such teams, they may perceive implementing local jobs as a hack or underestimate its value. Hence, they may reject adopting new methodologies.

00:27:10.890 When introducing local gems to a fixed mindset team, consider running small experiments within your application, focusing on showing how adoption of local gems enhances productivity. This strategy empowers continuous improvement amongst your team, fostering a culture of learning over stagnation.

00:27:38.950 Embrace the 'power of yet'—the potential for growth and learning. After implementing local gems and validating their effectiveness with metrics, you can motivate the rest of the team to engage in similar practices.

00:28:14.250 Please don't overlook the need for automated testing—specifically unit tests—essential for ensuring the stability of your dependency structure. I firmly believe that the strategies discussed here will empower you to build and maintain large Ruby applications.

00:28:38.000 Thank you for your attention.

00:28:39.000 So, do we have any questions?

00:28:46.000 Thank you, Enrico. I'm curious about how you manage separation of models and services.

00:28:52.000 Yes, about the separation of models and services, there are patterns to follow within the dependency structure that best align with object-oriented design principles.

00:29:02.000 Models should be encapsulated properly, and at times we've successfully used ActiveRecord, while other times, we had to be more innovative. Importantly, high-level entry points should not be aware of databases but should maintain a thin service layer.

00:29:15.000 Alright, thank you! It’s probably time we move on to the next speaker.

00:29:27.000 Thank you!

RubyConf AU 2016