Services and Rails: The Shit They Don't Tell You

Talks

Services and Rails: The Shit They Don't Tell You

Brian Morton

@bmorton

#ruby-on-rails

#microservices

#service-oriented-architecture

#scalability

#cross-functional-teams

#organizational-skills

Services and Rails: The Shit They Don't Tell You

by Brian Morton

In the video titled "Services and Rails: The Shit They Don't Tell You" presented by Brian Morton at the wroc_love.rb 2013 conference, the discussion centers around the complexities of building services and integrating them into Rails applications, specifically drawing from the experiences at Yammer. Morton explains the tension between wanting to create smaller, more encapsulated Rails applications through services while navigating the added complexities that come with them. Key points emphasized throughout the talk include:

Service-Oriented Architecture: The importance of breaking down monolithic apps into manageable services to improve scalability and maintainability.
Incremental Transition: The necessity of gradual extraction of features and components to optimize performance and scalability without overwhelming developers.
Loose Coupling: Developing services that allow teams to work independently, reducing the complexity of large-scale deployments and enhancing collaborative efforts across different teams.
Organizational Changes: The adoption of cross-functional teams to mitigate the pitfalls of siloed knowledge and improve responsiveness to project needs.
Real-World Challenges: Sharing Yammer's experiences with difficulties in managing data integrity, performance during transitions, and the implications of using dual data storage strategies.

Morton illustrates these points with specific examples from Yammer's architecture, including their approach to search functionality and how they navigate technological swaps without service disruption. He discusses critical concepts like Conway's Law, emphasizing that organizational structure affects service development. Another significant takeaway is the acknowledgment of iterative learning from mistakes which ultimately leads to more robust systems. The talk concludes with an emphasis on the balance between service complexity and the benefits they provide, encouraging a mindset shift in approaching service architectures as applications grow and evolve. The audience is encouraged to engage with the challenges and potential solutions as they navigate similar paths in their software development processes.

00:00:18.480 Today, I'm going to be talking about how Yammer builds services and how we integrate them alongside Rails. I want to discuss some of the things that we don't talk about very often.

00:00:30.960 My name is Brian, and I work on the Rails team at Yammer. One of the things we do is take components out of our monolithic Rails app and put them into services. The Rails team assists the core services in this integration. On a personal note, I love Zelda, video games, and music. I'm really excited to be here; this is my first time in Europe. I gave my first conference talk just a couple of days ago, so I am a bit scared, but this is awesome!

00:00:56.760 I have a pet peeve about talk titles that don’t connect to the content. So, I want to ensure that I keep my talk relevant to the title. Additionally, I needed to find a suitable template for my presentation. Since Yammer is owned by Microsoft, there are tons of templates available. I came across one that mentioned how full bed photos can evoke emotions and make presentations more memorable, so I decided to use that idea.

00:01:10.240 Before continuing, I want to highlight that this talk might not apply to everyone here yet. If you're building a startup to determine viability, you should ignore much of this advice and focus on getting it done. Building services adds a layer of complexity, and you probably don't have enough information yet to start this process.

00:01:20.360 Once you determine that you need to scale, however, you will have to make some decisions that might make you uncomfortable. I will discuss those hard decisions in this talk. At Yammer, we have a large Rails app with over 300 models and 200 controllers. It is backed by 20+ JVM services that we have built, and some of these services handle over a billion requests a day.

00:01:40.000 Even with our powerful services, the Rails app remains quite large, which makes things challenging as we move forward. We have managed to address some issues by extracting components into services and tackling the problem incrementally. Still, things get painful when dealing with tasks like sharding or upgrading Rails and Ruby. These transitions are all-or-nothing situations.

00:02:10.000 In this talk, I will touch on service-oriented architectures, the reasons we build services, and the benefits they provide. Our primary goal is to create components that can scale individually. By focusing on small services, we gain versatility and ease of use.

00:02:32.960 To illustrate this, at Yammer, we developed our Rails app alongside specific services, including indexing services and a search interface. Based on previous search experiences, we recognized the need to handle these components in three chunks to ensure individual scalability and reusability.

00:02:55.600 We created a denormalized data store that mirrors data from Rails, allowing various surface layers to utilize it efficiently. This architecture facilitates the addition of features, such as autocomplete for our search interface, which can be seamlessly integrated by leveraging existing components. Our data export service, named Slurpee, allows us to pull entire chunks of data from this denormalized store and export them conveniently.

00:03:16.560 Having multiple independent components provides significant advantages, particularly in scalability. It’s much easier to scale a service since we can predict its usage and performance patterns. Moreover, we avoid wasting resources on scaling the entire system if those resources are not required.

00:03:45.679 Another pivotal benefit of our services is loose coupling. Smaller, focused services allow us to encapsulate concerns and push out updates independently without necessitating massive deployments of the entire Rails app. While we still have large deployments, the ability to move smaller pieces along independently proves beneficial.

00:04:06.280 Currently, we are transitioning our files backend. This has been a smooth process as we shift away from our Node.js service towards a JVM-based solution without creating chaos in our operations. This emphasis on adaptability allows us the flexibility needed to swap technologies whilst maintaining stability.

00:04:30.000 Organizational scalability is another goal of service-oriented architecture. When working on a monolithic application, developers need to understand the entire codebase, which often leads to confusion and mixed responsibilities. By breaking down the application into services, we allow teams to gain the knowledge they need on a need-to-know basis, fostering distributed execution.

00:04:55.360 This loose coupling enables our teams to collaborate effectively, allowing them to agree on API standards. We can utilize dummy endpoints while building services in parallel, allowing different teams to work simultaneously towards a cohesive product.

00:05:15.360 However, moving to a service-oriented architecture does not happen overnight. Starting your business or application with a monolithic approach can offer an easy way to build and modify things quickly. With a single codebase, there is little overhead, and you can share code easily without facing the complexities that arise with services.

00:05:33.760 While these lessons are learned iteratively, forming a service-oriented architecture requires a mindset shift. At Yammer, we often discuss Conway's Law, which posits that organizations tend to develop systems that reflect their communication structure. This means that if we want to create services, we need to align our development teams accordingly.

00:05:56.080 Many organizations divide their departments vertically or horizontally, leading to silos that inhibit collaboration. At Yammer, we initially structured our teams with separation and specialization, but we found that this siloed knowledge limited our efficiency. As we grew, we realized it was essential to reorganize.

00:06:16.320 We shifted our approach to create cross-functional teams. For each new service or feature developed, we assemble a diverse team with representatives from every aspect of the project, such as Rails engineers, core services engineers, and mobile client engineers. This decentralized process fosters an environment where teams are autonomous and well-informed.

00:06:35.000 These teams have complete discretion in their design and development processes, producing well-structured and reusable systems. Importantly, these teams are ephemeral—they come together, solve a problem, and then disband, with a different lead taking charge on subsequent projects.

00:06:57.120 Cross-functional teams allow us to leverage shared knowledge across various domains, ensuring that we can work on countless projects simultaneously. They coordinate effectively between client and service APIs, leading to the emergence of naturally fitting services.

00:07:16.560 That said, these teams also face challenges. Without siloed experts, there are trade-offs, including the need for team members to continuously adapt to new domains. When a team forms, they need to learn and familiarize themselves with the project area, which can be time-intensive.

00:07:41.919 Additionally, while building features, we must be cautious not to couple the API implementation too tightly with client needs. It's easy to focus on immediate feature demands instead of looking at the broader picture. After project completion, the support burden is shared amongst other teams, which may not be as familiar with the domain.

00:08:00.480 There are various ways we set up our structures and processes. Although we have changed our organizational setup recently, solving these complex problems remains an ongoing effort. Often, the simplest solution involves placing all services behind Rails, preventing clients from interacting with them directly to mitigate complications.

00:08:21.360 This allows our services to maintain separate data stores without requiring clients to navigate multiple layers. However, there are trade-offs involved in maintaining Rails resources to access these services.

00:08:43.360 In some scenarios, we allow direct client interactions with services, which can simplify things. For example, our Mugshot service dynamically resizes images for us, streamlining caching and image management. Nevertheless, as we architect these systems, we must also factor in authentication and security concerns.

00:09:02.280 As our application grows, challenges emerge with managing data across services. Reading from the database can be manageable, but writing can introduce complexity. The difficulty arises because our caching mechanisms and data management structures need to align seamlessly between Rails and the services.

00:09:22.520 Active record is our primary means of managing data, which allows us to build our app quickly. However, as we untangle our data structures, we encounter challenges in maintaining data integrity, connecting models, and ensuring consistent validations across our services.

00:09:42.400 To mitigate these issues, we often treat our services as indexes, storing IDs in our services and hydrating relationships through Rails. Another option is to shift ownership of the data to the service layer, which can improve management but also leads to data duplication as we go.

00:10:00.680 Once we've determined we want to move away from Active Record, it’s necessary to plan this transition carefully. Downtime is not an option for us, so we have to implement incremental changes without disrupting service availability.

00:10:21.960 One effective approach is double dispatching: we backfill all data to services while also writing to the database simultaneously. During this phase, we can monitor performance and adjust traffic flow gradually, which ensures that we can analyze capacity handling before fully transitioning.

00:10:42.000 However, the dual data storage leads to yet another challenge: who manages the cleanup? Developers can become confused seeing data replicated across systems, leading to uncertainty around which data is the source of truth. Regularly, developers need to address data conflicts and unnecessary duplication.

00:11:02.280 Adopting a backup plan is a comfort zone that can be problematic. While it allows developers to revert to previous structures if a service fails, it can also hinder progress. We need to be comfortable with stepping out of our comfort zone and tackling new challenges.

00:11:19.360 However, maintaining a backup plan has its costs, as it requires additional resources and can create overhead that might not be worth it in the long run. Developers must focus on developing and delivering value, ensuring they don’t revert to monolithic solutions as a knee-jerk response to complexity.

00:11:36.440 At Yammer, we discuss the importance of learning from our mistakes. We have rewritten our search stack three times, but each iteration has taught us invaluable lessons, leading to a robust and efficient stack today. Similar reflections apply to our code: we often look back at past decisions and reconsider if those were the best choices.

00:11:56.020 This re-evaluation process is essential; assumptions about prior decisions may no longer apply in the current development landscape. While we maintain a large Rails app, our focus is to continually move closer to a more scalable and manageable structure.

00:12:16.920 In closing, any questions?

00:12:32.919 So you mentioned that you work in teams; how long do those teams typically last?

00:12:38.360 Our teams generally consist of 2 to 10 people, and we operate on a 2-to-10-week cycle. Beyond ten weeks, projects tend to drag on unproductively.

00:12:50.320 What happens if the team encounters support issues after a project ends?

00:12:58.320 We have another cross-functional team dedicated to support engineering. Their role is to handle support issues, though it can be challenging as they might not be familiar with specific domain knowledge. However, they know where to go for answers.

00:13:12.440 If your application is well-factored and modularized, what remains the issue with it?

00:13:30.679 Even a well-structured application can present challenges, particularly in complexity when services are introduced. The nature of your application greatly influences this aspect.

00:13:46.799 How do you manage communication and documentation when various teams work on the project?

00:14:01.280 When a cross-functional team is formed, we prepare a text document to outline the changes. This documentation is continuously updated throughout the project. At its conclusion, we have a repository of insights detailing challenges and decisions made.

00:14:21.600 Do you centralize documentation for all services?

00:14:38.200 We use a combination of written documentation, much of which resides in Google Docs. While it isn’t formalized, it serves us well for maintaining notes on projects.

00:14:52.100 When deciding on code extraction for service development, do you follow a specific protocol?

00:15:05.680 Historically, we have not implemented strict protocols, but that doesn’t preclude us from adopting improved procedures in the future.

00:15:26.160 To conclude, do you use something like OpenAPI for API documentation?

00:15:43.280 Our documentation is fairly loose, primarily consisting of notes taken throughout the project. While it might not be formal, it allows us to maintain agility and adapt as we proceed.

00:15:59.999 Thank you all for participating!

wroc_love.rb 2013