00:00:21.520
Our next two speakers are John Wilkinson and Anthony Zacharakis, who are from the desert planet of Lumosity, or Lumos Labs. John's been doing Ruby for about five years, and his favorite gem is Pry. Anthony has been working with Ruby for about two years, and he enjoys administrative tasks. They are here to talk to you about doing SOA without bursting into tears.
00:00:57.520
All right, hi everybody. Welcome to our talk! Thanks for coming. My name is Anthony, and this is John. We’re both developers at Lumosity, a site that creates brain training games to improve your cognitive performance. Today, we would like to talk to you a little bit about managing the complexity of your Rails application using an architectural pattern we found to be really effective: embedded engines.
00:01:19.360
What we want to cover today includes a look at our previous attempts at implementing this SOA architecture and why they didn't work out for us. We will then discuss what embedded engines are and how we use them, as well as the benefits we have gained from employing this pattern. Finally, we’ll cover when it might make sense for you to start using embedded engines as well.
00:01:46.079
To give you a little context about Lumosity, we have about 15 developers and 45 million users. Our application has been around since 2007, which means there has been a lot of thrashing in the codebase over the years. We have added numerous features, removed many, and rewritten parts of the application multiple times. Consequently, there is a lot of history embedded in each line of code, making our application quite large.
00:02:10.560
The term 'large application' often carries negative connotations. For example, large applications can be slow. You might find yourself loading numerous gems or files, which can make running a console or server take a long time. This inefficiency can significantly hinder your workflow and make development feel like a drag.
00:02:44.400
Here's an example: our main test suite takes about an hour and seven minutes to run on our CI server, which has 24 cores. Imagine how much longer it would take to run it locally! This makes it infeasible to run tests every time you make a change, resulting in a workflow where you can't get instant feedback and iterate promptly.
00:03:10.720
Another issue with large applications is their inherent complexity. You often end up with many 'God models,' usually models centered around the user or some central concept of your app. These models can be quite complicated and long, interacting with numerous other models, which creates a complex data flow that can be difficult to understand.
00:03:37.440
To illustrate this, when a user signs up, they begin with the signup page handled by our authentication system. This page manages everything related to registering users and logging them in. Following signup, the user proceeds to a personalization survey that sets up their basic training preferences for later training sessions.
00:04:08.000
This process initially works well; however, we encountered an issue when a story required the user to skip the entire signup step. This resulted in many things breaking because our training system depended on having these essential values set up. The dependency on prior steps wasn't something we explicitly documented, leading to implicit assumptions in our app design.
00:04:42.880
This fragility is a common trait of large applications. You might develop a feature only to find that you've inadvertently broken something entirely unrelated due to some subtle interaction in the codebase. Hunting down these bugs can be frustrating as you ask yourself, 'Why did this break? I didn’t even touch that part of the code.' It complicates the development of new features.
00:05:08.720
Recognizing these challenges, we explored potential solutions, leading us to consider Service-Oriented Architecture (SOA). For those unfamiliar, SOA involves breaking down your application into separate services that communicate through a well-defined API. This separation allows for explicit management of the application's dependencies, theoretically providing numerous advantages.
00:05:28.960
To test SOA, we thought a feature we were working on would be a perfect candidate for this architectural shift. However, six months later, we were overwhelmed; the implementation proved significantly more challenging than anticipated. We faced new bugs as we restructured our application, and we had to address concerns about API requests timing out, which were previously non-issues.
00:06:03.760
Transitioning to SOA also forced us to manage legacy code, which felt like surgery on our heavily intertwined application. Extracting services with interdependencies is complex and laborious, especially when factoring in institutional knowledge embedded in the code. Additionally, the separate repositories and dependencies required when using SOA introduced development friction, as managing multiple servers and services became necessary just for basic functionality.
00:06:44.560
Moreover, with SOA, testing becomes more challenging. While you can stub out API requests, keeping everything up to date requires ongoing attention, adding to your workload. We found ourselves grappling with these issues, not just from a development side but also from architectural concerns, such as security while using APIs over public channels.
00:07:21.240
After some reflection, we decided to pivot from SOA to a different approach, leveraging what we call namespace lib modules. This involved grouping self-contained features within their respective namespaces and library folders. This provided clarity and organization but fell short in encouraging the independence we sought due to Ruby's flexible nature, which can lead to unwanted interdependencies.
00:08:10.440
At this point, we began using embedded engines, a concept I’ll let John elaborate on. Embedded engines provide a balance between the minimalist namespace lib modules and a full-fledged SOA.
00:08:51.760
The core benefit of embedded engines is their isolation. Isolation, in this context, doesn’t mean that the engine can't access the main app's internals but rather that it offers a new application controller, application helper, and a distinct set of routes. So, for instance, if we take user authentication as a subject, we can create a new embedded engine called UserAuth.
00:09:32.800
UserAuth functions as a fully encapsulated MVC stack, similar to the main app, handling web requests that the main app forwards to the UserAuth engine at a specific route. This delegation gives the impression of a separate server without the associated overhead.
00:10:08.560
We run the Rails command to create the new UserAuth engine, which generates all the necessary boilerplate. Working on it feels like developing a new Rails app; we add controllers, models, and fine-tune it before integrating it back into the main application.
00:10:50.560
When considering whether to create a separate repository for an engine, it’s essential to recognize this can lead to complexity. You’ll have multiple pull requests for a single feature since changes in the engine must correspond with updates in the main app, resulting in cumbersome management.
00:11:15.520
To simplify the process, consider using a gem specification's path option in your Gemfile. This allows you to keep your engine in a folder, so when you develop, you can point to the engine without worrying about versioning issues. It will automatically switch branches and maintain version consistency.
00:11:50.480
As you plan to extract your engine, keep some considerations in mind. Any tables related to the behavior you want to extract should have their ActiveRecord models moved into the engine. It’s common to prefix table names with the engine's name, but flexibility exists in specifying names to avoid production data migration.
00:12:25.760
When dealing with associations across your main app and engine, avoid tight coupling as it complicates future modifications. Ideally, sever these associations, but if that’s impractical, create a corresponding table in the engine, allowing for cleaner separation.
00:12:46.720
While mounting the engine under a specific route, ensure that any actions or assets reflect this structure. Stay vigilant about any existing links in your app or elsewhere that will require updating to align with the new routes.
00:13:07.760
Our experience at Lumosity revealed many techniques for implementing embedded engines effectively, which we aim to share through examples in this talk.
00:13:35.360
Let's discuss engine-specific configurations. In your engine, it’s important to pass data from the main app that the engine relies on, such as configuration options for supported languages or currencies. You might also need to integrate feature flags or A/B testing mechanisms that should reside in your main app.
00:14:14.960
To manage this, we create an initializer in the main app dedicated to the engine, providing all necessary values and making it explicit what dependencies exist.
00:14:55.360
As an example, let's say we want to include a Facebook login button in our user authentication feature. In the initializer, we might create a configuration object with a Facebook enabled key, coupled with a block that executes specific code from the main app. This allows our engine to control whether it displays the Facebook option without being tightly coupled to the main app's features.
00:15:30.720
Dependency management is another strength of embedded engines. Aim to limit how much your engine relies on the main app's internal behavior to avoid complications when modifications are made in the main app that could cascade into the engine.
00:15:47.760
For instance, consider a mailing class in your main app responsible for sending user confirmation emails. This might require integrating behaviors from the engine while minimizing dependency. To achieve this, you might implement a callback in the engine to provide necessary functionality while keeping a loose coupling.
00:16:21.520
Now you can tap into engine behaviors without modifying the main app directly, which protects against breaking changes and keeps the engine's implementation internal. This decoupled structure allows your main app to remain resilient and adaptable over time.
00:16:51.520
When considering testing, it's advantageous to isolate engines from your main app, simplifying your tests significantly. Unload unnecessary components while focusing directly on the engine features you’re targeting.
00:17:24.360
The dummy app plays a critical role here, functioning as a shell app that allows you to load only what’s necessary for testing the engine. This enables you to expose hidden dependencies while working on specific features in isolation. For example, removing elements that exist only in the main app ensures that test environments remain lightweight.
00:18:02.040
Using a dummy app leads to faster feedback loops; changes can be made and tested rapidly, offering real-time insights. As a direct comparison, running our engine test suites takes approximately 26 seconds, while our main application tests can take over an hour.
00:18:47.840
Even with embedded engines offering isolated tests, remember the importance of integration tests within your main application. They can ensure broader workflows function smoothly across different engines, catching potential issues that may arise from interactive features.
00:19:14.560
If you're considering whether to use embedded engines, starting with a large, monolithic application that has grown bloated is an excellent opportunity to reassess your architectural decisions. The sooner you integrate systems into more modular structures, the easier future developments will be.
00:19:50.720
In contrast, for smaller new applications, decide wisely based on your level of domain knowledge. If you’re acquainted with potential structures and possible groupings, dive into using embedded engines immediately to facilitate clarity.
00:20:15.440
However, if you're unsure about the eventual architecture, wait until your application has matured. You can still reassess later, adapting to insights gained during this development phase.
00:20:59.600
For small teams, especially in Scrum environments, using engines enables your team to swarm on specific features during sprints. This ensures confidence that changes will not disrupt other features, resulting in better estimates and fewer missed deadlines.
00:21:31.200
Similarly, larger teams can benefit by breaking into smaller groups focusing on different engines, reducing inter-departmental complexity.
00:21:46.560
This strategy proved successful for us at Lumosity; shortly after adopting this approach, we organized major application sections into separate engines, amassing over a dozen embedded engines in total.
00:22:26.440
We encourage you to consider this approach, as adopting embedded engines may lead to faster testing cycles, increased clarity in code structure, and significantly easier understanding for both existing and new team members.
00:23:00.000
We’d like to thank you for attending our talk.
