00:00:21.520
Our next two speakers are John Wilkinson and Anthony Zacharakis, who are from the Desert Planet of Lumosity, or Lumos Labs.
00:00:26.640
John has been working with Ruby for about five years, and his favorite gem is Pry. Anthony has been working with Ruby for a bit less, only about two years, and he enjoys administrative tasks.
00:00:34.480
They are here to talk to you about doing service-oriented architecture (SOA) without bursting into tears.
00:00:57.600
Hi everybody, welcome to our talk! Thanks for coming. My name is Anthony, and this is John. We are both developers at Lumosity, a site that creates brain-training games to improve cognitive performance.
00:01:12.640
Today, we want to talk to you a little about managing the complexity of your Rails application using an architectural pattern we have found to be really effective—embedded engines.
00:01:44.720
Specifically, we want to cover our previous attempts at SOA architecture and why they didn’t work out for us. Then, we will explain what embedded engines are, how we use them, and the benefits we’ve gained from this pattern. Finally, we’ll discuss when it might make sense for you to start using it as well.
00:02:01.200
Let me give you a little context about Lumosity: we have about 15 developers, 45 million users, and an application that has been around since 2007. There has been a lot of thrashing on the code base since then; we’ve added a lot of features, removed many, and rewritten parts multiple times. Therefore, each line of code in our codebase carries a lot of historical baggage.
00:02:18.320
This complexity contributes to what you might consider a large application, which carries a lot of negative connotations. Large applications are often slow and can be cumbersome to work with because you are loading many gems or files.
00:02:29.440
For instance, our main test suite takes about an hour and seven minutes to run entirely on our CI server, utilizing 24 cores. You can imagine that running it locally would take even longer—it’s not feasible to run it after every change.
00:02:41.680
Consequently, when making changes, you aren’t able to run the tests, get immediate feedback, and iterate upon it; instead, you must push it out and come back later.
00:02:59.599
Another challenge with large apps is their complexity. You often end up with what we call "God models," which are usually centered around the user or some central concept of your app. These models can be extremely complicated and lengthy, resulting in a convoluted data flow that is hard to fully grasp.
00:03:36.000
For example, imagine a user signs up. The onboarding process starts with the signup page, handled by our authentication system, which is responsible for all the user registration and login processes. After signup, the user goes through a personalization survey to set up their basic training preferences.
00:04:06.640
This system works well until there’s a story that requires the user to skip this entire step. That’s when we discover that skipping this causes significant issues, as our training system depends on various values being set beforehand.
00:04:31.680
This implicit dependency was something we had never explicitly documented: the system assumed that users would sign up and go through the flow first. This realization shows how fragile large apps can be.
00:04:59.679
Even if your tests cover everything, modifying one feature can inadvertently break something unrelated, causing you to hunt down bugs due to changes in other areas of the code.
00:05:32.000
For those of you who have felt this frustration while working on a feature, you know exactly what I mean. We recognized this issue and began searching for a solution, with the typical answer being to adopt SOA.
00:05:50.000
SOA, or service-oriented architecture, essentially breaks down your application into separate services that communicate via a well-defined API. This approach has several pros, which is why we decided to implement it for a feature we believed was a perfect candidate.
00:06:18.800
Six months later, however, we found ourselves pulling our hair out! It turns out that breaking down a large app into SOA architecture is significantly more challenging than we anticipated. We encountered a lot of new bugs—new architectural frameworks often introduce unforeseen issues.
00:06:55.840
For example, we had to manage legacy code and dependencies when trying to extract functionality into a separate service, which required in-depth work. This kind of migration feels like doing surgery on your code and is particularly tough when institutional knowledge is embedded in the existing codebase.
00:07:15.120
Additionally, SOA introduces a lot of development friction. Suddenly, you are managing multiple repositories and dependencies instead of having everything housed under one app. Running multiple servers to get basic functionalities up and running can be burdensome.
00:07:35.200
Testing becomes more complicated as well. Although you can stub out API requests when testing, you now also have to keep track of updating everything, which adds yet another layer of complexity.
00:08:17.440
We took a step back and thought about whether we could try a different approach. Thus, we started utilizing what we call namespace lib modules, taking parts of our app that we considered self-contained features and placing them into their specific namespaces, defining what each part does.
00:08:50.880
This method did help clean up our code, but it didn’t encourage independence as we had hoped. In Ruby, there are no inherent constraints preventing touching another class, making it easy to reintroduce dependencies and defeat the purpose of separation.
00:09:10.400
Thus, we sought to explore a different concept called embedded engines, which I will let John speak about.
00:09:42.400
According to Rails, engines serve as a happy medium between more minimalist namespace lib modules and full SOA approaches, which are often more complicated yet robust. The greatest advantage we found with engines is their concept of isolation, allowing us to create new application controllers, helpers, and route sets.
00:10:15.360
For instance, consider an engine called user_auth for user authentication. User_auth is fully encapsulated, just like the main app. The primary distinction is that the main app mounts the user_auth engine at a specific route, channeling requests to the user_auth's route.
00:10:40.880
It feels similar to managing a separate server without actually setting one up. We can create this user_auth engine through a Rails command, and it behaves like a new Rails app where we can build out controllers and models before integrating it back into our main application.
00:11:06.080
This approach limits the overhead of managing separate repositories for features that only exist within one app. It becomes cumbersome to handle multiple pull requests for a single feature because you need to make requests both for the engine and the main app.
00:11:37.440
You must also manage bundling, which can result in having numerous gem versions on your local machine.
00:12:06.080
However, there’s a workaround. You can pass a path option in your gem specification within the Gemfile, which allows you to have an 'engines' folder in the root directory where the engine is housed. Pointing to the engine simplifies the transition when switching branches.
00:12:30.480
When planning to extract functionalities into an engine, keep in mind that any tables specific to that behavior should have their corresponding Active Record model moved into the engine. You may also prefix the table names with the engine name or specify alternative table names to avoid complicating data migration.
00:12:52.800
A critical aspect to consider is associations. Leaving an association spanning between your main app and your engine creates tight coupling that may become increasingly difficult to manage. Ideally, you should sever the association or create a new corresponding table with shared identifiers, enabling the easing of the original table.
00:13:12.160
Since you’re mounting the engine to a particular route, ensure that any actions or assets you move will incorporate this new route. This is important if you have existing links, as you will need to update them accordingly.
00:13:39.680
While the process of extracting functionalities can seem daunting, we have developed a set of techniques at Lumosity that help facilitate this process, which we will elaborate on through examples.
00:14:01.600
When configuring your engine, you will likely need to pass data from your main app that should be available in the engine, such as configuration options or feature flags for A/B testing.
00:14:31.680
These variables can be set up in an initializer in your main app that provides the necessary values while keeping everything explicit.
00:15:02.240
As an example, with a user authentication engine, you could create a configuration object to enable a Facebook button via a specific configuration passed through the initializer.
00:15:20.000
This allows your engine to call the configuration without needing to know about these dependencies explicitly. Additionally, we need to avoid tight coupling by establishing clear public behavior access from the engine.
00:15:46.400
This could involve setting up proper accessors in the engine that accept user data and handle internal behaviors internally, allowing the main application to access necessary functionality without entanglement.
00:16:13.760
The pure isolation afforded by engines enables enhanced testing methods, ensuring that tests can be run independently of the main app and focusing on specific functionalities.
00:16:35.040
Dummy apps can serve to expose hidden dependencies and provide faster feedback loops. Despite the engine's isolated environment, don’t omit integration tests for interactions across multiple engines.
00:17:00.000
When dealing with older monolithic applications that feel bloated, consider adopting embedded engines as soon as possible. The sooner you implement this structure, the more efficient and maintainable your app can become.
00:17:30.000
But for smaller or newly developed apps, consider waiting until there’s more established code before implementing engines. It’s also beneficial for small teams working in Agile environments to manage specific features collaboratively.
00:17:50.000
Even large teams benefit from this framework as smaller sub-teams can focus on individual engines, reducing the complexity of communication across features. It allows greater autonomy in development.
00:18:19.000
Our approach proved so successful that we adopted it fully, resulting in the establishment of over a dozen embedded engines in our app.
00:18:51.360
We encourage you all to consider using embedded engines as well. They allow for quick specs, rapid feedback, and peace of mind while developing features.
00:19:06.720
This structure fosters better understanding for both existing team members and newcomers to the project. Thank you very much for attending our talk!
