00:00:15.000
Hey everybody, it looks like it's about time to start, so I'm going to go ahead and begin. I figured since I was in Hawaii, I'd dress for the occasion with a pretty shirt. Hawaii is really beautiful, as you may know.
00:00:22.400
Anyway, let's get this started. I'm going to be talking about facing the monolith. How many of you have built or worked on a monolithic application of some kind? Yeah, just about everybody. You've been there, so you know that pain, right? Well, that's what I'm going to discuss and how we solved it on one of the applications that I've worked on.
00:00:33.320
To start us off, I'm going to introduce myself. I'm Charles Max Wood, the lead developer at Intentional Excellence Productions, which is my own consulting company. If you want to hire us to build something for you, just talk to me, and we'll make that happen. I'm also the host of the Ruby Rogues podcast. I see James over there, who is also on the show. We have a good time every week discussing a variety of interesting topics. If you haven't listened to that, you can find it at rubyrogues.com.
00:01:03.840
I also have a JavaScript show at javascriptjab.com and a freelancing show at rubyfreelancers.com, so if you're interested in those, feel free to check them out. I'm not going to dwell too much on what the application did, but I will talk about some of the components without getting too deep into the details. I mention this to illustrate that if you've worked on a monolithic application, you sometimes do wind up neck-deep in the complexities.
00:01:28.840
Let's talk about this for a minute. When you're starting to build an application, the mindset is often, "We just want something that does this little thing over here," right? You end up with something that's relatively simple and manageable. Your boss tells you to tame this thing, which, if you've ever tried to tame a cat, you'll know it's a challenge. The goal is to make it do something it didn't do before and solve a specific problem.
00:02:00.000
In our case, we were building a lead generation platform, and our directive was to create a system that could organize the leads. Sounds pretty simple, right? But as things evolve, demands grow, and suddenly the requirements change. Now, we need it to do something else. Just like cats don't bark, our application suddenly required that capability. This led to genetic manipulation of our application, making it do things it wasn't originally intended to do.
00:02:42.440
As a result, we ended up with a huge, unwieldy system, and fostering another feature became a Herculean task. Your boss keeps insisting, "Hey, tame this thing and make it work for us," which leads to keeping track of all its different heads and hoping it won't take your head off with its claws. It became a major pain in the neck.
00:03:03.360
Initially, we had an application that was essentially a bucket. But instead of just a bucket, it was a file cabinet, where we needed to organize leads. Databases can manage this organization reasonably well. You build tables and make them look nice, but then we realized we needed a system to collect the leads too. This might require an external API of some kind, which is relatively simple.
00:03:50.040
Then, we recognized that we need to make money by selling these leads. This required a mechanism to extract leads and send them to those willing to pay for them. Additionally, we needed websites for people to enter their information to generate leads. For SEO purposes, having some kind of CMS would also be helpful to draw people in.
00:04:29.480
People would come in, learn about majors they could pursue, and find the correct lead form to enter their information. Then we realized we needed user administration because not just anyone should access these leads. Reporting was crucial to understand how much money we were generating, and we verified leads through a call center, necessitating a call center app to verify the leads' quality. Eventually, we ended up with a convoluted mess.
00:05:15.080
The lead collection workflow became this elaborate process, and your boss would add new features that required you to teach the application new tricks without losing your head. Each time you added a new feature, it felt like shaving a yak—doing tedious work just to get what should be a straightforward feature implemented.
00:05:55.720
As I started writing this talk, I realized that a lot of the complications could be resolved with cleaner, more organized code. However, given the numerous systems we had working in tandem, it made sense to split the application into a service-oriented architecture (SOA). This approach facilitated better management as our monolithic application grew so large that it could tear the server apart.
00:06:34.840
Before we adapted SOA, we faced issues with scaling because the application wouldn’t just hang; it would completely stop functioning under heavy loads. SOA made it possible to scale horizontally and vertically by adjusting resources accordingly. I'll illustrate some differences between when you can just improve your code and when you need to rethink your architecture into a more manageable setup.
00:07:20.000
Managing a large, complex project is challenging due to the various elements working together. These systems are tightly coupled, which can be overwhelming as you try to keep all your concerns in order. Break them down into smaller issues makes it more manageable.
00:07:47.600
So we broke down the application into smaller standalone systems, which allowed us to focus on individual components. Everything above the line is customer-facing, while everything below consists of services primarily focused on data management. This de-coupling allowed us to streamline communication across services.
00:09:25.560
We had to address infrastructure issues such as how different applications communicate. Some processes need to be synchronous, meaning you require immediate responses while others can be asynchronous. For instance, when writing data, the user input is validated on the front end, and then dropped into the lead store asynchronously.
00:10:18.875
This included maintaining a job queue system which allows processing without needing to confirm each step immediately. If a failure occurs, the callback system can help identify jobs that need to be re-inserted into the queue, allowing redundancy. It's important to ensure that any asynchronous processing is regularly monitored to prevent bottlenecks.
00:11:08.520
Here’s a brief look at how HTTP connections were handled within our architecture. We called our lead service, which would return JSON data. This was done in a standard Rails format. Once the service returned, the necessary actions could be taken according to the business logic.
00:12:06.860
If a synchronous approach was necessary for reporting, we could make direct HTTP requests to those services. This ensured that timely updates were made without causing delays in the queue.
00:12:58.320
To summarize the organization, we had multiple web services and a communication structure between back-end and front-end services, which controlled how data flowed through our system. This was vital for authenticating users and managing access.
00:13:50.000
We initially used BeanoQue for queue management, but as our setup matured, we transitioned to using Rescue due to its user-friendly management capabilities. The lead service would process requests and send data to schools and other entities when necessary.
00:14:23.680
The queuing system would also handle writing new leads asynchronously while maintaining fast responses from our back-end services.
00:14:57.680
The fine-tuning of the queuing service is necessary to monitor its performance and ensure tasks are executed efficiently without bottlenecks.
00:16:21.600
When I originally prepared this talk, I wanted to touch on an idea proposed by the Ruby Rogues podcast regarding handling feeds. The concept is to utilize feeds to manage data flow and state consistency but ensuring that the same operations yield the same results.
00:17:07.799
In implementing front-end services, we developed interactive applications that interacted seamlessly with back-end services, creating a robust system for data management and user interactions.
00:18:18.880
We also built a service for user authentication and authorization, emphasizing best practices for how we manage and communicate authentication requests across our system.
00:19:13.400
Through organizing our services and ensuring each service only returned necessary data, we improved efficiency, reduced payload sizes, and enhanced our overall system architecture. The separation of concerns became vital in managing our complexity.
00:20:43.560
As for naming our services, we opted for a domain-based approach, making navigation and management more intuitive.
00:21:32.160
Security is crucial when managing service communication. Implementing SSL for encrypting communications, server registration, and utilizing API keys when necessary, can increase the robustness of your application.
00:22:25.320
If you're building JSON APIs, several considerations come into play regarding security and data management. I'll share further resources for deep-diving into these API-related topics.
00:23:53.680
The transition to a service architecture wasn't without its challenges, especially when working with integrations. There's a different set of skills and practices required when managing multiple addresses and ensuring smooth operations across everything.
00:24:11.640
If users encounter problems with communication or security breaches, we need rapid response protocols. This includes being proactive in monitoring services and having a solid deployment strategy.
00:25:46.320
I appreciate your attention throughout this talk. If you have any questions, feel free to take the opportunity to ask as I will be around.
00:26:48.159
As for branding our applications, we often considered public key encryption, user authentication methods, and strategies for token exchanges to ensure security.
00:27:43.480
From our experience, it's important to evaluate if multiple applications will share certain services and how to properly namespace those services when necessary.
00:29:37.640
Testing integration points, when multiple applications interact, can sometimes be challenging. Our approach involved setting up environments simulating those external calls to ensure everything worked seamlessly.
00:31:43.360
The setup process can indeed be cumbersome, primarily when trying to ensure that all services operate efficiently with each other.
00:34:49.480
Before we wrap up, please do connect with me via Ruby Rogues or on Twitter @Cmaxw if you need further insights or support regarding service-oriented architectures.
00:36:16.680
Thank you all for your time, and let's keep the dialogue going. If anyone wishes to discuss anything in depth, I'll be near the exit.
