Facing the Monolith Overcoming Monolithic Applications with SOA

Talks

Charles Max Wood

4 talks

#monolith

#service-oriented-architecture-soa

#ruby-on-rails

#api-development

#code-quality

#software-architecture

Facing the Monolith Overcoming Monolithic Applications with SOA

by Charles Max Wood

In the video titled "Facing the Monolith: Overcoming Monolithic Applications with SOA," Charles Max Wood discusses the challenges and solutions related to monolithic applications, particularly as they grow in complexity. He describes his experience dealing with a lead generation platform that evolved from a simple system into an unwieldy monolith due to changing requirements and feature demands.

Key points from the talk include:

- Introduction to Monoliths: Wood introduces the concept of monolithic applications, highlighting the common experiences developers face regarding complexity and feature expansion.

- Evolving Requirements: Initially designed to manage leads, the application quickly escalated in complexity, requiring additional features that transformed simple processes into convoluted systems.

- Service-Oriented Architecture (SOA): To address these issues, Wood advocates for refactoring into a service-oriented architecture. He explains that breaking the monolith into smaller, independent services simplifies management and enhances scalability.

- De-coupling and Service Management: Wood emphasizes the importance of separating concerns in applications, allowing for easier communication and management of data flow. This also facilitates horizontal and vertical scaling to deal with varying loads.

- Asynchronous Processing: The speaker notes the necessity for asynchronous processing in some of the application’s workflows, particularly for tasks like data validation and user actions. Queue management systems like BeanoQue and later, Rescue, are mentioned as vital for monitoring and executing jobs efficiently.

- Security and Authentication: The discussion touches upon the importance of implementing security measures, such as SSL encryption and token exchanges, to protect service communications and user data.

- Testing and Integration Challenges: Wood shares insights on the complexities involved in testing integration points within an SOA, highlighting the significance of simulating external calls in testing environments.

- Resources and Community Engagement: He invites the audience to connect for further resources related to service-oriented architectures and addresses the importance of community in tackling such challenges.

In conclusion, Wood underscores that transitioning from a monolithic to a service-oriented architecture can significantly enhance an application's manageability, scalability, and security. By adopting these practices, teams can overcome the complexities associated with monolithic applications and improve feature development efficiency.

00:00:15.000 Hey everybody, it looks like it's about time to start, so I'm going to go ahead and begin. I figured since I was in Hawaii, I'd dress for the occasion with a pretty shirt. Hawaii is really beautiful, as you may know.

00:00:22.400 Anyway, let's get this started. I'm going to be talking about facing the monolith. How many of you have built or worked on a monolithic application of some kind? Yeah, just about everybody. You've been there, so you know that pain, right? Well, that's what I'm going to discuss and how we solved it on one of the applications that I've worked on.

00:00:33.320 To start us off, I'm going to introduce myself. I'm Charles Max Wood, the lead developer at Intentional Excellence Productions, which is my own consulting company. If you want to hire us to build something for you, just talk to me, and we'll make that happen. I'm also the host of the Ruby Rogues podcast. I see James over there, who is also on the show. We have a good time every week discussing a variety of interesting topics. If you haven't listened to that, you can find it at rubyrogues.com.

00:01:03.840 I also have a JavaScript show at javascriptjab.com and a freelancing show at rubyfreelancers.com, so if you're interested in those, feel free to check them out. I'm not going to dwell too much on what the application did, but I will talk about some of the components without getting too deep into the details. I mention this to illustrate that if you've worked on a monolithic application, you sometimes do wind up neck-deep in the complexities.

00:01:28.840 Let's talk about this for a minute. When you're starting to build an application, the mindset is often, "We just want something that does this little thing over here," right? You end up with something that's relatively simple and manageable. Your boss tells you to tame this thing, which, if you've ever tried to tame a cat, you'll know it's a challenge. The goal is to make it do something it didn't do before and solve a specific problem.

00:02:00.000 In our case, we were building a lead generation platform, and our directive was to create a system that could organize the leads. Sounds pretty simple, right? But as things evolve, demands grow, and suddenly the requirements change. Now, we need it to do something else. Just like cats don't bark, our application suddenly required that capability. This led to genetic manipulation of our application, making it do things it wasn't originally intended to do.

00:02:42.440 As a result, we ended up with a huge, unwieldy system, and fostering another feature became a Herculean task. Your boss keeps insisting, "Hey, tame this thing and make it work for us," which leads to keeping track of all its different heads and hoping it won't take your head off with its claws. It became a major pain in the neck.

00:03:03.360 Initially, we had an application that was essentially a bucket. But instead of just a bucket, it was a file cabinet, where we needed to organize leads. Databases can manage this organization reasonably well. You build tables and make them look nice, but then we realized we needed a system to collect the leads too. This might require an external API of some kind, which is relatively simple.

00:03:50.040 Then, we recognized that we need to make money by selling these leads. This required a mechanism to extract leads and send them to those willing to pay for them. Additionally, we needed websites for people to enter their information to generate leads. For SEO purposes, having some kind of CMS would also be helpful to draw people in.

00:04:29.480 People would come in, learn about majors they could pursue, and find the correct lead form to enter their information. Then we realized we needed user administration because not just anyone should access these leads. Reporting was crucial to understand how much money we were generating, and we verified leads through a call center, necessitating a call center app to verify the leads' quality. Eventually, we ended up with a convoluted mess.

00:05:15.080 The lead collection workflow became this elaborate process, and your boss would add new features that required you to teach the application new tricks without losing your head. Each time you added a new feature, it felt like shaving a yak—doing tedious work just to get what should be a straightforward feature implemented.

00:05:55.720 As I started writing this talk, I realized that a lot of the complications could be resolved with cleaner, more organized code. However, given the numerous systems we had working in tandem, it made sense to split the application into a service-oriented architecture (SOA). This approach facilitated better management as our monolithic application grew so large that it could tear the server apart.

00:06:34.840 Before we adapted SOA, we faced issues with scaling because the application wouldn’t just hang; it would completely stop functioning under heavy loads. SOA made it possible to scale horizontally and vertically by adjusting resources accordingly. I'll illustrate some differences between when you can just improve your code and when you need to rethink your architecture into a more manageable setup.

00:07:20.000 Managing a large, complex project is challenging due to the various elements working together. These systems are tightly coupled, which can be overwhelming as you try to keep all your concerns in order. Break them down into smaller issues makes it more manageable.

00:07:47.600 So we broke down the application into smaller standalone systems, which allowed us to focus on individual components. Everything above the line is customer-facing, while everything below consists of services primarily focused on data management. This de-coupling allowed us to streamline communication across services.

00:09:25.560 We had to address infrastructure issues such as how different applications communicate. Some processes need to be synchronous, meaning you require immediate responses while others can be asynchronous. For instance, when writing data, the user input is validated on the front end, and then dropped into the lead store asynchronously.

00:10:18.875 This included maintaining a job queue system which allows processing without needing to confirm each step immediately. If a failure occurs, the callback system can help identify jobs that need to be re-inserted into the queue, allowing redundancy. It's important to ensure that any asynchronous processing is regularly monitored to prevent bottlenecks.

00:11:08.520 Here’s a brief look at how HTTP connections were handled within our architecture. We called our lead service, which would return JSON data. This was done in a standard Rails format. Once the service returned, the necessary actions could be taken according to the business logic.

00:12:06.860 If a synchronous approach was necessary for reporting, we could make direct HTTP requests to those services. This ensured that timely updates were made without causing delays in the queue.

00:12:58.320 To summarize the organization, we had multiple web services and a communication structure between back-end and front-end services, which controlled how data flowed through our system. This was vital for authenticating users and managing access.

00:13:50.000 We initially used BeanoQue for queue management, but as our setup matured, we transitioned to using Rescue due to its user-friendly management capabilities. The lead service would process requests and send data to schools and other entities when necessary.

00:14:23.680 The queuing system would also handle writing new leads asynchronously while maintaining fast responses from our back-end services.

00:14:57.680 The fine-tuning of the queuing service is necessary to monitor its performance and ensure tasks are executed efficiently without bottlenecks.

00:16:21.600 When I originally prepared this talk, I wanted to touch on an idea proposed by the Ruby Rogues podcast regarding handling feeds. The concept is to utilize feeds to manage data flow and state consistency but ensuring that the same operations yield the same results.

00:17:07.799 In implementing front-end services, we developed interactive applications that interacted seamlessly with back-end services, creating a robust system for data management and user interactions.

00:18:18.880 We also built a service for user authentication and authorization, emphasizing best practices for how we manage and communicate authentication requests across our system.

00:19:13.400 Through organizing our services and ensuring each service only returned necessary data, we improved efficiency, reduced payload sizes, and enhanced our overall system architecture. The separation of concerns became vital in managing our complexity.

00:20:43.560 As for naming our services, we opted for a domain-based approach, making navigation and management more intuitive.

00:21:32.160 Security is crucial when managing service communication. Implementing SSL for encrypting communications, server registration, and utilizing API keys when necessary, can increase the robustness of your application.

00:22:25.320 If you're building JSON APIs, several considerations come into play regarding security and data management. I'll share further resources for deep-diving into these API-related topics.

00:23:53.680 The transition to a service architecture wasn't without its challenges, especially when working with integrations. There's a different set of skills and practices required when managing multiple addresses and ensuring smooth operations across everything.

00:24:11.640 If users encounter problems with communication or security breaches, we need rapid response protocols. This includes being proactive in monitoring services and having a solid deployment strategy.

00:25:46.320 I appreciate your attention throughout this talk. If you have any questions, feel free to take the opportunity to ask as I will be around.

00:26:48.159 As for branding our applications, we often considered public key encryption, user authentication methods, and strategies for token exchanges to ensure security.

00:27:43.480 From our experience, it's important to evaluate if multiple applications will share certain services and how to properly namespace those services when necessary.

00:29:37.640 Testing integration points, when multiple applications interact, can sometimes be challenging. Our approach involved setting up environments simulating those external calls to ensure everything worked seamlessly.

00:31:43.360 The setup process can indeed be cumbersome, primarily when trying to ensure that all services operate efficiently with each other.

00:34:49.480 Before we wrap up, please do connect with me via Ruby Rogues or on Twitter @Cmaxw if you need further insights or support regarding service-oriented architectures.

00:36:16.680 Thank you all for your time, and let's keep the dialogue going. If anyone wishes to discuss anything in depth, I'll be near the exit.

Aloha RubyConf 2012