Ruby Video
Talks
Component-based Architectures in Ruby and Rails
Stephan Hagemann
@shageman
#ruby-on-rails
#component-based-architecture
#rails-engines
#software-architecture
#test-driven-development
#modular-design
#event-driven-architecture

Component-based Architectures in Ruby and Rails

by Stephan Hagemann

The video titled "Component-based Architectures in Ruby and Rails" presented by Stephan Hagemann at MountainWest RubyConf 2013 explores the concept of organizing large applications in Ruby and Rails through component-based architectures. Hagemann emphasizes that while Ruby gems and Rails engines are often seen merely as packages for distribution, their true potential lies in structuring and organizing code effectively, particularly in growing codebases.

Key Points Discussed:
- Rails Applications vs. Gems: Hagemann begins by contrasting Rails apps with gems, noting that gems come with a namespace, whereas Rails apps start in an unstructured environment, which complicates maintaining coherence as they grow.
- Component-Based Architecture Importance: The presentation stresses the significance of breaking applications into smaller, manageable components, suggesting that this allows developers to maintain control as complexity increases. He mentions a quote from Justin Meyer that highlights the necessity of avoiding the construction of large monolithic applications.
- Iterative Architectural Improvements: Hagemann reveals eight steps for improving the architecture of a sample application characterized as an email announcement system.
- Normal Rails Application: Initially chaotic with indistinguishable components.
- Adding Structure with Modules: Introduces a higher framework structure without altering functionality.
- Gem Component App: Involves creating an independent gem to manage functionality, allowing for testable pieces of the application.
- Rails Component App: Discusses how Rails engines facilitate the structuring of Rails applications in a more cohesive way.
- Event Counter Engine: Separates concerns for better maintainability and clarity in app responsibilities.
- Service-Oriented Architecture (SOA): Suggests an eventual transition to a service-oriented app model.
- Extracting Gems: Emphasizes that successful apps can have certain components packaged as reusable gems.
- Conclusion on SOA: Indicates the final step of splitting a Rails app into microservices and managing them separately.
- Visual Metaphor: Hagemann concludes his talk with analogies about climbing a staircase versus cycling in a circle, illustrating that the path to architectural improvements is cyclical rather than linear.

Takeaways: The core message from Hagemann’s talk is the necessity of adopting a component-based approach when developing Ruby on Rails applications. As applications evolve, effectively managing their complexity through independent modules, gems, and services is crucial for maintainable and scalable architecture, thereby ensuring long-term success.

00:00:20.480 Thanks, Mike, thank you for the invitation. I want to start with a picture from this building, which I have to say I love this venue, and especially this building.
00:00:25.720 Especially when it's not on fire! I think we were really dumb yesterday, so if it starts beeping, I will be leaving this time. If you have not done it, I suggest you walk up this round arch; you can walk all the way up to the top.
00:00:39.920 And on that walk, you will see a stair somewhere where it says 'the beginning'. I want the beginning of my talk to be the end of my last talk on a very similar topic.
00:01:01.640 That was an idea that I was proposing and that was the realization that every gem you start begins with a namespace, while when you develop a Rails app, you never start with a namespace.
00:01:15.400 So, if you compare the two, a gem is like a box with a little label on it, while a Rails app is, let's say, a gorgeous infinite nothingness, complete with all the bells and whistles of Active Everything.
00:01:28.520 To gauge how relevant that is, who writes Rails apps on a daily basis? That's about two-thirds, maybe three-quarters. And who writes Ruby but never touches Rails? That's way less, although there are still a few.
00:01:41.200 I hope that for the second group, what I'm going to say will apply a little bit less, possibly because of Rails not being around. The idea from my last talk was that the next time you start an app, just put everything in a namespace.
00:02:05.000 I ended with what I found to be an awesome finishing slide: 'Give yourself a box so you can start thinking outside of it.' The idea here is that if you have a gem and you give something a name, putting things inside of it, you can open that box.
00:02:24.239 You might go, 'What the heck?' when you find something that obviously doesn't fit in the box. If you try that with a Rails app, you don't have a box; you don't have a name, so everything's going to fit.
00:02:31.200 The best game you can play is, 'What doesn't belong with the others?' For example, I don't know, red, green, loud; what doesn't belong with the others? You're going to know it's loud.
00:02:48.000 But if you look at your typical Rails app, what are you going to have? You're going to have a user, I don’t know, company roles, maybe the pricing engine that you wrote, and then the marketing part for that. You still think you can play that game and find out what may or may not belong to this app? It’s very hard.
00:03:13.320 This situation pushes us toward just putting all of our app in there.
00:03:25.040 What I want to talk about today is component-based Ruby and Rails architectures. I promise you, I put this slide in before that happened.
00:03:31.040 If I may want this button more desperately than you, trust me. Please stop me if I'm going too fast.
00:03:37.360 I want to talk about large applications, and one thing I heard is that you just never build them.
00:03:50.040 I work for Pivotal Labs, and we may start a new project every few weeks. Many of them are greenfield, so many of them are tiny when we start. However, you heard from Matt yesterday that we have pretty big apps as well.
00:04:03.720 For every app, our definition of success is that a client can stay successful after we leave. Thus, we have to strive for some sort of architecture that allows for application development over time.
00:04:18.560 We need to think about the app being larger than it is while we develop it. Despite TDD and all, Sandy Metz wrote her awesome book last year and gave a few talks.
00:04:36.000 One thing she mentioned was that your app is out to kill you; the growing complexity is going to chase you. If the app were a bull, you might feel like that guy.
00:04:54.080 This is exciting for a second, but then it becomes painful. Who has felt like their app was behaving like that bull? Okay, I have felt that way as well.
00:05:02.760 They’re typically not fast; quite the opposite, but they behave like something that is too big to handle.
00:05:11.759 So what I want to get to is for us to be more like this guy: calm, in control, with sunglasses and a hat.
00:05:17.080 But I don't really like bullfighting, so I actually want us to be this guy, just hanging out. I have to admit, I'm somewhat of a consultant, so I'm a little bit mercenary.
00:05:30.080 I want to go home at five on a Friday night. I think this guy is probably the one who's going to do that most successfully.
00:05:42.319 This never-build-large-apps is part of a larger quote from Justin Meyer, who stated the secret to building large apps is to never build a large app. Instead, break your application into small pieces.
00:05:55.560 Then assemble those testable bite-sized pieces into your big application.
00:06:06.919 I think he was talking about JavaScript; I'm talking about Ruby and Rails. You can tell from the font that you're supposed to read this slide.
00:06:20.360 So I'm going to use a tiny sample application throughout this talk.
00:06:27.800 If you cannot read this after all, just search for 'GitHub and the next big thing' and I think it comes up first. I promise this thing is tiny and it does only one thing.
00:06:40.840 It's an announcement page: you can announce whatever you have to announce and people can sign up for updates.
00:06:54.400 I actually don't know if I spell with two N's, and I don't know if my Twitter handle is only one. We'll see how often I tweet.
00:07:00.919 So the server thanks you for signing up. I just hit return, and when I made this sample app, I had to come up with a bit more than just that to actually have something to work with.
00:07:11.160 I realized that pressing return makes it really easy for me to do that many times, but of course I only want my email address registered once.
00:07:17.639 So I thought I'd give the server the ability to feedback the level of annoyance it feels while you're continuously pressing return.
00:07:24.560 At some point, it'll just freak out.
00:07:30.879 Ah, there's the freak-out picture. Let’s try again. The wireless is gone.
00:07:36.240 It's an awesome picture of a guy with a fantastic hairdo.
00:07:44.039 Okay, I need your help though. I need you to imagine something really big, because this app is tiny.
00:08:01.560 I'm going to make examples of refactorings or re-architecting that don’t make sense in that app. So bear with me and always imagine something very large.
00:08:09.720 I'm going to go through eight ways of architecting this application and I have roughly two minutes for each.
00:08:14.800 So let’s start with number one: a normal Rails application, where everything is in one place.
00:08:23.720 We have a controller here with two actions: 'new' and 'create'. Let's briefly see what happens.
00:08:30.800 It's no surprise; it's not a well-written method. John would refactor this rightly so. We’re trying to find the entry that you gave in this form.
00:08:43.760 If we find it, we update the tries on that entry and we put it into this annoyance meter, which will tell us how annoyed we are.
00:08:50.760 If we don’t find the entry, we're just going to say 'thanks for signing up', and there's some catching of unexpected cases, but that's pretty much what's happening in this controller.
00:09:01.200 Now, there’s a model backing this, called 'Entry', which has the email address and the number of tries. There's also the annoyance meter, which isn't really important, it just derives a new string from these counts.
00:09:17.320 So I suspect everyone here has seen this, and I assure you that in every one of these stages, you can run the tests, and if I didn't pick the wrong commit, they will pass.
00:09:36.000 I have tests, but I won't show them. I'll just be moving them around for everyone.
00:09:48.040 Now remember, I said that at the beginning of every Rails app, it’s kind of this infinite void. A good picture for that is probably a big dumb site.
00:10:02.880 I've already asked if you ever felt like the guy in front of the bull; maybe you also felt like the driver in that bulldozer.
00:10:09.399 The reason I'm saying this about such a tiny app is that there are two things in there that are now indistinguishable from 30,000 feet.
00:10:22.440 If you look at it, there's no real structure; it pretty much doesn't exist.
00:10:35.040 So for all intents and purposes, if you extrapolate these two classes to say 50 or 100, you have no clue who’s interacting with whom, what’s going on, and why.
00:10:49.680 Unfortunately, that might not look chaotic right now, but if I were to write 100 class names here, it would be chaotic—you wouldn't be able to assess any sort of structure.
00:11:08.160 So, a first way of getting structure into this application is through modules. The benefit we get from that is a higher-level structure.
00:11:19.600 Not much changes: the controller doesn’t change at all except it now references the 'Entry' and the 'AnnoyanceMeter' within their namespaces.
00:11:33.600 The annoyance meter class is now under a module. I've moved some other stuff around, mainly for testing.
00:11:49.920 So now we have this structure. If we think about the metaphor, we've moved from a dump site to a recycling yard.
00:12:02.079 Recycling yards are awesome; I should know because I remodeled my house and visited those places very often.
00:12:13.520 You can point to a corner of the recycling yard and say, 'That's where the scrap metal is.'
00:12:20.640 However, if you've ever gone to such a place, you've also seen people stick the hard-to-recycle stuff under their car.
00:12:35.360 They try to pawn it off to the recycling yard because they just want to get rid of it.
00:12:41.360 That’s indicative of a lack of guarantees regarding the independence of these classes and whether they're doing what they say.
00:12:56.360 So, at least I didn’t have to write 'entry' anymore because I now have a higher-level concept of 'email signup'.
00:13:09.600 The third step is what I want to call or improve on this: I already hinted at it by wanting to prove that these two pieces are independent.
00:13:21.720 I call it the gem component app. Now I may need to explain a bit more about this step.
00:13:31.680 First off, you're not seeing that engine's folder right there; we’re looking at the annoyance gem instead.
00:13:46.840 I made a folder called 'gems' and put an 'annoyance' folder there. This is a complete folder structure of a gem.
00:14:00.640 We have a gem spec that defines our annoyance gem.
00:14:11.120 There’s a gem file that tells us we’re running tests with RSpec, and magically, we don’t need to look at the tests now.
00:14:21.800 Now that I have tests in this gem, if I go into that subfolder, I can now run them independently.
00:14:33.760 Because of the way RSpec loads these files, I can prove to you that these tests for levels and for the meter of annoyance will pass without needing to know about email signup.
00:14:50.160 That matters a lot if you're dealing with tens or hundreds of files. I can prove that this thing is independent.
00:15:05.440 Now you might ask me two questions: one, why am I not using Git submodules? The answer is that I'm not smart enough. Two, why am I not using a different repository? The answer is I'm too lazy.
00:15:24.160 You can just do this.
00:15:32.720 How do I use it in the application? The gem file of the main application now points to the annoyance gem—it's referring to it by its path.
00:15:42.560 That’s all there is to it. This will be loaded and I can access it like before.
00:15:54.440 If I look into the teaser controller, we are still down here just loading that annoyance meter; nothing else special has happened.
00:16:01.640 The only thing that happened is that we now have provably independent tests, which is of great value.
00:16:08.840 In the words of Eric Evans, choose modules that tell the story of the system and contain a cohesive set of concepts.
00:16:21.679 We are now able to prove that a set of concepts, namely within that gem, is independent of the others.
00:16:32.000 I found something funny to put in this slide: it's good for push and pull to be so explicit, but if you look at the doors, it's not that explicit as to how to open them.
00:16:45.360 Now look at Icelandic signs; you know how to open these doors—they scream at you how to open them!
00:16:58.360 For the structure of the app, now there’s a second component, and it’s getting to be bold because it is provably independent.
00:17:09.999 We still have that email sign-up around, so we got a little more structure. Is that cool so far?
00:17:22.800 Let’s continue.
00:17:31.040 Now, gems make up a portion of apps, but if I have dependencies toward anything Rails, I have to do a lot of homework myself.
00:17:49.440 In the next step, I want to call the Rails component app. I’m still going to do that for a Rails component.
00:18:02.360 So, I want provable structure for Rails.
00:18:05.920 The way to do this—or more readily—is with Rails engines. If a Rails engine were actually a train engine, then Ruby on Rails applications would be trains.
00:18:22.760 I should clarify that engines still have this wrong perception of being for pagination, generic administration, and authentication.
00:18:38.200 They are not. If you look at the docs, it says Rails engines allow you to wrap a specific Rails application or subset of functionality and share it with other applications or within a larger packaged application.
00:19:01.080 I should know because I pushed that change last time I gave a talk like this. It’s great that it says that now.
00:19:15.320 So now, we have an engines folder in addition to the gems folder.
00:19:30.480 And you may have noticed that the app folder went away. Since we only had one controller, I just pulled something out.
00:19:41.120 There’s no longer a necessity for this app to contain code itself; it only contains gems.
00:19:55.320 Let's start with the more self-contained one: email signup. If we go in here, there is still that 'Entry', and it's now within a module.
00:20:05.280 This ‘Entry’ is now also an engine, which you can check out in the source code or find out more about it.
00:20:18.240 I can’t go into more detail now, but I’m happy to discuss that in the questions.
00:20:30.560 So essentially, this means I have my database migrations in here, I have tests for it, and it runs independently.
00:20:39.440 The main app, as before, just includes those gems and references them.
00:20:48.720 Let's quickly review the teaser, though, because the teaser now contains the controller that did before.
00:20:56.960 The teaser also contains asset controllers and views in my example; however, it does not contain models because it doesn't have any data itself.
00:21:04.960 The email sign up only has the model; it has only this entry.
00:21:14.160 The teaser must be getting that data from somewhere else, and indeed it requires email sign-up to be present.
00:21:25.040 It references that dependency in the gem spec so that it's a valid gem definition.
00:21:32.000 The controller hasn't changed much, though I made some modifications to the entry management.
00:21:39.640 You can find more about that in the code, but essentially, we're still just accessing the entry.
00:21:49.640 That's cool. I should probably look at the build scripts at this point or should have done that a while ago.
00:21:58.440 Since I have all these dependent components, I no longer just say 'rspec spec'. The top block just runs everything in the main app.
00:22:07.520 I think it just loads up the whole thing and sees that nothing breaks.
00:22:18.480 Then we go into the email signup engine and have model specs there, and then we go into the teaser engine.
00:22:39.880 We test the controllers, then have requests in the teaser request specs, Jasmine specs in the teaser, and still the test for the annoyance.
00:22:50.400 It's all in one. You could split this up; at this point, you can have about six parallel builds on Jenkins, all just a few seconds long.
00:23:00.560 The new structure is a shrinking Rails up top, a big teaser engine that controls what the app can do, and the annoyance gem.
00:23:10.640 Did anyone see a code smell around single responsibility in any of this?
00:23:20.000 I’ll give a hint: did anyone see a code smell around single responsibility? There is a thing called 'email signup entry'.
00:23:30.640 It contains the number of tries you took to enter your email. It’s not very visible, but if you think about what the output of such a table is when you finally publish your announcement.
00:23:46.920 What do you want to do? You want to send out emails to all the people. Why would this table need to know how many times someone tried to do that? It doesn't.
00:23:57.600 So this thing knows too much. Please bear with me and extrapolate to 100 files, and this issue will be everywhere. You will have a couple of god classes, and it will be hard to refactor them.
00:24:06.960 So we might as well do it now. Shoot, what was my number? Ah, number 5.
00:24:16.240 I want to get to the next point and make that problem go away. I want clearer responsibilities.
00:24:27.040 I will introduce what I’ve dubbed the event counter. Every time I try to sign up, it’s kind of an event.
00:24:38.200 However, because of my app's requirements, I'm not interested in seeing every event; I just want to count them.
00:24:46.160 So, I introduced an event counter engine in this case. It comes with a new class, event logger. It can log, using a generic object identifier and event identifier.
00:25:05.720 What you get back is a count of the number of tries you've already logged.
00:25:15.760 The teaser controller has changed a little bit; it injects another dependency down here. It now uses this event logger.
00:25:33.440 But other than that, you still see the code's essentially the same.
00:25:41.680 So I got rid of these two roles that the entry had. It didn’t make sense at this point, but it does make sense.
00:25:51.040 And if you think 'event counter' is kind of a stupid engine, imagine how Facebook is doing their likes: the event is just a status post.
00:26:09.360 The like is the action. Why not separate that into its own engine?
00:26:16.360 You can potentially use Redis within Postgres to make the storage and queries faster. If it doesn't need to be together, it shouldn’t be together.
00:26:28.480 So we might move toward a more loosely coupled architecture instead of the lousy coupling that was there before.
00:26:39.040 Now I want to come back to this logger class. I want to call this my sixth iteration on the architecture.
00:26:55.440 This iteration revolves around the service-oriented app.
00:27:03.840 I will not be talking about HTTP SOA; that’s point 8. This is just the service pattern.
00:27:14.840 You can find this in Eric Evans' book. The first thing you might notice is that there’s this active record class.
00:27:23.799 It's private, no it's not. You cannot make a class private like this. But, right now, that doesn’t matter.
00:27:36.160 What we're really offering here is what I want to call a service class: the logger that gives me one method.
00:27:46.040 You can log an event, and here’s the identifier. I want to do this to reduce surface area.
00:27:56.320 If you have not seen Justin C’s talk on mocking and testing, he makes good points about that.
00:28:06.920 Just some quick numbers: if you look at how many methods are on these two classes, the count and the logger, it’s way less on the logger.
00:28:17.440 If you subtract what an object has, then the logger only has one method; it screams at you what you should do with it.
00:28:30.560 I think this is especially important when extracting components and making them live on their own—they should state what they are about.
00:28:40.720 An active record class, I would argue, never states what it is about because you can always do so much stuff.
00:28:54.560 It’s very hard to prevent it. The structure doesn’t change, but we have just this event counter.
00:29:05.120 What I want to talk about is rather the next number: number 7.
00:29:13.440 You may have been wondering if I’m finally going to take one of these gems and rip it out.
00:29:24.640 In practice, I have done that but only after months on a project, when pretty much one project was done.
00:29:34.480 We realized, oh, that gem is actually one where we're interfacing with the main architecture, and it’s all repetitive stuff.
00:29:46.720 Then you're two steps away from that gem: build, gem, push, and you have a gem.
00:29:54.760 So it’s just the engine vanishing from the source code; it now lives in another directory.
00:30:05.680 I also pushed it to GitHub, and nothing else changed. Actually, one thing does change: the reference to this gem in the gem spec of the teaser now has a version.
00:30:13.440 In the gem file, there's no longer the reference to that path.
00:30:22.880 Again, there’s no change in structure, only that the middle point—the event counter—is now somewhere else.
00:30:31.640 This brings me to my last point: the HTTP SOA.
00:30:45.440 Marty Hot, who now organizes RailsConf, had this tweet about many talk submissions on how to split your mega Rails app into services.
00:31:00.200 Since my talk didn’t get accepted, I'm not going to talk you through it.
00:31:07.960 However, it's pretty clear what the next steps are. We already extracted an engine to handle the event counter.
00:31:15.720 What are you going to do next? You might take a tiny app like Sinatra, require that gem, and make one endpoint for logging.
00:31:27.480 Instead of in the teaser directly going to that event counter, call out to the service.
00:31:36.480 If it gets more complicated, you might want to revisit earlier talks to get your testing right.
00:31:47.600 This pattern, among all the structural architectural patterns that I've discussed today, I've only used once, and that was for auditing.
00:31:57.960 We knew this thing was always going to take all this data; it's never going to change.
00:32:10.360 We’re not interested in it while developing an app; you’ll be stuck maintaining multiple servers.
00:32:24.360 You will have to version stuff, but in the grand scheme of how to structure an application, this isn’t as special as many make it sound.
00:32:34.360 It’s just one more step that you can take, but by no means, you have to.
00:32:46.840 I've been running you through these eight different architectures for this tiny app.
00:32:59.680 And I've posed the HTTP SOA at the top of a staircase. What’s funny about reaching the top when it isn’t raining is that it looks pretty up there.
00:33:10.760 So again, you should definitely go up there. But the steps at the beginning and the end face in different directions.
00:33:18.800 I think this metaphor of a circle is more appropriate: as soon as I hit the next block and start a new app, I can start asking all these questions again.
00:33:28.800 I bet every one of you has touched one or pretty much all of these at some point in some application.
00:33:37.560 That's all I got. Thank you!
00:34:02.440 n
Edit