Do You Need That Validation? Let Me Call You Back About It

Talks

Tobias Pfeiffer

Do You Need That Validation? Let Me Call You Back About It

by Tobias Pfeiffer

In the talk titled "Do You Need That Validation? Let Me Call You Back About It," Tobias Pfeiffer discusses the complications that arise from intertwining validation logic and model callbacks in Ruby on Rails applications. He shares insights on how these practices can lead to convoluted code, debugging challenges, and performance issues as business and view logic become entangled over time.

Key Points Discussed:
- Problems with Callbacks: Pfeiffer introduces the concept of 'before-validation' callbacks, illustrating their pitfalls through a story about building an event management application. He emphasizes how such callbacks can obscure debugging by causing unexpected behavior in the application if not carefully managed.
- Common Validation Practices: Using the GitLab user model as an example, he critiques the excessive custom validations resulting from specific view-related concerns that inevitably clutter model logic.
- Performance and Test Setup Issues: Pfeiffer explains how constant validations can inflate test setup complexity and runtime by requiring multiple database records for validation checks, thus increasing the overall performance burden when testing.
- Affordances in Code Logic: He makes a case for re-evaluating how models are structured within the Rails MVC architecture, arguing for clearer separation between business logic and validation logic.
- Alternative Solutions: He explores various methodologies to address these challenges, including:
- Form Objects: Simplifying the validation process without entangling models with unnecessary validations.
- Inheritance and Active Type: Using inheritance to create models devoid of excess logic.
- Change Sets in Functional Programming: Describing a more structured data manipulation concept he admires in Elixir, emphasizing its benefits in clarity and organization.
- Separation of Operations and Validators: Advocating for a clean structure between controller actions and business logic to improve maintainability and clarity.

In conclusion, Pfeiffer urges developers to critically assess their use of validations and callbacks, prioritizing a cleaner, more manageable structure in their Rails code. By acknowledging common pitfalls and adopting alternative approaches, Rails developers can enhance their code quality and streamline their applications. This talk highlights the balance between leveraging Rails effectively and managing complexity in code architecture.

00:01:03.729 Good morning everyone! Can you hear me? Yes? Great! Thanks, everyone, for making it. I know yesterday was party time, so I hope you all had some time to enjoy and relax. We're going to start this off very slow.

00:01:09.050 To start off, I'm going to tell you a little story. I call this the 'Story of the Magic Hook.' So, imagine we have a problem. We want to build a new application to manage events. An event has a name, a location, and a date. At some time, the crew arrives, and then performers arrive, and so on. However, our events are just user groups, so they only ever take place on one day at a time. We always just want to mention one day at a time. This is fairly repetitive. We want them to be date times so that they can be efficiently compared in the database with other dates, but we have to enter the date all the time. This is frustrating, you know? We want to be user-friendly, focusing on UX. It’s obviously much better if we just enter the date once and then enter all the times.

00:02:08.270 So, what do we need to do to make this work? We write the following code. This is a before-validation callback that calls 'set day times to date.' We first get the base date, which is our date field, and we convert it to a DateTime object so we can manipulate it. Then, we have a constant called 'date_time_fields,' which includes all the fields we want to set to that specific date – when our crew arrives and so on. We get the original time of those fields and then adjust the time by taking our base date and changing the hour and minutes to the hour and minutes that we entered. Finally, we set that value.

00:02:36.880 So how does it look? We create a new event for 'Ruby on Ice' on the 24th, and then, okay, our crew didn't actually arrive at 6:45, but I arrived somewhere around 9:30. That’s fine. So if we run the code, then all of the fields get populated for Sunday, February 24. We have it all done right. This is some actual magic, you know? I don't need to make anything else; this just works for us. It's a great idea. Let's write a test!

00:03:15.770 We want to build a new event using FactoryBot, and we set it, for instance, to end in the year 2042. Now we write a test. So first, we look at the end of the event, and sure enough, it really is set for the first of January 2042 after we save it. However, our magic hook triggers and says, 'No, no, no. You're not on the first of January 2042 anymore; thanks to the factory, you're set to the 24th of February.' This is surprising, and it's probably not that surprising to you right now because I just showed you all of this. But now, imagine you're writing a test for something completely different. You wrote this magic callback a year ago, and now you want to create a new query to fetch events that are happening in the next 23 years.

00:04:12.830 So I create this event, and I know I just look at the 'ends at' field. I want to create it with the 'ends at' field set to more than 23 years in the future, and this just creates it. Then, when MicroM decides not to include the event, I will spend hours debugging because I think my query is wrong. I won't think that my factories ran their setups wrong. I first always look for the fault in the code that I just wrote because I don't even know about the other code right now.

00:05:01.020 Maybe we are even lucky if we figure out that there’s a bug sitting somewhere in a concern, but we usually find that out very late. And I’m not telling you this as an adjusted example; I have spent hours debugging this very same problem, so it is not made up. So, what do I think about this? I think before-validation callbacks are a smell. A smell is an indication that something’s wrong. There may be unseen errors. This is a good idea, but it’s just very hard to think about right now because the question is: Why does the model have to care about cleaning up the data that it has parsed? Shouldn’t this process be handled before? We did this specifically for one or two views – the create event view and the edit event view – for nothing else.

00:05:51.620 However, the before-validation callback will always be run, no matter if we go through the view or not. This is very view-specific code that clutters our model callbacks. The only thing that clutters your model is something that is very specific. Let’s talk about validations. This example is from the GitLab codebase; it’s from the user model. I want to clarify before I take this example: It’s not that I think there are bad engineers at GitLab. On the contrary, I think they are extremely capable engineers. The opposite is true. They do great work, and I am thankful for the open-source nature of their projects.

00:06:29.110 I can show you this example specifically because of their good work. If we look at this, we see a bunch of custom, written validations for things like unique emails and notifications. They all check what changed to consider this change. Why are they spending so much effort on this? Why not handle these validations at an earlier point in the process instead of having a 'last line of defense' at the model level? For instance, if we stick with the analogy of football, if a goalkeeper is trying to catch all the shots coming his way, it’s a clear sign that we don’t want to run this check all the time. We inadvertently set it somewhere where it runs all the time, but we could have caught it earlier.

00:07:16.600 Now, you might say, 'Oh, Toby, maybe they do it for performance reasons or because those validations are very expensive.' But it’s extremely fast to validate the presence of something. I mean, validating the presence of a field is super fast. Now, let’s take the example of a doctor’s appointment. A doctor’s appointment needs to have a practice, a doctor, and a patient. All of these need to be associated. So first, when we make the appointment, we want to see if the practice is indeed open at that time. It should not overlap with any other appointments – either of my doctor or my practice, because I want to ensure I can be at my appointment when needed. But all of these involve checking other appointments, and these are very expensive database operations that I have to perform.

00:08:22.590 Of course, it gets even worse if I change associated models, because they also run all their validations. Validations aren’t free, at least not the ones that mean anything. Another problem is that when you set up your tests, it can become extremely expensive. For example, if we have a belongs-to relationship, it’s never optional. So, we check on every valid call if all those associations belong to the same entity. While in theory, this is a good thing, in practice, it might not be, especially when testing. Oftentimes, you will need records in the database to perform whatever tests you want to run.

00:09:31.390 If we're sticking with our appointment example and want to create an appointment, since it belongs to a practice, I need to have that practice in the database and additionally, each of the doctors and patients. But it doesn’t stop there! The practice is also in a specific area and has its own tariff systems. All of these need to exist in the database either because of the validations that are run or because the belongs-to relationships we check, just once! Any co-pays I was working on required creating potentially 23 models, which means inserting 23 records in the database to make everything valid. And that’s really bad because there's no wonder that test execution might take 20 minutes if you need to create that many records in the database.

00:10:12.000 Looking back at the code I mentioned earlier, I actually didn’t show you everything of it. I cut out the validations that do not pertain to emails. However, you can see that all of these are checking some way or form of validation related to changes in emails, and I find myself asking: How many ways can I possibly change an email? At GitLab, I don't know, but I would guess it’s maybe two or three at most four. The user model is used everywhere and is manipulated in many places. However, we become the goalkeeper, trying to prevent violations from coming through these expensive validations.

00:10:36.200 To clarify, I’m not saying these if-statements are bad. These if-statements are merely symptoms of a larger problem. I also have code that looks like this, and if your code doesn’t look like it, perhaps it should. Because it may end up executing validations all the time that shouldn't be executed. Giving you a little example, let’s go back to our doctor appointment: if we change the state of the appointment from 'open' to 'done,' do we need to run all the validations? No. We didn't change anything else. Why would we run all those validations? It makes no sense. In fact, these checks are more of a symptom than the main problem.

00:11:55.100 If you know me, you know I have a particular relationship with comments. When I looked at this, I found relevant comments saying 'in case validation is skipped.' This piqued my interest, so I dug a bit deeper. We execute the same callbacks, set public email and set commit email, twice: once for before_validation and once for before_save, just to take precautions in case somewhere, someone decides not to execute validations because they are expensive. Just so we ensure execution, we add a before_save callback. I trust the engineers at GitLab a lot, but something drove them to write this code, and it leads to executing validations twice if validations are not skipped.

00:12:34.760 My big question is: Why are we doing this? Why is this something we actually think is good to write? I’ve thought about this for a long time, and even if I’m nervous that everyone here will think, 'Oh, we never write code like this,' I hope that's not your current feeling. The point here is that I've come up with an answer, which is about affordances. Affordances is a term I learned in a course on human-computer interaction and user experience design, and it stuck with me. The simplest definition is: how does an object want to be used? For instance, a door handle wants to be pushed, a button wants to be pressed, a chair is easy to sit on.

00:13:51.050 In programming, when we implement a class in an object-oriented language, we want to represent our solution space with a method that acts as the affordance. In functional programming, we write a function that takes an input and produces an output. But in Rails, we have a Model-View-Controller (MVC) architecture, where we are advised not to put logic in our controllers and instead put all the logic in our models. The result? We might have one or two use cases that impact every model they touch. The models themselves become a big mashup of concerns, resulting in added business logic, validations, and callbacks.

00:14:55.280 Now, you might say, 'Oh no, Toby, we have service objects.' Service objects are a whole other topic. A few years back, people began to say, 'Okay, we don’t want all this business logic in the models; let’s extract some of that to a layer above.' This alleviates some problems, so our models get a bit smaller. However, the validations and callbacks still mix in with the model logic, which I think is even worse. Validations run by default all the time, whereas the business logic runs only when commanded. You might manage a situation where models become crowded and difficult to manage, but at least the logic only runs when instructed, unlike validations.

00:16:06.150 I had one case in a production application where there was a stack trace error because two before_save callbacks were playing ping-pong with each other. I had to debug this funny situation because it only occurred under very specific conditions – a full moon in February! It was very challenging to debug. I’ve also been dishonest with you slightly because I don’t only have control over service objects and models; we also have views. You might say, ‘No! Views shouldn’t call controllers; this is wrong.’ You are right, but the views – especially forms – dictate what data structures the controllers deal with.

00:17:26.580 For instance, consider the typical Rails user model. I focused only on the concerns I want to talk about here since the user model is typically much larger. The validations are all related to sign-ups or edits; otherwise, I don’t care if the email is entered. I care about the email only when signing up or editing – nothing else. The same applies to passwords and the acceptance of terms of service. I don’t care about that under any other circumstance. Also, validation concerns leak into the model from the view. My personal favorite is the 'validates_terms_acceptance' validation. This is just a purely view-related concern.

00:18:59.210 So the question remains: Why do we have that? What I began to realize is how something above looks affects how something down here reacts. I'm solving this problem of validations and callbacks all the way down here, and they run all the time. Why am I playing this game if it doesn’t need to be this way? I don’t want to do this, especially when it could be avoided. Let’s consider another point of view, starting from a framework perspective. Initially, we might say, 'Run all callbacks all the time! That's great!' But eventually, we change our mind: 'I want to run this callback because it gives me a bug.' Or 'That callback is slow – I don't want to run that.' In the process, we lose sight of where we came from and what actions were executed.

00:20:37.600 Why can’t we move this phasing up? What if we do it between the controller and the service? This way, we know what controller is executing and what validations we want to run. Maybe we can even move our logic to a middleware, like Rack, that validates before it ever hits the controller, because we know exactly what state we are in. You've just listened to me complain about validations for 20 minutes; is this a good use of your time? I hope so.

00:21:58.880 I want to clarify the problems as I see them. If I start telling you about solutions without being clear on the problems, none of you will listen to me or care. I believe it is vital to put these issues into perspective. Some perceived problems include: first, particulars of some features that are used in only one or two controller actions clutter the model, which makes it very hard to grasp anything because everything is mixed together. Additionally, you don’t always know what is happening at what point in time, which is frustrating. Perhaps the worst issue is that validations run all the time, leading to odd interactions between callbacks that make debugging a nightmare.

00:23:46.730 Further issues include performance impacts and a harsh test setup. In this next section, we will look into some ActiveRecord originals and see how we can make it better. But before we do that, let’s see what Rails has in store for us. Is there no solution for this in standard vanilla Rails? One suggestion might be to simply place everything into concerns. You can include validations in a concern and use it for user registration. However, while this may help organize things, it doesn’t really fix the underlying issue, since all those validations and callbacks still run all the time.

00:25:32.440 This merely offers a grouping of operations in your model. If you suppress validations, it can indeed prevent them from running partly – that’s my favorite thing in the world. You may know about suppress, where within a block, no notification record will be created. This is useful if you want to copy a project over and create new project instances, but you don’t want to fire notifications for it. But is this a healthy solution? When I checked both the GitLab and this course co-pays, I was relieved to discover they don’t use suppress.

00:26:43.760 We also have custom contexts with validations. You probably know that on validations, you can pass on create, on update, or similar, so you can explicitly designate which validations to run. This might solve part of our problem but only for validations. Unfortunately, this doesn’t apply to callbacks, so it's not a real long-term solution either.

00:28:46.420 So what other solutions exist? How can we change this? First, let’s consider Form Objects: Form Object libraries are plentiful, but we could work with plain Active Model. This way, we don’t need to add extra gems, making it simpler to integrate into existing codebases. We can have familiar validations; we need to add a tree accessor because we pass the object to the view and that object needs to have setters to deal with the form. There's a need to maintain compatibility with Active Record and define methods similar to Active Record methods.

00:29:44.050 Next, we can execute our callbacks in a more straightforward manner. Instead of having a callback that sets the password digest directly, we can create a method like `hash_password` to handle that. We pass the result from the user to a base user model that has none of these validations or callbacks. Similarly, after saving the user, we can have a method `send_welcome_email`. This way, it becomes much clearer when things are executed, and it simplifies managing callbacks and validations when associated models are involved.

00:31:59.510 Next, we can consider another approach that involves inheritance. In this model, we inherit from something that seems unusual. This library is called Active Type, made here in Ruby. With inheritance, our user model appears without the validations or callbacks present, allowing you to add specific logic only necessary for each subclass. You might wonder why you need a library for that, but Active Type manages special cases such as single table inheritance and routing for you, so it combines various benefits into a single solution.

00:33:04.610 Now, let’s discuss Change Sets. I personally favor this approach as you can see, it isn’t Ruby but rather Elixir. The pipe operator passes the user as the first argument to a function, and whatever that function returns gets passed to the next operation. Why am I mentioning Elixir? Because I find this approach outstanding, even though I haven’t seen something quite like it in Ruby. Notably, Phoenix—an equivalent of Rails for Elixir—has this concept. A context allows you to define data manipulation while enforcing a strict structure on the parameters, controlling which fields can be modified, such as email or terms of service.

00:34:11.620 We then state which validations to run and include callbacks. We maintain clarity on when various actions occur; we avoid mixed concerns within one process. This also promotes easier debugging, which is vital for any productivity. A further extension of this idea involves keeping various contexts separate, allowing for clear and organized structures within the codebase. For example, you can create multiple change sets that can define boundaries for attributes and their validations.

00:35:24.490 Lastly, we should look into separate operations and validators, a concept often used in newer Ruby frameworks such as Hanami and Trailblazer. Here we break down an operation into a policy for access control and separate validations. In the controller, we can pursue a registration operation through normal parameters – while the operation itself creates the model, builds the contract, and addresses validations. This straightforward framework sets an excellent structure when treating operations separately.

Ruby on Ice 2019