Ruby Video
Emil Ong
Putting Rails in a corner: Understanding database isolation
Emil Ong
@emilong
#activerecord
#concurrency
#transactions
#data-integrity
#ruby-on-rails
#postgresql
Summarized using AI

Putting Rails in a corner: Understanding database isolation

by Emil Ong

In this video titled "Putting Rails in a Corner: Understanding Database Isolation," Emil Ong provides an insightful discussion aimed at Rails developers dealing with database transactions and isolation levels. He addresses the common issue of inconsistent data appearing in applications even when code is wrapped in transactions, emphasizing the importance of understanding database isolation levels in Rails applications.

Key Points Discussed:

  • Understanding Database Transactions:
    • Transactions are sequences of operations performed on a database. They must follow the ACID properties: Atomicity, Consistency, Isolation, and Durability.
  • Focus on Isolation:
    • Isolation ensures that concurrent transactions appear to be executed serially. This is vital for data integrity.
  • Spectrum of Isolation Levels:
    • The video outlines various isolation levels, from "Read Uncommitted" to "Serializable," explaining the trade-offs between performance and data consistency.
    • Common Isolation Levels Explained:
    • Read Uncommitted: No guarantees; can read uncommitted changes.
    • Read Committed: Only reads committed changes but can still have varying results if read multiple times.
    • Repeatable Read: Ensures consistency in reads within a transaction but can still lead to inconsistencies if not handled properly.
    • Serializable: Full isolation, prevents any anomalies, but can be the most performance-intensive.
  • Rails and Isolation Levels:
    • Since Rails 4, developers can control isolation levels directly in their transactions. However, increasing isolation can lead to performance issues due to increased load and potential deadlocks.
  • Practical Examples:
    • Ong provides a hypothetical scenario using a congratulatory card application to illustrate race conditions that can cause inconsistent data statuses. He explains how simultaneous actions within transactions can lead to unexpected outcomes.
  • Testing and Handling Exceptions:
    • Recommendations on testing strategies that accommodate isolation levels and how to identify and manage rollback scenarios in Rails applications. Testing isolation changes in combination with transactions can pose challenges.
    • Emphasizes that specific isolated transaction tests may require database cleaning strategies instead of standard transaction rollbacks.

Conclusions and Takeaways:

  • The Necessity of Isolation Levels: Understanding and configuring database isolation levels is crucial for ensuring data integrity in applications.
  • Performance vs. Consistency Trade-off: There is always a balance to maintain between performance gains and transaction isolation requirements.
  • Ongoing Improvement Needed: Ong advocates for a more accessible way to manage these complexities in Active Record and Rails, implying that community contributions may be necessary to facilitate better practices for developers.
00:00:11.210 I'll go ahead and get started. Thanks everybody for coming.
00:00:15.150 The title of this talk is "Putting Rails in a Corner: Understanding Database Isolation," and that was originally a reference to a quote from the movie Dirty Dancing.
00:00:18.330 Much like a lot of lines from 80s movies, it kind of came out of nowhere and doesn't really go anywhere. So, this isn't really a good Dirty Dancing-themed talk, but instead, you get these lovely botanical illustrations, which I find quite nice.
00:00:30.840 So, I'm Emil Ong, and I'm a software writer at Aptiv. Aptiv is a great company; we work in the gov tech space, helping foster agencies with the goal of finding every child a family. We're hiring, so if you want to talk to us about that, feel free! I'm also on Twitter, and you'll find my handle in the bottom right-hand corner. If at any time you want to tweet something about this talk, I occasionally talk about computers, but mostly I share music theory, jazz, and sometimes cute animals.
00:01:02.520 Now, let me answer the first question that should be on your minds right now: who might benefit from this talk? If you have some controller actions that you've wrapped in a transaction and you're still finding that you're having inconsistent data afterward, stick around; this is probably for you. If you have slow background jobs that use Active Job or something alike, and those need to run transactions, you're another good candidate. Or if you simply write anything to the database based on something that you read from the database, that’s quite general.
00:01:35.100 In other words, this talk is for people who need confidence in their data and probably use a SQL database. Otherwise, this might not apply as much. But if, on the other hand, someone just called you and said that they have ice cream and it's going to melt, like, totally, that sounds better, so feel free to duck out.
00:01:46.110 Great! Thanks for staying. Now, let me make a kind of meta note before we really dive in. This is a dense topic, and the takeaway here isn't that you're going to walk away and immediately write code based on, you know, configuring database isolation levels. Instead, I recommend that you get a sense of the kinds of problems affected by this space, how you might think about configuration, and then just remember some terms so you can go back and search for them later.
00:01:57.580 Let’s consider the context we’re discussing. We’re talking about transactions and how they interact with ORMs, especially Active Record. It was awesome that DHH set this term up in our mind: "leaky abstractions." That's what this is all about and why we still need to talk about this.
00:02:10.140 So just a reminder: transactions are sequences of interactions with the database that you hope have certain properties. If you've heard the term ACID, it's an acronym standing for Atomicity, Consistency, Isolation, and Durability. Atomicity means that either everything you tried to do happened, or nothing happened. Consistency means that when you run a transaction, it starts with a consistent state of the database and ends with a consistent state of the database. It might be the same state or a different one, but it should be consistent. Isolation is the primary focus of this talk, and durability means that the things you write and commit are expected to stay there until you delete them.
00:02:46.750 What we’re going to discuss is how we can control isolation and, in effect, control consistency—not so much about durability and maybe a little about atomicity. In this talk, Active Record is fantastic; it gives us a really helpful object model for dealing with our data. It even gives us some controls for transactions. We can start a transaction and roll it back. As of Rails 4, we can control the isolation level, which is super helpful. But unfortunately, that might hide some of the finer points about what's going on with the database that we need to know about to make our code behave as we expect.
00:02:59.960 Many of you may have used transactions, wrapping a block of code, thinking it should have those kinds of ACID properties. But let's consider a scenario with an imaginary app called the "Congrats App." This app is designed to send a card from a group of people to one person, usually to congratulate them.
00:03:15.430 In this app, you can create a card, identify a recipient, and set a group of people who can sign that card. People can come in, sign, change their signatures, or remove their notes. There's a deadline for sending the card, and if not everyone has signed by that deadline, we go ahead and send it. If you've ever tried to get a bunch of people to sign a card for someone, you know there are always some stragglers.
00:03:28.000 Let's say our timeline begins when Edie gets a promotion. Edie has a friend named Pat, who decides to send a congratulatory card and invites Dana and Rhys to sign. Dana immediately signs, but then some drama happens between Dana and Edie. At the same time that Rhys decides to sign, Dana decides to remove their signature. We’ve set up our app, so Edie is getting a card, but when, and whose signatures are on it?
00:03:45.920 Let's zoom in on that simultaneous event. Rhys clicks the sign button, which hits our controller. We load the card, add Rhys's signature, see if everybody has signed, and if they have, we go ahead and send the card. Meanwhile, Dana clicks the remove button, signaling that they no longer want to be associated, which deletes their signature. This outcome could depend on various factors like garbage collection.
00:04:09.750 What’s supposed to happen in either of these scenarios? First, we could choose not to send the card until the send-by date, ensuring that only the people who intended to sign during that time are represented. I think that’s probably what I would want: Dana's signature should not be there, but everyone else's should be.
00:04:37.250 Alternatively, we might send the card before the send-by date, which could lead to inconsistent data. This brings up the issue of why we can end up in these inconsistent states. When we actually perform the sending, we assume the context hasn't changed, but this assumption could lead to problems.
00:05:04.590 The crux of the issue is that databases make trade-offs: they relax isolation standards for improved concurrency and performance. Keep in mind the cute dog photo I used earlier to help you remember this essential point.
00:05:14.910 There's a spectrum of isolation levels we can choose from, trading off performance, cost, and how isolated we are. You will always be in one of these isolation levels, depending on your database, which may have a different default implementation.
00:05:38.180 Let’s go through the isolation levels. Starting with Read Uncommitted: there are no guarantees about isolation here. You can read rows that might be updated or inserted in other transactions that haven't even committed yet. If you read data during your transaction and then read it again, it could be entirely different, leaving you completely unaware of any changes.
00:06:05.700 Then we have Read Committed. In this case, you can only read records that have been committed by another transaction. Still, there’s a chance of reading the same data twice and getting different results. This level is better than reading incomplete work but doesn’t provide full assurance.
00:06:18.480 Next up is Repeatable Read, which means that if you read a row and try to read it again, you'll only see changes that you made. This level offers some guarantees, but you can still end up with inconsistent data.
00:06:57.540 Serializable is the highest isolation level, meaning transactions only occur in a way that they could have been executed serially. In this mode, there are strict safeguards against inconsistent data. It's the most expensive in terms of performance, but it provides the highest level of correctness.
00:07:17.600 Let’s revisit our initial scenario. If we were using Repeatable Read or Serializable when Reese attempts to sign the card while Dana is removing theirs, the database would warn us that our assumptions are invalid. This function acts as a helpful alert to avoid inconsistent data.
00:07:51.500 Now that we've grasped the severity of these isolation levels and their functions, it's crucial to remember that Rails gives us the ability as of Rails 4 to set isolation levels within our transaction blocks. This means we can adjust the way our apps handle database interactions.
00:08:09.700 However, increasing isolation guarantees can hinder performance due to the added load on the database. This could involve holding locks or needing to repeat transactions. That's just the cost of doing business.
00:08:22.830 Let's explore how to manage these transactions in Rails. It involves defining the isolation level as a parameter within the active record transaction block.
00:08:37.950 Although it's a simple parameter change, it's essential to consider the potential performance impacts. As mentioned, increasing isolation may cause load on the database and could lead to deadlocks.
00:09:08.710 There's also the nature of Active Record abstraction to keep in mind. While it handles data caching and writing intelligently, it might complicate the awareness of when data is read into a model. This requires using special procedures like preloading and eager loading to ensure proper data read during transactions.
00:09:35.450 For example, say we implement send-by dates utilizing the Active Job framework within Rails. You could instruct Active Job to perform certain actions based on the card model at a later time, such as sending the card.
00:10:03.940 The challenge arises because, while the initial read happens to confirm the state of a card, it may take place outside the transaction. Thus, leveraging the serializable isolation level for that job doesn’t adequately account for all risks.
00:10:32.200 To improve this situation, we need to ensure that data is re-fetched within the transaction context based on the card ID associated with the job.
00:11:00.720 Another technique involves adding a boolean field within the card model that tracks whether a card is committed to being sent. This adjustment simplifies decisions regarding sending, minimizing redundancy in processing.
00:11:30.680 When reflecting on transactional approaches, be mindful of potential loops. For instance, sending reminder emails to users who haven't signed might require wrapping multiple actions within a transaction. This demands care to avoid any inconsistency.
00:12:01.770 Greek advice states that keeping transactions small can help minimize errors. Aim for a consistent ordering when accessing resources; targeting consistent ID order is a good start. Inconsistencies or dangers of deadlocks rise when unpredictable access patterns evolve.
00:12:35.950 A better alternative could be gathering cards in batches that may need reminders, wrapping each within separate transactions, and then triggering reloads to ensure data correctness.
00:13:03.580 To test these concepts thoroughly, bear in mind that identifying rollbacks can be tricky. Active Record helps by throwing the "ActiveRecord::StatementInvalid" error. While the precise reasons can vary, understanding how to invoke isolation errors through various tests can guide you.
00:13:29.660 When it comes to testing, setting up transactions around your tests helps ensure clean results, rolling back everything after the test runs, but beware; changing transaction levels during tests can lead to complications or performance drops.
00:13:54.380 Lastly, when testing concurrency, try manual testing alongside load testing approaches. This includes random sleeps to replicate potential issues, allowing you to see what goes wrong consistently.
00:14:24.290 In summary, remember the dog! Databases aim to strike a balance between isolation and performance. Active record allows you to set levels, but you should remain wary of the need for code and test adjustments to support it.
00:14:51.390 Achieving correctness is essential, sometimes even mandatory. This can be complicated.
00:15:09.390 After listening to other talks, I realized that making your code safe from isolation errors can often feel cumbersome. It evokes the challenge of threading code, yet I believe it can be streamlined.
00:15:44.860 Understanding the use cases behind these practices can inform better patterns we might apply in forthcoming Rails developments. This way, we support a safe and robust database environment without overcomplicating our code.
00:16:09.550 Thank you to Slides Carnival for the nice slide deck. Thank you, everyone!
Edit
Explore all talks recorded at RailsConf 2018
+98