Multiple Databases

Summarized using AI

Keynote: The Future of Rails 6: Scalable by Default

Eileen M. Uchitelle • May 17, 2018 • Pittsburgh, PA

In this keynote address at RailsConf 2018, Eileen Uchitelle discusses the future of Rails 6, focusing on its scalability and the enhancements that will be implemented to ensure that Rails is "scalable by default." Throughout her talk, she addresses the misconceptions surrounding Rails' scalability and emphasizes that many successful companies have effectively used Rails at scale, despite its historical challenges.

Key Points Discussed:
- Introduction and Context: Eileen thanks the conference organizers and introduces herself as a senior systems engineer at GitHub, a member of the Rails core team, and discusses the persistent myth that Rails does not scale.
- Rails' Scalability Issues: She highlights how Rails has been criticized for its inability to easily scale applications, citing examples from various companies and the importance of embracing scalability in development.
- Goals for Rails 6: Eileen outlines her vision to make Rails 6 scalable by default, ensuring developers experience minimal friction during development and testing as their applications grow.
- Parallel Testing Feature: An important new feature discussed is parallel testing, which allows tests to run concurrently using forked processes or threads. This improvement aims to significantly reduce test suite execution time, ultimately enhancing developer efficiency.
- Multiple Database Support: Eileen emphasizes the need for better support and documentation for multiple databases, noting the challenges developers face when trying to configure multiple databases in Rails apps. She shares her experience on the time-consuming process of setting up multiple databases and the lack of existing tools in Rails.
- Collaboration and Community Involvement: Throughout the keynote, Eileen stresses the importance of upstream contributions, encouraging developers to build generic solutions that benefit the entire Rails community, rather than creating isolated fixes.
- Conclusions and Call to Action: Eileen concludes by advocating for improved collaboration within the Rails community to create scalable solutions. She asks developers to look at their tools and contributions that can benefit Rails and to help define its future, ultimately fostering a culture of shared growth and innovation.

Eileen's talk serves as a rallying cry for Rails developers to engage in community-driven enhancements that position Rails as a sustainable and scalable framework for future applications, ensuring it remains relevant and robust for years to come.

Keynote: The Future of Rails 6: Scalable by Default
Eileen M. Uchitelle • May 17, 2018 • Pittsburgh, PA

RailsConf 2018: Keynote: The Future of Rails 6: Scalable by Default by Eileen M. Uchitelle

RailsConf 2018

00:00:11.000 Good morning! I hope everyone who went running today is still out there.
00:00:18.330 Oh good! Well, I’ve never actually done the 5k, but I’ve always wanted to.
00:00:23.400 Then I find out it’s at like 7:00 or something, and that’s when I wake up to go running. So cool!
00:00:33.090 First, I want to say thank you to the organizers. This wouldn't be possible without them putting it all together.
00:00:39.989 So let’s give a round of applause for the organizers and sponsors, and everybody who makes this happen.
00:00:50.420 I'm Eileen Uchitelle. I live in Kingston, New York, which is about two hours north of New York City.
00:00:56.850 It's about a seven and a half hour drive to Pittsburgh. My flight was canceled, so I did the math.
00:01:03.870 Getting in a car was going to be faster than waiting for Newark Airport to get their act together.
00:01:10.830 You can find me online at the handle EileenCodes, and I occasionally blog at EileenCodes.com.
00:01:16.409 I’m a senior systems engineer on GitHub’s App Systems team. We do a lot with open source, Rails, Ruby, and Rack.
00:01:26.009 We focus on how these technologies interact with the GitHub application, keeping GitHub up to date.
00:01:32.009 I’m also a member of the Rails core team. For anyone new to the Rails community, the core team is responsible for defining the future of Rails.
00:01:38.579 We manage releases, fix issues, and handle all the good and bad that comes with releasing new versions of Rails.
00:01:45.750 If you’ve been in the Rails community for any amount of time or just saw Mark’s clickbait title yesterday,
00:01:51.180 you’ve probably heard the phrase 'Rails doesn’t scale.' We’ve heard it on Hacker News, in various NBC meetings,
00:01:58.380 and we even heard it in a misguided Vanity Fair article about Twitter, claiming they couldn't effectively combat harassment.
00:02:04.320 They referred to Rails as 'Fisher-Price software.' It’s funny because they meant it as an insult, saying Rails is for children.
00:02:12.680 But Fisher-Price and Rails are actually alike in that they are robust, well-designed, and unforgettable.
00:02:20.270 The interesting thing about Rails getting a bad rap for not being scalable is that it has taken many companies to a scale they never dreamed of.
00:02:25.730 If Twitter had tried to build their app to handle the scale it is today, they probably would have gone bankrupt before having proof-of-concept.
00:02:31.520 The reality is that Rails does scale; it just doesn’t scale easily. Historically, Rails hasn’t done a great job at making scaling easy.
00:02:40.520 While we pride ourselves on developer happiness, we were missing the mark on scaling and frankly falling short.
00:02:46.490 In this talk, we’re going to look at ways Rails has struggled with scalability and what we plan to do about it.
00:02:54.950 I want scaling Rails to be as enjoyable as running the Rails new command.
00:03:01.580 Scaling means different things depending on who you’re talking to or what your app is supposed to do.
00:03:06.890 For the purpose of this talk, scalable means that over time, an application’s growth in code, data, and traffic doesn’t degrade its performance.
00:03:14.269 While Rails can’t handle every aspect of scaling in your web application, there are two important areas I want to focus on.
00:03:22.010 The first is that when you scale your web application, your test suite shouldn’t block development.
00:03:27.830 As an application grows, new code shouldn’t create a ton of friction. Developers should easily prototype and build features quickly.
00:03:34.519 The time it takes for your test suite to complete shouldn’t hinder development. While we’ve made improvements to test performance in the past,
00:03:41.330 there's really no replacement for concurrent testing.
00:03:46.550 The other area where Rails has struggled with scalability is in handling increased data and traffic over time.
00:03:52.040 The database shouldn’t crash due to increased users or excessive data. It should be easy to distribute the load across your database.
00:03:59.209 Rails has had support for multiple databases, but that support is undocumented, hard to understand, and difficult to implement.
00:04:04.880 While Rails can’t tackle scaling every part of your application, I believe we can improve developer happiness around these two main issues.
00:04:11.720 My goal is to tackle these issues to make Rails 6 scalable by default.
00:04:18.820 Scalable by default means that engineers can scale their Rails application without having to spend hours writing a custom setup or using many external tools.
00:04:24.130 Scaling your application doesn’t have to be hard, and Rails should be able to take on at least some of that work.
00:04:29.440 Some of you may be thinking: 'Hold up, wait, Rails 5 just came out! Why is Eileen talking about Rails 6?'
00:04:34.810 Major versions of Rails allow us to reconsider what’s possible. It’s going to be a while before 6 is out, but I wanted to share my plans with you.
00:04:40.530 In the past few months, I’ve done a lot of work to refactor Rails and add features to make scaling easier.
00:04:47.169 We’ll look at what’s been done to speed up your test suite and improve Rails’ multiple database handling to ensure scaling is simple and intuitive.
00:04:52.360 The first feature I’d like to introduce for Rails 6 is parallel testing.
00:05:01.240 The parallel testing feature will allow you to run your tests concurrently using either forked processes or threads.
00:05:06.150 Parallelizing test suites is crucial for scaling web applications because as you add more code and tests, execution time increases.
00:05:12.610 You can mock a lot of calls or ensure you’re not making external requests, but that won’t speed up the time it takes to run ten thousand tests linearly.
00:05:18.780 Parallelizing large test suites can significantly reduce the wait time for builds to finish.
00:05:25.180 Many other companies like GitHub, Shopify, and others have been writing their own implementations of parallel testing for years.
00:05:31.610 The problem with us each writing our implementations is that we weren't sharing our ideas with the broader community.
00:05:37.320 Rails is popular for taking the complexity out of your app and turning it into reusable generic code.
00:05:44.640 By adding parallel testing to Rails, we set a standard for parallelization and make it simple to scale your test suite.
00:05:52.120 Earlier this year, Aaron Patterson and I worked together to write the parallel testing feature in Rails, based on the parallel testing infrastructure at GitHub.
00:05:58.150 While inspired by the GitHub code, the code in Rails has been rewritten to have a smaller footprint, be more generic, and easier to use.
00:06:05.660 Rails’ parallel testing also supports multiple databases out of the box, which was a requirement for us since we use several databases at GitHub.
00:06:12.430 The concurrent test runner in Rails 6 can parallelize your test suite using either forked processes or threads.
00:06:19.500 First, let’s take a look at the implementation for the forking parallelizer.
00:06:27.070 Forked process parallelization is implemented in Rails 6 using Distributed Ruby, also known as DRB.
00:06:34.150 DRB is a distributed object system in Ruby and can be implemented with zero dependencies.
00:06:41.350 The use of DRB is really cool and one of those features that I don’t think gets a lot of attention, so I encourage you to check it out.
00:06:48.420 Parallelization is invoked in your tests through the test helper with the parallelize method.
00:06:58.110 This method takes a workers argument, determining how many processes to fork.
00:07:04.720 Within Active Support, it’s backed by many tests.
00:07:10.630 When you start your test suite using the forking parallelizer, the Testing Parallelization class initializes with the number of workers you requested.
00:07:19.950 In the initialization method, we set a queue size, a server, and a URL instance variable.
00:07:28.250 The URL starts the DRB server locally and represents the URI for the server.
00:07:34.180 Parallelization in Rails is supported by MiniTest.
00:07:40.600 Once this class is initialized, all other methods within the class are actually called by MiniTest executors.
00:07:46.690 MiniTest will call the start method in the parallelization class, which starts an empty pool array to a block that forks the processes.
00:07:54.960 It then calls the after fork hook, allowing Active Record to hook into the parallelization code to create a database for each process.
00:08:01.600 After forking, a DRB object is created that represents the queue of work to perform.
00:08:08.380 The loop runs a method that executes the assigned tests for each parallel process.
00:08:14.390 At the end of the test suite run, each process will report which tests passed and failed.
00:08:23.649 Finally, the MiniTest call of shutdown will empty the queue and wait for each process to terminate.
00:08:30.240 This is essentially the entire implementation of work process parallel testing in Rails 6.
00:08:36.540 The underlying implementation is really simple and is only about 100 lines of code.
00:08:42.830 Don’t worry if you didn’t follow everything; you don’t need to know how it works to use parallel testing in Rails 6.
00:08:48.380 For newly generated Rails 6 applications, parallelization will be automatically included in your test helper.
00:08:56.040 For upgraded applications, you just need to call parallelize in your test helper.
00:09:02.540 The parallelize method takes a workers argument that accepts an integer. The default number of workers is two.
00:09:09.270 This means that Rails will automatically fork your tests into two processes, creating a separate database for each.
00:09:15.780 Each database is suffixed with the worker number, starting with zero.
00:09:22.480 These databases are generated in the after_fork hook we looked at earlier, then automatically deleted before the processes terminate.
00:09:29.780 Active Record and parallel testing do all this work for you, so no extra effort is required.
00:09:35.900 For those using multiple databases in their Rails application, we also provide hooks to handle additional databases.
00:09:41.780 The parallelize_setup hook is called after the fork and can be used to set up your secondary databases.
00:09:48.220 The parallelize_teardown method is used to clean up your work databases.
00:09:54.779 Using these hooks allows for a more streamlined process without complicated code.
00:10:01.680 If you're using a single database, you won’t need to use these hooks because Rails automatically manages the databases.
00:10:07.820 As you can see, the API for forked parallelization is relatively simple and easy to use.
00:10:14.650 The other type of parallel testing that Rails 6 provides is threaded testing.
00:10:20.460 You'll want to use threads instead of processes in some cases.
00:10:27.300 For example, if you’re using Windows because UNIX sockets that DRB relies on aren't supported.
00:10:33.650 You might also want to use threads if your tests are I/O bound.
00:10:41.500 Most of your tests are not I/O bound, but if they are, they’ll run faster with the threaded parallelizer.
00:10:46.420 The caveat, though, is that your code must be thread-safe to use the threaded parallelizer.
00:10:54.400 The parallelize method can take a second optional argument that defaults to processes.
00:11:02.070 Setting it to two threads is that simple.
00:11:09.240 The parallelize setup and teardown hooks are not supported by the threaded parallelizer as it uses a single database.
00:11:15.640 When running tests in a suite, they must use the same type of parallelization or number of workers.
00:11:23.020 For example, you can’t pass five workers to one set of tests using threads and three workers to another set.
00:11:31.000 If you need different behaviors, split your tests into two separate suites.
00:11:37.300 Threaded parallelization in Rails 6 uses MiniTest’s threaded executor.
00:11:44.030 We didn't have to write any additional code for the threaded parallelizer, since MiniTest already uses threads.
00:11:51.520 When Aaron and I started working on the threaded parallelizer, we experienced some challenges.
00:11:58.820 We were running tests and suddenly experienced random failures, deadlocks, and strange behavior.
00:12:07.020 We first assumed the issue was related to my demo app using multiple databases.
00:12:13.600 After switching to a new app with a single database, we observed the same issues.
00:12:19.680 We discovered that each thread was connecting to the database with the same connection ID.
00:12:27.420 This raised isolation problems. When I reviewed my earlier coding on system tests, I realized it was my fault.
00:12:34.360 When I implemented system tests, I enabled a connection to the database on one thread.
00:12:40.700 Capybara booted a Puma server, opening a second thread and second connection to the database.
00:12:46.700 This created inconsistent data between test runs because the transactions didn’t know about each other.
00:12:52.460 To fix this, each thread needed its own connection.
00:12:59.700 I realized this the moment I recognized that my previous code was creating this issue.
00:13:05.660 With the threaded parallelizer, threads should maintain their own connection to the database.
00:13:14.600 Fortunately, since I wrote the original problematic code, fixing it was relatively straightforward.
00:13:20.320 We added the ability to configure the setting internally, allowing each thread to open a separate connection.
00:13:27.780 However, system testing may not work with the threaded parallelizer.
00:13:34.520 While running your parallelized tests doesn’t require extra work, you can set them up in your test helper.
00:13:41.560 Simply run Rails test like you normally would, and Rails will handle the rest.
00:13:47.580 Sometimes, you may want to adjust the number of workers based on your environment.
00:13:55.920 For example, you might use two workers locally but 14 on CI.
00:14:02.060 Rails provides an environment variable called PARALLEL_WORKERS that takes precedence over workers passed in.
00:14:09.500 If the number of workers is one or fewer, parallelization won’t be applied.
00:14:14.800 If you’re interested in the PR for parallel testing, you can check it out on GitHub.
00:14:23.860 Parallel testing will enhance the speed of your test suite, and I'm excited about this feature.
00:14:31.660 Now that we've got your test suite squared away, let’s examine how Rails handles scaling under increased traffic.
00:14:39.330 I’m not talking about DDoS traffic; for that, you’d want to use an external vendor.
00:14:44.850 Instead, we’re discussing scaling because a high-profile user, like Justin Bieber, starts using your app.
00:14:50.380 You need to scale fast, and you need to scale yesterday.
00:14:57.950 I’ve been in these situations before, and it’s not fun.
00:15:04.920 You wake up to find your primary MySQL database struggling under the load.
00:15:11.190 You know you need to split tables from your primary MySQL cluster, needing read and write databases.
00:15:18.150 Rails has had support for multiple databases for a while; we use this at GitHub.
00:15:25.390 However, implementing this is an arduous task.
00:15:32.280 It took me three hours to incorporate multiple databases into a demo app in Rails 5, and that’s after I’d already done it.
00:15:39.260 For someone who has never worked on this kind of setup before, it would take even longer.
00:15:45.490 The difficulty stems from a lack of Rails standards.
00:15:50.080 Although the underlying plumbing works well, we're not providing straightforward documentation or the tools to utilize it.
00:15:56.730 The second key highlight we’ll discuss are improvements to using multiple databases with Rails.
00:16:03.010 These may not be flashy features like parallel testing, but they are critical for scaling web applications.
00:16:10.110 Multiple databases become necessary when a single instance can’t accommodate all the traffic.
00:16:16.890 If your application is crashing due to database issues, it may require splitting tables or adding read replicas.
00:16:22.580 At GitHub, we operate multiple primary databases with corresponding read replicas.
00:16:29.330 While Rails has supported multiple databases for a long time, thanks to work by Arthur Neves and Matthew Draper,
00:16:36.320 Rails 5 made it significantly easier than Rails 3 or 4 to handle.
00:16:43.919 However, as I set out to build a demo app using multiple databases, I quickly realized there was much missing.
00:16:50.070 Let’s explore what an application with multiple databases looks like, identifying where Rails falls short.
00:16:57.060 Imagine we have an application with two databases: one primary database with a flowers table and one animals database.
00:17:03.200 By default, Active Record is familiar only with the primary database connection.
00:17:10.430 To connect to and interact with the animals database, we face challenges.
00:17:17.060 For any setup involving multiple databases, best practices aren’t laid out in Rails.
00:17:24.780 There’s no documentation on how to structure your database YAML, models, and migrations for the secondary database.
00:17:31.341 Following our setup at GitHub, I initially structured my database YAML like this:
00:17:38.240 I used a two-tier configuration alongside the default production namespace and a second namespace for animals.
00:17:43.180 However, this might not be ideal and is no longer the recommended method with Rails 5.
00:17:50.240 Now, using a three-tier setup is best when you need multiple databases. This leads to clearer organization.
00:17:57.350 The 'primary' database is technically an arbitrary name; I recommend keeping it as 'primary' for clarity.
00:18:02.850 Next, we have the animals entry for managing tables of dogs and cats.
00:18:09.920 In the three-tier configuration, we can distinguish which databases belong to a production environment more easily.
00:18:16.340 If you utilize two-tier configurations, or even a single database per environment, you do not need to alter your settings.
00:18:23.260 Once the databases are set up, we must instruct Active Record where the migrations for the dogs and cats tables reside.
00:18:29.780 I’ve worked with migrations across multiple databases before, and a few options exist.
00:18:36.280 One option is to modify your migrations to set the connection at the beginning, which I don't recommend.
00:18:43.040 Instead, I prefer organizing all migrations for the animals database into a separate directory.
00:18:50.220 It's cleaner and more straightforward to group migrations by connection.
00:18:57.049 To use this method, however, we need to specify migration paths whenever we call migrate.
00:19:05.000 This works with custom rake tasks but is challenging with existing Rails code.
00:19:11.950 Existing tasks assume migrations reside in the DB migrate directory without awareness of additional directories.
00:19:18.200 Thus, the second problem with multiple databases in Rails arises as migrations don’t work seamlessly.
00:19:24.710 We can write rake tasks that will execute the migrations, but if you rely on Rails internals, they may not operate correctly.
00:19:30.900 Next, we establish animals connections for the dog and cat models.
00:19:37.000 We create a new base class, naming it AnimalBase.
00:19:43.240 For it to function correctly, we need to set abstract_class to true.
00:19:50.150 This informs Active Record not to use the implied STI table from the parent class.
00:19:55.960 For example, we don’t want the dog model to look in the animals base for the dogs table.
00:20:03.260 Now, we need to establish a connection to animals for our child classes.
00:20:11.320 Rather than inheriting from ApplicationRecord, these models need to inherit from AnimalBase.
00:20:17.780 This tells Rails to utilize the connection corresponding to the animals database.
00:20:24.250 Though we’ve identified some setup issues, it seems straightforward.
00:20:30.250 Unfortunately, none of the rake tasks will work for non-primary databases.
00:20:36.770 Create, drop, and migrate will strictly operate on the primary database.
00:20:43.850 There’s no Rake task for migrating or managing non-primary databases.
00:20:51.030 We need to devise separate Rake tasks for each secondary database.
00:20:57.150 After writing the necessary tasks, I encountered another problem.
00:21:03.200 When running the create task for the primary database, my app threw an error.
00:21:09.520 I double-checked my Rake tests, ensuring the three-tier config was correct.
00:21:16.000 After debugging, I discovered that the default connections for the three-tier config were not working.
00:21:23.900 When establishing a connection, Rails didn’t know which database to connect to due to multiple entries.
00:21:30.600 At this point, I've identified multiple difficulties with scaling databases in Rails.
00:21:38.020 We've noted no documentation, limited migration support, absent rake tasks, and mismanaged default connections.
00:21:45.880 Setting up multiple databases in a Rails application remains a challenge.
00:21:52.520 My scope for the original project was to extend GitHub’s behavior with multiple databases.
00:21:59.150 However, it evolved into three distinct problems each requiring substantial time.
00:22:05.610 In these moments, I felt overwhelmed and questioned why I’d even chosen to tackle this.
00:22:11.860 I then refocused on addressing migrations as an initial priority.
00:22:18.950 Addressing documentation first wasn’t effective, so I opted to tackle migrations.
00:22:25.320 The parallel testing needed to support how we handle multiple databases.
00:22:32.660 Previously, you had to set migration paths manually before calling migrate.
00:22:38.950 This was manageable for direct calls, but it was inconvenient with Rails internals.
00:22:45.460 Ideally, multiple databases should offer easier interactions with migrations.
00:22:52.350 Aaron and I revamped the migrations handling to store paths on the connection instance.
00:23:01.030 We needed Rails to query the model for migration paths instead of relying on application level settings.
00:23:07.620 This adjustment allows Active Record to operate more seamlessly in databases.
00:23:14.560 With the restructure to accommodate the models, Rails can determine migration directions effortlessly.
00:23:22.380 We discussed needing to create standard rake tasks for managing secondary databases.
00:23:28.620 Existing commands assumed only the primary database was relevant.
00:23:35.520 I wanted commands like rake db:create and rake db:migrate to operate on all databases.
00:23:42.740 However, the challenges with the three-tier config build up.
00:23:48.900 Many Rails frameworks are predicated on the assumption of having a simpler two-tier config.
00:23:55.370 But with each modification or testing, we need to clarify which database applies.
00:24:01.910 By switching to object representations for config hashes, we can more effectively query.
00:24:08.220 Now, with our new DatabaseConfig object, determining namespaces and environments becomes feasible.
00:24:14.710 This allows us flexibility and control over database returns.
00:24:20.630 We can ask specific questions about configurations and use methods to manage tasks.
00:24:27.400 Let’s consider how the migrate tasks work within Rails 5.
00:24:34.050 The existing tasks would load the primary config and invoke migrate.
00:24:40.580 Due to its reliance on existing database connections, it wouldn’t accommodate multiple databases.
00:24:46.370 To rectify this, we will use the 'configs for' method to migrate both primary and animals databases.
00:24:54.430 When employing a block, we gather the configuration hash and instantiate the connection.
00:25:02.460 These updates allow migrations to run more intuitively.
00:25:09.960 We can start connecting and invoking migrations on all databases.
00:25:15.790 The get-configs and other lookup methods sufficiently manage performance.
00:25:22.590 The goal is to effortlessly push forward user-defined tasks for database configurations.
00:25:28.640 We can automatically create intuitive tasks for different databases.
00:25:36.480 I wanted the names of these tasks to be easy to discover, so they read intuitively.
00:25:43.200 Naming conventions will also help users know which commands map to which task.
00:25:50.300 This setup allows for clear control over operations across databases with ease.
00:25:56.570 These adjustments conclude the current steps we've managed.
00:26:03.330 Though the three areas need improvement, I expect these changes will enhance the Rails' experience.
00:26:10.250 Next on our agenda is addressing the default connections issue with three-tiered setups.
00:26:18.390 With a temporary solution in place, I am dedicated to resolving it more effectively.
00:26:26.390 Aaron and I are working to change Active Record’s underlying connection management.
00:26:34.070 This will enable better selection of required configurations.
00:26:40.490 It's undoubtedly an extensive undertaking, as connection management in Rails is complex.
00:26:46.930 I hope that by next RailsConf, I can present all the features that will emerge from this refactoring.
00:26:53.350 There’s much more work to do to make multiple databases truly user-friendly with Rails.
00:27:01.750 I aspire to continuously improve the API for splitting reads and writes and further enhance performance.
00:27:09.320 We also need to develop thorough documentation regarding these features.
00:27:15.610 My hope is that the work done so far inspires further enhancements in scaling the database management in Rails.
00:27:23.260 I’d like to thank my friend and colleague Aaron Patterson for his collaboration on these features.
00:27:31.220 He kicked off the work on parallel testing and supported me in navigating connection management issues.
00:27:39.090 I would not have accomplished any of these features without his help.
00:27:46.420 A couple of weeks ago, someone asked me if I'm concerned that Rails will become bloated with scaling code.
00:27:53.080 This is indeed a valid point. Most apps don't need GitHub's scale.
00:28:00.960 Yet, when I consider how GitHub has necessitated Rails scaling over time, I can't help but wonder.
00:28:08.280 What if we had introduced these features five years ago?
00:28:16.330 How many more applications could be utilizing parallel testing and multiple databases?
00:28:23.320 Would we still hear people say that Rails doesn't scale, instead of inquiring how to make it scale even more?
00:28:31.880 Over the past year at GitHub, I have worked on upgrading our application from Rails 3 to 4.
00:28:37.780 While doing this, I have explored GitHub’s entire stack, from code hosting to wikis.
00:28:44.570 I noticed the different decisions made over the past decade—both beneficial and otherwise.
00:28:52.250 I know that GitHub isn't the only environment that has forced Rails to scale.
00:28:59.080 I have also worked on custom multiple database setups and parallel testing with other companies prior to GitHub.
00:29:06.760 I know Shopify has their own solutions for multiple databases and parallel testing.
00:29:13.380 I began to wonder why we all create similar tools yet don’t share them.
00:29:18.970 This occurs from our belief that our scaling problems are unique.
00:29:26.520 This mindset leads to an inward focus on solutions.
00:29:34.780 When faced with urgent issues or bugs, we often prioritize quick fixes over long-term solutions.
00:29:41.710 These emergencies motivate us to write stopgap measures instead of building comprehensive scaling solutions upstream.
00:29:48.420 As emergencies arise, it becomes tempting to rush to fix the issues at hand without planning for future scalability.
00:29:55.620 Later, those patched solutions can become tangled and difficult to maintain.
00:30:03.180 Applications often evolve into fragile structures where one change can lead to much larger issues.
00:30:10.100 When this happens, skepticism arises regarding whether Rails truly scales.
00:30:17.900 The constraint lies not within Rails but within the custom code we wrote.
00:30:24.680 We often build specialized solutions that restrict flexibility, making future changes challenging.
00:30:31.340 While developing tools without communal input, our code becomes rigid and unmanageable.
00:30:37.350 We find ourselves creating similar tools repeatedly, resulting in duplicate efforts.
00:30:44.800 However, we can approach issues with an upstream-first perspective.
00:30:50.150 First and foremost, we must accept that our applications aren’t extraordinarily unique.
00:30:57.440 If our applications succeed, then most likely, others will require scaling solutions.
00:31:04.370 Therefore, sharing solutions can enable us all to benefit.
00:31:10.660 I’ve witnessed several organizations reinvent parallel testing, forcing scaling solutions without contributing upstream.
00:31:17.760 It's clear that maintaining unique setups can stifle progress.
00:31:25.540 I don’t intend to sound accusatory.
00:31:32.240 Instead, I believe we are building near-identical solutions.
00:31:39.000 When scaling issues arise, we tend to seek fixes within instead of reaching outside.
00:31:46.680 This leads to missed opportunities for collaboration and improvement.
00:31:53.700 I believe it’s time to embrace the notion that sharing solutions enriches our ecosystem.
00:32:00.640 From a community standpoint, we can gather knowledge and reduce redundancy.
00:32:07.460 By pushing our solutions upstream, we can help build a more cohesive Rails framework.
00:32:14.640 This can lead to down-the-road advantages, enabling us to build more flexible and adaptable applications.
00:32:21.640 Upstreaming our code offers numerous benefits for the organization and community.
00:32:28.500 The first step towards success is acknowledging that our applications aren’t special.
00:32:35.470 If our applications succeed, scaling issues are universal.
00:32:42.160 Let’s share our solutions instead of hoarding them.
00:32:49.200 By sharing our knowledge, we uplift the entire community and enhance Rails’ adaptability.
00:32:56.410 I firmly believe my upstream multiple database handling will aid many other companies—and they can improve it for me, too.
00:33:03.240 This collaborative effort diminishes the burden of individual work.
00:33:10.240 Working alone in a vacuum isn't enjoyable; sharing creates an environment of constant learning.
00:33:15.960 During my years at GitHub, we’ve continually refined our application to scale more efficiently.
00:33:22.680 Now is the time to give back, sharing our experiences with the entire community.
00:33:30.000 By reflecting on how we have spurred Rails’ evolution at GitHub, we can enhance Rails' framework.
00:33:38.240 I want everyone who uses Rails to discover that developer happiness is achievable over the years.
00:33:44.640 But this is only possible if Rails continuously equips developers with the needed tools.
00:33:52.320 So, to answer the question: Do I think Rails will become bloated with scaling code? My answer is 'no'.
00:33:58.160 Contrarily, Rails is maturing, not dying.
00:34:05.580 If Rails aims to remain the go-to for web applications, the Rails team must meet those applications' needs.
00:34:12.360 We must build our applications not just for prototypes but for long-term reliability.
00:34:20.180 This maturity path involves talking about the scalability of features upstream.
00:34:28.080 So I ask you: How can your applications and contributions elevate Rails?
00:34:34.950 What have you done to encourage Rails to scale for you?
00:34:43.080 What can you give back to our community? Can you enhance parallel testing?
00:34:50.600 Do you have insights into optimizing multiple database handling?
00:34:56.580 Do you have tools that can be generalized for broader use?
00:35:03.450 Rails is defined and will continue to evolve through you, its community.
00:35:10.270 Let’s collaboratively define the future of Rails, ensuring Rails 6 is scalable by default.
Explore all talks recorded at RailsConf 2018
+98