00:00:24.859
Hi everybody, can you hear me? Oh, that's better. Yeah, okay. Cool. Thank you, Adam, for waking everybody up for me and for making them all move forward. Good afternoon, everyone! We had a lot of great talks today, and I'm really excited to be back at RubyNation. I was here three years ago for my second talk.
00:00:44.129
It was on Active Record. This talk is not about Active Record, but that table is how I started contributing to Rails. I’m Eileen Uchitelle, a Senior Systems Engineer at GitHub, on the GitHub Platform Systems Team. That's a bit of a mouthful, but essentially, my team is responsible for working on internal and external tools for the GitHub application. We work on improving Ruby, Rails, and other open-source libraries.
00:01:02.910
Somehow, I have become personally responsible for upgrading GitHub from Rails 3 to 4 and then 4 to 5. One day, I’d like to do that slowly over there, Paul. I'm also the newest member of the Rails Core Team, which means that I have finally gotten access to the Rails Twitter account. That’s all we want, right? You can find me on GitHub, Twitter, Seekers Deck, or anywhere I lean codes. It's what I use everywhere unless I don't want you to find me, and I will just use email.
00:01:39.509
Today, we're going to talk about the new system testing framework in Rails that I spent six months building. I will take an in-depth look at my process for building it, the roadblocks that I hit, and what’s unique about building a feature for open-source software. But first, let's travel back in time a few years. Some of you may remember the infamous 2014 RailsConf, where DHH declared that test-driven development (TDD) was dead. He felt that while TDD had good intentions, it was ultimately used to make people feel bad about how they wrote their code.
00:02:17.190
DHH insisted that we needed to supplement test-driven development with something that motivates programmers to test how their applications function as a whole. In a follow-up blog post titled 'TDD is Dead, Long Live Testing,' he stated that Rails does nothing to encourage full-system tests and that there's no default answer in the stack. He claimed that was a mistake that the team needed to fix. Fast forward three years later – a little longer than that – and I'm happy to announce, as other people have communicated earlier, that Rails 5.1 finally makes good on that promise by including system testing as part of the default stack.
00:03:04.380
The newest version of Rails now includes Capybara integration, allowing you to run system tests with zero application configuration required. Generating a new scaffold in Rails will automatically set up all the requirements for system testing without the need for you to provide any configuration. It just works!
00:03:31.099
At this point, it’s a good time to address exactly what I mean by 'system test.' You may be familiar with Capybara being referred to as an acceptance testing framework. While it uses all the same libraries, the ideology behind system testing in Rails goes deeper than that. The intention of system tests is to evaluate your application as a whole. This means that instead of just testing individual parts or units of your application, you assess how those components work together.
00:04:03.400
With unit testing, you ensure that a model has a required name, and in a separate controller test, you check that the controller throws an error. However, you can't actually test that a user sees the error message when the system is tested. In system testing, that becomes possible. You can verify that when a user inputs their names incorrectly, the appropriate error message is displayed in the view. This method also allows you to test your JavaScript and how it interacts with your models, views, and controllers.
00:04:39.669
Now, before we take a look at what it took to build system tests, I want to show you what a system test looks like in a Rails application. When you generate a new Rails 5.1 application, the Gemfile will include Capybara and Selenium WebDriver gems. Capybara is locked to version 2.1.3 and above to ensure your app can use some of the new features pushed upstream. Upon creating your application, a system test helper file called Application System Test Case is generated.
00:05:04.090
This file includes the public API for Capybara’s persistent tests. By default, applications will use Selenium with the Chrome driver running in the Chrome browser, with a screen size of 1400 by 1400 pixels for screenshots. If your application requires additional setup for Capybara, you can include that in this file instead of cluttering your test helper. Every system test that you write will inherit from your Application System Test Case. Writing an individual system test isn’t much different from writing Capybara tests, except that Rails includes all of the URL helpers.
00:05:49.060
Thus, you can simply write post_url or posts_path instead of having to include the Rails URL helpers as many do. For instance, you might want to test that a user can visit the post index and assert that the h1 selector is present with the text 'Posts'. You can run these tests using Rails' test system, but keep in mind that Rails does not run system tests in conjunction with others because they can be quite slow. Typically, they’re run in a separate CI build since they depend on a real browser.
00:06:38.980
Instead of having to separate them, it’s now automatically organized. Let’s take a look at systems in action! I recorded a demonstration because, in a brand-new application where not much is happening, the tests run too quickly for demonstration. So here you have it. First, we're going to write a second test for creating a post. The test visits the post URL, and we will click on the 'New Post' button just like a user would. I’ll fill out the post attributes for the title 'System Test Demo' and include some lorem ipsum as the content.
00:07:38.390
Just like a user would, the test clicks on the 'Create Post' button. After redirecting, we'll ensure the text on the page matches our expectation, which is to find that 'System Test Demo' was created successfully. When we execute it with Rails Test, the server boots up, and Chrome starts to display the test setup.
00:08:06.220
You might then wonder why it took three years to build system tests? There are a few reasons. The first is that system tests needed to inherit from integration tests in order to utilize all the URL helpers. However, integration tests were sluggish, and their performance was abysmal. The Rails team could not advance system testing through integration tests without facing significant backlash from the community.
00:08:49.040
No one wants their test suite to suddenly run for ten minutes instead of five. That kind of performance hit isn’t acceptable. Speeding up integration tests had to occur before implementing system tests. Back in 2015, I worked with Aaron Patterson on enhancing integration test performance. Once we had integration tests running marginally better, system tests were free to inherit from them.
00:09:10.880
Another reason for the delay is that, contrary to what many believe, the Rails Core Team does not have a secret feature robot. We’re all volunteers, and if there's no one interested in implementing a feature, it simply won’t happen. Although we individually have ideas about what we’d like to see in Rails 6, hardly anyone can call it a roadmap. Most features emerge from real problems encountered in our applications.
00:09:50.950
System tests are an excellent example of this. Prior to my time at GitHub, I worked at Basecamp. While building Basecamp 3, we decided we needed to incorporate system testing through Capybara. I firsthand experienced the significant work it took to establish system tests in our application. This was a primary driver in getting system tests into Rails 5.1.
00:10:25.370
The effort required to implement system testing in Basecamp 3 reignited the motivation to develop this feature for Rails, allowing others to spend less effort so they can focus on what truly matters: writing software. David Heinemeier Hansson asked me if I’d be interested in incorporating Capybara and integrating it into Rails for an upcoming release.
00:11:04.350
To that end, much of my contribution to Rails has taken the form of performance improvements, refactorings, or bug fixes. I was excited to build a brand-new feature into the Rails framework, but I faced an initial hurdle: I had never used Capybara before. I know that sounds ridiculous, given that I’ve been involved in Rails development for years, but I had struggled with writing system tests based on other frameworks.
00:11:50.310
I had never set up an application using Capybara nor created an entire test suite before. However, this lack of experience allowed me to approach the project without preconceived notions about what would be easy or hard to implement.
00:12:31.510
By not having prior experience with Capybara, I was able to view the feature primarily from the perspective of Rails and Rails applications—important as maintaining the integrity of Rails’ ecosystem was paramount. Rails is highly opinionated about what code looks and feels like, so it was crucial that the implementation for system tests aligns with that philosophy.
00:13:09.610
It's essential to implement system tests that require minimal setup, allowing programmers to focus on their code rather than on configuring tests. Consequently, when implementing something you’re not familiar with, it’s best to establish guiding principles. These principles help you make design and implementation decisions more straightforward. Without defined goals, scope creep can lead to confusing function arguments regarding minor details.
00:13:50.060
Having guiding principles enables you to assess whether your code aligns with those guidelines. For the system tests, I naturally used the Rails Doctrine. The Rails Doctrine consists of nine core tenets that drive all decision-making related to code within the Rails ecosystem. While building system tests, I would regularly base my decisions on these principles, ensuring that system tests meet all of them.
00:14:31.679
I’d like to highlight three of these core tenets that clearly exemplify how to build requirements. The first tenet is 'optimized for programmer happiness.' This overarching theme is relevant in all of Rails' products; the primary goal is to make programmers happier. Frankly, I am spoiled because of this—writing Rails makes me happy, and I hope our endeavors bring you the same joy in your daily work.
00:15:14.250
However, what does not make me happy was all the implementation required to get Rails running with Capybara. The bare minimum code required for applications to utilize Puma, Selenium, and Chrome for system testing alongside MiniTest was frustrating. Many applications faced tedious setup processes to support various drivers or to meet custom settings, but Rails 5.1 changes that by allowing you to use Capybara without any configuration.
00:15:52.300
Programmers are the driving force behind integrating system testing into Rails applications. You don’t need to understand how to initialize a Rails application with Capybara, how to handle various drivers, or how to change your web server settings for system tests. All that work has been covered so you can focus on writing code that excites you.
00:16:31.870
If you are an architect, the Capybara integration was already implemented by Sam Phippen, who has opened a pull request to replace the existing Rails System Testing with an enhanced version. If you come across bugs in any of these implementations, it's worth reporting them back to the respective contributors.
00:17:10.740
Let’s take another look at the simple method driven by. When you generate a new application, the Test Helper file generated will include this method. If you upgrade your application, this file will also be generated when you create your initial system test.
00:17:51.410
All the code we looked at earlier is encapsulated within this one method that initializes Capybara for your Rails app, pointing the driver to Selenium, using Chrome as the browser, and defining a customized screen size. Rails inherently values being an integrated system, or a monolithic framework, addressing the entire web application development process, from databases to views, to web sockets and testing. By being an integrated system, Rails minimizes duplication and external dependencies.
00:18:38.560
Before Rails 5.1, Rails didn't solve the demand for system tests. The addition of this feature enhances Rails into a more complete, robust, and integrated system. As DHH noted in 2014, Rails was incomplete in terms of system testing. With the release of Rails 5.1, that gap is now filled, and you no longer need to look outside of Rails to add system testing to your applications.
00:19:19.750
You might hear people say that Rails sacrifices stability for progress. While this often means beta releases, release candidates, and even final releases may have a few bugs, it also means that Rails has not stagnated over the years. There’s been tremendous progress in the framework, and we care deeply about our users, also considering how the framework meets the needs of both present and future users.
00:20:04.080
Some features have been stabilized over time, but you won't truly know a feature is working perfectly until someone else tests it. I could have spent years working on it to resolve every single issue, but I ended up merging it when I was aware of a few bugs remaining. I realized that the community would uncover issues and contribute fixes that I hadn't thought of.
00:20:47.160
By prioritizing progress over stability and merging the system tests when they were approximately 95% complete, instead of waiting to achieve 100% stability, many community members tested the beta release and identified fixes for bugs already present. A few features were even added during this time, and some functionalities were incorporated upstream to Capybara.
00:21:39.880
The evolution of system tests progressed more effectively by merging them early when some bugs were present than by delaying until they were entirely stable. Now that we've explored the driving principles behind system testing, let's delve into the decisions surrounding their implementation and architecture within the Rails framework.
00:22:26.100
The first configuration default I want to highlight is my choice of Selenium as the driver. The barrier for entry for system tests should be zero, making it easy for beginners to utilize Capybara. The default Capybara driver, RackTest, isn’t great for system testing because it can’t handle JavaScript and doesn't take screenshots. It also presents a challenge for those trying to learn how to test their systems.
00:23:11.430
Many users shared their opinions, suggesting that Poltergeist would be a better default choice due to its speed. While Poltergeist is indeed quicker, I leaned toward using Selenium for several reasons. One of the most exciting aspects of Selenium is that you can actually see it running tests in real browsers rather than using headless drivers like Poltergeist and Capybara WebKit.
00:23:52.950
Watching Selenium execute tests in a real browser feels almost magical. It's rewarding to see your code run automatically. This visibility makes it easier for new programmers to learn and differentiate what’s happening and spot potential mistakes. The best part about system tests is you can switch drivers easily; to change the driver used for system tests, just open your Test Helper file and modify the 'driven by' method.
00:24:43.820
The caveat is that you will need to install PhantomJS if you want to use it rather than Selenium, which adds additional complexity that I tried to avoid. Capybara restricts integrations to Selenium, Poltergeist, and a few other drivers, which highlights the limited flexibility compared to other setups.
00:25:30.490
Another decision filing from Capybara’s longstanding default was the choice to use the Chrome browser with Selenium instead of Firefox. I argue that many developers conduct their work in Chrome. While I know many appreciate Firefox, it wasn’t reliable when I began working on system tests. I saw how frequently people complained about Firefox malfunctioning with Selenium.
00:26:12.690
As such, I opted for Chrome to avoid potential problems. You can easily switch to Firefox by simply altering the 'using' keyword argument in the driver settings. Note that the 'using' argument applies only to Selenium drivers. Other drivers function headlessly and don’t require modifications.
00:27:04.550
I hope in the future to facilitate support for additional browsers like Safari, and if anyone is seeking Rails contributions, I’m open to that. One of the optional arguments all drivers accept, except RackTest, is the 'screen size' parameter. This option defines the browser's maximum height and width, which is beneficial for testing your website's responsiveness across different layouts.
00:27:49.080
Another significant feature of system testing is the automatic screenshot taken when a test fails. This capability is fantastic for diagnosing failures, allowing you to see the project in its failure state. This feature works with all the supported drivers in Capybara, excluding RackTest. The Rails framework contains an 'after teardown' method that captures the screenshot during the test failure if the functionality is supported.
00:28:40.050
Let’s modify the earlier test we wrote to expect a failure and then execute the test. It will show all the server initialization, and upon failure, we’ll get a link to an image. As you can see, the text displayed differs from what we expected, which explains the test failure. This transparent feedback is invaluable in debugging processes.
00:29:21.270
Additionally, you can capture screenshots at any point during your test execution by invoking the 'take_screenshot' method. This can be highly useful for tools such as CI pipelines for comparing front-end changes or saving snapshots of your site's appearance at any given moment. One of the less obvious changes with system testing is the fact that Database Cleaner is no longer necessary.
00:30:33.470
Those familiar with Capybara know there were issues regarding database transactions during tests running with Capybara. Previously, the lack of proper rollbacks lead to a multitude of complications. Effectively, when a test began, Rails would start a transaction, and the web server often opened a second connection to the database in a competing thread. Consequently, changes on one thread were invisible to the other.
00:31:24.610
This issue frequently led to unique constraints failing and left over data during subsequent test runs. It took me a significant amount of time to analyze and address these database concerns while building system tests. The first step was understanding how Active Record interacted with the database in a multi-threaded environment.
00:32:09.730
Fortunately, I sought advice from Aaron Patterson and Matthew Draper, who understood Active Record's concurrency issues. Their differing opinions prompted me to attempt Aaron's approach, which involved having Active Record check and return connections once their usage was complete. However, this resulted in numerous failure points.
00:32:52.790
Findings prompted us to pursue Matthew's improved solution. It mandates every thread during a test to use the same database connection so that all database transactions and updates are consistently seen across threads. When a new server starts, it utilizes the existing connection rather than creating a new one, ensuring all data modifications get rolled back upon the transaction's conclusion.
00:33:41.700
Before you resonate a problem with this, rest assured, this approach applies only in test environments; it does not alter database integrity during production scenarios. Leveraging these capabilities meant there was no necessity for Database Cleaner, which is a great tool, yet it’s unnecessary when we expect records to rollback.
00:34:22.380
We spent a lot of time examining individual settings and public API for persistent tests. Now it’s time to understand the Rails plumbing underneath it all, which enables all of this functionality. None of this code should ever need to be manually adjusted; Rails is structured to alleviate you of configuration burdens, allowing you to focus on writing tests.
00:35:08.170
In Rails, system tests reside under the Action Dispatch namespace, right next to integration tests, which they inherit from, thus leveraging all of the URL helpers provided in the integration test framework. Because the entire class can't fit on this slide, I will cover the methods invoked by your existing application.
00:35:47.090
When you execute a system test, the start application method is invoked, booting the app and initiating the Puma server. Your Test Helper file then calls the driven_by method, where your default settings are configured. When called, this method instantiates a system testing driver class with the arguments you provide.
00:36:32.420
The browser, screen size, and options are established for the driver, where the browser argument is specifically utilized by Selenium. The running test is initialized safely and effectively, allowing multiple drivers to be used for each test suite according to individual test classes.
00:37:17.270
The use method is subsequently employed here, registering the respective driver. The register_selenium method is used to target the browser Chrome along with the passed options and screen size settings.
00:38:09.330
Through this concise plumbing network, testing in Rails has become relatively straightforward. Fortunately, none of you need to worry about constructing any of this setup, because the entire foundation is already in place.
00:38:43.800
One notable aspect of this feature's development was that it significantly deviated from building features for an organization. Open-source work remains public, as opposed to product-based development, which is usually kept confidential until launch.
00:39:31.210
In a project such as this, I found that while the bulk of my previous contributions pertained strictly to performance improvements, refactorings, or bug fixes, the introduction of a new testing framework prompts widespread smiles – with ample opinions flying around.
00:40:27.060
Once the pull request for system tests was opened, I quickly received 11 reviews and 161 comments. It highlighted one of the major challenges of open source. It's difficult not to feel as if you’re under constant judgment at times.
00:41:10.680
When you’re engaged in open-source work, you expose yourself. Every comment and commit feels subject to public scrutiny. This sensation can deter newcomers from participating in open source. I still experience an adrenaline rush when merging changes to the primary branch.
00:42:05.260
Occasionally, I find myself sweating over the build status after a merge, terrified of any post-merge breakdowns. I recently broke the Rails tests after a merge and was rather upset about it. After years in open source, I still feel vulnerable.
00:42:47.200
It’s challenging to maintain confidence while conducting work that is publicly visible. I often battle the urge to angrily abandon it all due to fatigue from repetitive debates regarding implementation, even when an alternative viewpoint is valid.
00:43:39.420
Open debate is an intrinsic part of open source, and you will continually have to advocate for your choices. Consequently, if your confidence falters, it can lead you to seek consensus with those reviewing your work.
00:44:18.520
However, when working with stakeholders who have differing investment levels in the end outcome, finding consensus can be challenging. When developing a feature for a company or client, you typically know the primary stakeholders involved.
00:45:16.160
In open source, you’re often unaware of who will become invested until you submit the pull request. I knew that the Rails team and Capybara contributors would be interested, but I did not anticipate the number of other interested parties.
00:45:59.090
While all this interest can be beneficial, generating constructive feedback from community members, it can quickly become overwhelming when you need to debate each individual's opinions on ideologies surrounding testing.
00:46:47.490
Differing opinions exist across the Rails and Capybara teams regarding which driver is best, whether it’s acceptable to change longstanding Capybara defaults from RackTest, and the necessity of including screenshot functionality.
00:47:34.850
Consequently, navigating expectations of diverse stakeholder opinions while constructing system tests has proven challenging. I recognized the importance of maintaining my sense of ownership while also respecting others' inputs.
00:48:10.800
Working to please all stakeholders by pursuing consensus leads to a lack of focus. The feature can become muddled, fostering a code style that deviates away from what feels important to me.
00:48:52.700
I continually reminded myself that we all share a common goal: to integrate system testing into Rails. Even though we may disagree on implementation specifics, finding that mutual understanding kept me grounded during the process.
00:49:37.030
One major takeaway from all of this is that managing expectations in open source is crucial. Ultimately, you can’t hold anyone other than yourself accountable, and the same rule applies to your contributions.
00:50:18.230
You’re the architect of your scope and thus must enforce any necessary rejections. Many suggested features for system tests await consideration, but had I pursued every idea before merging, it wouldn't have made its way into Rails.
00:51:00.530
I took it upon myself to manage the scope and expectations of those supporting the project while keeping it on budget. While I fully respected each individual's input, I was ultimately building the feature for Rails.
00:51:46.050
Consequently, these system tests needed to align with Rails’ aesthetic and philosophical framework, which dictated that the tool should facilitate beginners entering testing without unnecessary complexity.
00:52:21.050
In the end, the Rails principles should prevail over other opinions, maintaining the aim of making system testing as clean and productive as possible. Even if disagreement abounds, being receptive to suggestions improves the project for everyone.
00:53:10.890
I often find it challenging to detach from the work I’ve poured time into, but it’s an essential skill to realize that code no longer belongs to the original author; it belongs to the entire community of Rails users.
00:53:55.020
Being open to suggestions, modifications, and changes becomes paramount as the ecosystem continuously evolves through collaborative efforts. Open source thrives thanks to contributors.
00:54:38.790
As a final note, the open-source community does indeed function due to active participation. One simple example is how I merged the initial version of the system tests, despite it not being 100% stable understanding that contributors exist to assist in addressing the problems as they arise.
00:55:21.120
You cannot push an unfinished project to a client. In that scenario, your employer would likely dismiss you. In contrast, you can provide a foundation for others to enhance in open source.
00:56:03.920
The melding of my processes ensured to deliver a reliable foundational tool while welcoming contributions from my peers. After releasing Rails 5.1, I was amazed when it was quickly followed up with RC1 less than a month after combining contributions.
00:56:38.420
The majority of identified issues were resolved during that time, and I attribute that success to the community contributions. A notable contribution came from Kevin, a maintainer of Capybara who introduced additional assertions to enhance tests.
00:57:21.570
Users running system tests with MiniTest would see inconsistent numbers of assertions due to Capybara handling them differently related to RSpec. This was an issue that caused significant headache, and I take much joy that it’s resolved.
00:58:06.000
Overall, by successfully merging the initial code for system tests, we lay down a reliable structure that will evolve over time due to the supportive nature of contributors. With these efforts, the community stands to benefit from improvements made over time.
