Front-end Testing for Skeptics

Talks

Luke Francl

@look

#javascript

#capybara

#test-driven-development

#web-development

Front-end Testing for Skeptics

by Luke Francl

The video titled 'Front-end Testing for Skeptics' features speaker Luke Francl discussing the importance and evolution of front-end testing, particularly for JavaScript applications. Initially a skeptic of front-end testing, Francl shares his journey towards adopting effective front-end testing methods that integrate JavaScript functionality.

Key Points:

- Definition of Front-end Testing: Francl defines front-end testing as full-stack integration testing, which includes JavaScript operations within applications. By the end of his talk, he aims for attendees to understand how to set up fast, reliable front-end tests using simple Ruby.

- Historical Context: The presentation contrasts past practices of avoiding JavaScript and relying on server-side logic for testing, which was prevalent when JavaScript was unreliable. With JavaScript's advances, particularly in AJAX and Web 2.0, there's a growing necessity for effective testing on the client-side.

- Challenges Encountered: Francl recounts his experiences at SwiftType, where the need for robust JavaScript testing became apparent as their JS widget was deployed frequently across many platforms. He notes how challenging it was to ensure the code functioned well under diverse conditions.

- Modern Tools: Francl introduces three significant tools to aid in JavaScript testing:

- Capybara: A framework for front-end acceptance testing.

- PhantomJS: A headless WebKit browser that starts quickly, ideal for continuous integration.

- Poltergeist: A driver that connects Capybara with PhantomJS, enabling smoother interaction.

- Practical Example: Francl demonstrates a sample app that showcases basic user interactions, including auto-completion using a popular library. He walks through setup instructions and shares testing strategies to identify JavaScript errors effectively.

- Debugging and Best Practices: Tips for utilizing debugging features within these tools to identify and resolve issues are highlighted. He discusses the importance of testing both positive and negative cases to ensure comprehensive validation and security.

- Future Considerations: Francl expresses interest in exploring JavaScript unit testing frameworks and perceptual diffs for UI consistency. These tools help in spotting minor changes between expected and actual UI results.

Conclusions:

While Francl remains a skeptic about fully embracing front-end testing culture, he acknowledges the invaluable role these modern tools play in enhancing his workflow. He emphasizes that, as JavaScript becomes more integral to applications, implementing thorough front-end testing is crucial to maintaining the reliability and functionality of web applications.

00:00:16.400 Hello, welcome! My name is Luke Francl, and I am a front-end testing skeptic. This is the story of how I learned to tolerate front-end testing, and you can follow along with my journey.

00:00:27.119 When I talk about front-end testing, I'm referring to full-stack integration testing, including running the JavaScript that is likely part of your application. By the end of this talk, you'll be able to set up pure Ruby front-end tests that are fast and reliable most of the time and can become a valuable part of your workflow.

00:00:40.399 When I started doing front-end development, testing JavaScript looked a little bit like this. Naturally, we tried to avoid using JavaScript as much as possible. We aimed to push as much logic onto the server and test it there, where we could be confident our code would actually work.

00:01:02.320 This approach had a corollary in developing native GUI applications, where you would have models, views, and controllers, and you would try to keep your view as a thin layer on the top of your application, since with most native toolkits, that was just pixels. Automating this part reliably was challenging.

00:01:20.880 During my research for this talk, I revisited the C2 Wiki to explore their views on GUI testing. After going through numerous pages, I found nothing satisfactory. Below that level, however, you could have system tests that cover the rest of the application and unit tests to test your models, which were reliable, repeatable, and automatable.

00:01:38.400 When we began developing web applications, we faced a similar situation with our models and controllers on the server-side. While we controlled that layer, the HTML, CSS, and JavaScript on top were hard to test. However, we had this wonderful abstraction of HTTP, allowing full-stack tests using HTTP to simulate a web browser, as browsers communicate at that level. Additionally, unlike classical GUI applications, HTML, CSS, and JavaScript are just text, making it possible to examine and ensure they match our expectations.

00:02:22.160 Since I came from a background of avoiding JavaScript, I never really saw the cost-benefit analysis favoring front-end testing. I preferred to keep testing at the HTTP level and below. However, as Paul Graham noted regarding AJAX and Web 2.0, 'JavaScript works now.' Over the last five to six years, JavaScript has become a crucial part of our applications, to the point where entire applications are primarily written in JavaScript.

00:02:40.079 This shift has opened up fantastic possibilities within our applications. Still, the old testing methods need to catch up. For me, two main factors caused me to rethink my position and find a good way to test the front end. The first was working at a startup called SwiftType.

00:03:02.079 SwiftType provides search as a service, and besides our developer API that allows you to index and search content via a REST API, we also have a consumer product. This product gives you a JavaScript embed to enhance your search box and deliver a modern search experience, all through client-side JavaScript.

00:03:16.080 Here's a demo of how that works from my blog. You can see how when you type into the search box, you get autocomplete results. If you click on one of those results, it takes you directly to it, and you can also receive results in a traditional list format. This functionality is all produced through third-party JavaScript.

00:03:50.560 If you're interested in implementing third-party JavaScript, I highly recommend checking out a book written by two engineers from Disqus, a commenting system you may be familiar with. However, one of the challenges of working with third-party JavaScript is that you execute in an environment you don't control. The page you are in can set variables that may interfere with your code. Additionally, you need to ensure you aren't exporting global variables that could disrupt their content.

00:04:01.840 It's a very constrained environment, which complicates development. As we expanded our product, we started placing this JavaScript on tens of thousands of web pages, hundreds of thousands of times a day, so it had to work flawlessly. The graph that scares me the most is the total number of search engines utilizing this service, as I’m relied upon to answer support requests about it. If we break this JavaScript, feedback would be rapid and painful for me, so I needed to ensure it operated without issues.

00:04:14.560 Addressing specific problems led us to develop an improved mobile experience. We designed a solution that, if a user is on an iPhone or Android phone, and using our JavaScript embed, provides mobile-optimized search results. While doing this, we also wanted to clean up some code duplication and refactor for better efficiency. However, it was essential that it continued to function correctly.

00:04:32.479 Testing this without properly running JavaScript would be incredibly tough, if not impossible. We used to deploy updates to important sites and ensure everything was operating as expected, which could be quite tedious. The second factor that prompted me to reconsider client-side JavaScript testing is that the tools have dramatically improved. When I began exploring JavaScript automation, the state of the art was Selenium, which recorded your clicks and played them back through a browser.

00:05:23.040 This was a slow, painful process that fell victim to constant breaking if any webpage changes occurred. Today, I will discuss three tools that can help us with JavaScript testing. The first is Capybara, which is a framework for front-end acceptance testing, providing a DSL for DOM interactions. The second tool is PhantomJS, a headless WebKit browser, unlike the older version of Selenium.

00:05:41.760 PhantomJS starts quickly and does not require a frame buffer, making it easier to install on CI servers. The third tool is Poltergeist, which utilizes the PhantomJS driver for Capybara, enabling Ruby code to connect the two. I will not be covering Cucumber, as I don’t fully understand it. If you find Cucumber useful and successful, by all means, use it. However, it's not necessary for testing front-end code.

00:06:06.560 To demonstrate these techniques, I created a sample app available for download on GitHub. This app allows you to type in the names of countries and provides a list of results. It utilizes Twitter's Typeahead.js library, which is an impressive autocomplete library. When you type in countries, it auto-completes them, and if you select one, you are taken to the corresponding country page.

00:06:38.240 To begin, you'll need to install PhantomJS. It's easy with Homebrew: running 'brew install phantomjs' takes just a few seconds. Next, you'll want to add Capybara and Poltergeist to your Gemfile. In this example, I’ll be using RSpec, but if you prefer MiniTest or another framework, you can use it. Here’s the URL for the GitHub repo.

00:07:04.560 Finally, you need to instruct Capybara to use Poltergeist by registering the driver name 'poltergeist.' In this setup, I have an option for JavaScript errors set to true, which will cause tests to fail if any JavaScript errors are encountered. The second option enables the web inspector, allowing you to connect and debug your tests live as they run.

00:07:32.480 The last thing I set is the JavaScript driver to Poltergeist. This setting is necessary because Capybara defaults to not running JavaScript using RackTest, which is faster. However, if your tests necessitate executing JavaScript, you must enable the JavaScript driver.

00:08:04.320 Now, what is the simplest possible test that will demonstrate value in your application? The test consists of visiting a path on your website and filling out a field on the page. The test will throw an error if a JavaScript error occurs. This functionality is significant, as you may have deployed to production and overlooked a crucial page with a JavaScript error or left a console log present. This test will catch any such mistakes.

00:08:19.920 One last detail: I'm using the 'js' option set to true, as previously mentioned. As I registered Poltergeist as the JavaScript driver earlier, passing 'js: true' enables the use of JavaScript in this test. Before we dive deeper, let’s examine the Capybara DSL and some functionalities available.

00:08:52.320 First, you can visit a page in your application using either a string URL or Rails routes, which is quite handy. You can also click on things, whether links or buttons, by their name, text, or ID. While using the ID is often better for long-term stability, since it is less prone to change, this method ensures that if the text of 'Login' were modified to 'Sign In,' the test wouldn't break if the ID remains the same.

00:09:14.959 You can also fill in forms, such as using this example where I populate the field labeled 'username' with 'railsconf.' The next two lines demonstrate how to assert that elements exist on the page using a CSS selector, along with how to find those elements using a CSS selector to interact with them. This feature of Capybara is particularly valuable, as it allows for retrying continuously until the tested condition is met or a timeout is reached.

00:09:39.679 The importance of this lies in testing asynchronous JavaScript, where your DOM may be altered while your test is running since actions like clicking a link result in server interactions that change the page. Furthermore, you can evaluate JavaScript within PhantomJS and return the result to Ruby. For instance, I can evaluate whether the JavaScript expression that the number one equals the string 'one,' which is true, although one of JavaScript's more confusing features.

00:10:17.679 Next, I will discuss various tests related to the sample app I previously mentioned. For context in understanding these tests, let’s take a look at the DOM structure on that page. The input field for the country is the base HTML, and by integrating the Twitter Typeahead library, the rest of the necessary elements are generated.

00:10:44.320 Inside the Twitter Typeahead span, you will find the dropdown menu, which includes 'tt dataset' for countries and, beneath that, 'tt suggestions' which contains individual suggestions. For example, if I start typing 'Af,' Afghanistan is one of the suggestions. Conversely, if I enter text that doesn't match any countries, the 'tt suggestions' will return no children.

00:11:08.000 In a slightly more complex test, we will visit the home page and fill the country field with 'norw,' short for Norway. I will explain one section of this test shortly, but after clicking on the body, I will locate the first suggestion and proceed to click on it. As discussed, doing so should redirect us to the Norway page.

00:11:44.320 As I run the test, I assert that the current path should equal the country path for Norway. Regarding the earlier section that I mentioned, this part seems like a hack, and I'll clarify why I needed it. While developing the sample app, I experienced a test failure, stating that the element 'tt suggestion' was not found since there were no suggestions available. I was typing in 'Norway,' and PhantomJS couldn’t retrieve any suggestions.

00:12:02.880 Debugging can often be tricky, especially for JavaScript tests, but Poltergeist and PhantomJS simplify the process with a feature called 'page.driver.debug.' This feature launches the web inspector, just like how the 'inspector true' setting does. This way, you can navigate the DOM of the running test, similar to using a console with IRB in Ruby.

00:12:24.960 In this video, you can observe how I figured out what went wrong with that failed test. Like Chrome or Safari's web inspector, you can inspect the DOM, and within it, you can identify the Twitter Typeahead span. There's the dropdown menu present, but it's not visible because it was set to display: none. This situation likely stems from the Typeahead library using event triggers like click or keyup, but my method of populating the field didn’t trigger the required events.

00:12:52.640 Thus, I used a workaround by clicking the body, which caused the dropdown menu to appear. The debugging tools available make addressing issues like this significantly easier. Additionally, it’s crucial to test negative cases, as individuals often conduct tests solely on the happy path. For instance, if I input 'zzz,' which matches no countries, I need to validate that no suggestions are generated.

00:13:22.880 To check that there are no suggestions, I employ the evaluate script feature, combining JavaScript to assert that 'tt suggestions' has no children. This evaluation reinforces the power of integrating JavaScript functionalities into your tests.

00:13:39.200 At SwiftType, we also strive to prevent global variable leaks. Our RSpec matcher tests for this by checking for unintentional global variable exposures. Essentially, we maintain a string containing a JavaScript function to inspect window properties, capturing those names in 'before globals.' After executing our proc, we analyze the array of names again, comparing the results and illuminating any disallowed globals.

00:14:03.360 Moreover, this matcher includes an 'allowed' argument, which permits specifying expected properties to be present in the window object. The tests will only pass if there are no unexpected global variables present, helping maintain good coding practices.

00:14:32.880 An additional noteworthy aspect of testing involves cross-site scripting protection. While Rails implements security features to defend against cross-site scripting attacks, it’s possible that if user-generated content is displayed, vulnerabilities may appear. For testing, I create a document where I intentionally insert a script tag. The test visits that document and verifies that the title’s content matches expectations, ensuring proper escaping of potentially harmful scripts.

00:15:01.919 The second check evaluates the value of 'window.xss,' which should not be set. Maintaining secure applications is paramount, and even with existing best practices, it's essential to remain vigilant against any new vulnerabilities introduced as the application evolves.

00:15:21.919 To summarize this segment, even if you're not injecting third-party JavaScript across a multitude of sites, you likely have an application flow that employs JavaScript—perhaps through user login or account creation processes that validate usernames. Conducting tests on these critical functionalities is necessary; otherwise, if broken code is deployed to production, significant issues can arise.

00:15:43.439 Being able to test those essential flows ensures that you aren't caught off guard when assessing any declines in user uptake on your platform. To conclude, you should prioritize testing crucial application parts. I’m not suggesting configuring JavaScript tests for every single aspect of your site, but the key functionalities warrant attention.

00:16:03.679 During a conference earlier this month in Portland, I witnessed a remarkable presentation where the speaker's computer crashed mid-talk. Surprisingly, the speaker maintained composure, using a laser pointer skillfully while conveying engaging material. Could I have handled that situation as adeptly?

00:16:27.760 Next, I’d like to share some of the challenges we encountered with these technologies. Although the upsides are apparent, there are several nuances to navigate. First, Capybara will wait for certain elements on the page to change, which works far better now than it did previously.

00:16:47.680 In earlier versions of Capybara, as we loaded our JavaScript, we often faced challenges where we had to manually establish wait conditions. Waiting for various elements to appear led to unfavorable solutions such as embedding sleep commands, which have largely been addressed in Capybara 2.

00:17:07.920 However, one lingering issue arises when evaluating scripts, as that type of operation doesn’t interact with Capybara's native synchronization process. Kyle Vanderbeek wrote a blog post detailing how to synchronize JavaScript for Capybara with custom helpers that implement retry logic to evaluate scripts until conditions are met or timeouts are reached.

00:17:38.240 The second notable challenge involves PhantomJS crashing, which occasionally leads to tests failing without any clear reason. Although far from being a dealbreaker, it happens about one percent of the time. We often encounter these failures and shrug it off as 'PhantomJS-related issues.'

00:18:04.160 Finally, we need to acknowledge transactions and database setup. We utilize MongoDB, which is not designed for transactions in the traditional sense. The issue arises because Capybara creates separate threads for running tests, allowing them to interact with the live version of your Rails app and connect to your real test database.

00:18:34.960 With relational databases, Rails rolls back after each test, but we used Database Cleaner, which wiped the database clean before every test. This led to some tests failing at random times during execution depending on their order in our test suite since the database state may have changed as tests were running.

00:19:07.680 Fortunately, RSpec provides a seed for randomizing test order, allowing me to recreate test failures and trace them back. After some investigation, I deduced that the issue derived from Capybara launching a server and communicating with the same test database simultaneously. We remedied this by running our standard Rails unit tests and controller tests first, followed by other Capybara tests that require JavaScript.

00:19:43.599 This sequencing prevents interference between tests, and I appreciate how straightforward that fix was. Now, let's take a look at some prospective future explorations in JavaScript testing.

00:20:03.520 While we are not currently using JavaScript unit testing frameworks like Jasmine, QUnit, or Mocha, these tools have caught my attention. Each of these frameworks requires loading tests in a browser for execution, and although Jasmine has an updated test runner, we seek to utilize our existing headless browser capabilities.

00:20:28.560 Matthew O'Riordan authored a compelling blog post that outlines how he successfully integrated automated JavaScript testing with headless browsers—an exemplary resource. His article not only outlines the transition path but also discusses hurdles he encountered, which helped in his implementation of tests, allowing for individualized feedback on JavaScript issues.

00:20:52.560 Soon thereafter, I learned about the Teabag gem, which consolidates many of the steps detailed in Matthew’s post. This gem simplifies the setup process significantly; you merely install the gem and get comprehensive test coverage with ease. The integration of testing JavaScript files through the asset pipeline makes the process even more seamless.

00:21:16.120 One additional area I find compelling is perceptual diffs. This testing approach allows developers to visualize differences in UI components, enabling them to catch minor discrepancies before they become major issues. As demonstrated with two rectangles, perceptual diffs pinpoint even the smallest variations, ensuring UI consistency is maintained during deployment.

00:21:38.840 Brett Slatkin’s team at Google utilizes perceptual diffs as part of their automated testing frameworks. With these tests, they capture the differences between expected and actual UIs, facilitating improved decision-making prior to production release, hence maintaining a high level of quality.

00:22:05.120 In a recent example, a chart failed a visual inspection, but perceptual testing highlighted the precise areas of concern. Integrating this methodology would help us ensure our JavaScript updates don’t compromise the user experience.

00:22:36.000 Overall, we’ve covered the necessity of front-end testing as applications integrate more JavaScript, shifting more logic to the client-side. With Capybara, PhantomJS, and Poltergeist, we can make front-end testing not only more feasible but also less burdensome; the example app illustrated various practical debug scenarios.

00:22:58.000 Ultimately, I may not become a total enthusiast for front-end testing, but thanks to these tools, they have forged an invaluable place in my workflow, providing assurance that SwiftType's JavaScript operates smoothly. Thank you for your attention!

00:37:20.160 You.

RailsConf 2013