Test Framework
It's not your test framework, it's you
Edit
#behavior-driven-development-bdd
#cucumber
#test-driven-development-tdd
#capybara
#integration-testing
#test-framework
#software-quality
Summarized using AI

It's not your test framework, it's you

Robbie Clutton and Matt Parker • February 21, 2013 • Burbank, CA

In the talk "It's Not Your Test Framework, It's You" at LA RubyConf 2013, speakers Matt Parker and Robbie Clutton examine the challenges and backlash surrounding Behavior-Driven Development (BDD) methodologies and the test frameworks associated with them. They argue that the root issues affecting developers' experiences with BDD are often rooted not in the frameworks themselves but the misuse and misunderstanding of how to implement them effectively.

Key Points Discussed:
- The BDD Hype Cycle: The speakers discuss the rapid adoption of BDD, followed by criticism regarding its sustainability, particularly focusing on brittle tests, inefficiencies, and poor communication between developers and product owners.
- Cargo Culting: They highlight the dangers of adopting methodologies without fully comprehending them, which often leads to ineffective practices.
- Understanding BDD's Origin: The talk traces the history of testing, from unit testing to Dan North's introduction of BDD, noting how it was meant to enhance communication about requirements by defining clear features and scenarios.
- Collaboration with Product Owners: Importance is placed on conversations with product owners, ensuring all requirements and scenarios are understood, which helps prevent common pitfalls like miscommunication and misunderstanding of intent.
- Common Problems: Issues like brittle tests, flickering tests, and slow build times are identified as major pain points developers face while using BDD tools like Cucumber.
- Good Practices in BDD: The speakers encourage developers to write acceptance tests that focus on desired outcomes rather than just testing steps, advocating for a transition to domain-specific languages (DSL) for clarity.
- Improving Test Suites: They suggest re-evaluating test strategies, consolidating redundant tests, and employing journey testing to streamline build times without sacrificing quality.
- Maintaining Tests as Living Documentation: The importance of maintaining readability in tests, making them accessible, and ensuring they serve as effective documentation is emphasized.

In conclusion, Parker and Clutton assert that developers must take responsibility for their use of BDD and testing frameworks and ensure that tests are treated with the same care as production code. They advocate for thoughtful, effective practices to alleviate the challenges associated with BDD and improve the overall quality of software development.

It's not your test framework, it's you
Robbie Clutton and Matt Parker • February 21, 2013 • Burbank, CA

The BDD hype cycle is over. Recently, there's been a lot of backlash against popular BDD libraries like Cucumber. Some developers blame their test frameworks for brittle test suites and long build times. Others go so far as to claim that acceptance testing is simply not sustainable, period. In this talk, we'll do some root cause analysis of this phenomenon with shocking results - it's not the test framework, it's not the methodology, it's you. You've abused your test framework, you've cargo-culted the methodology, and now you're feeling the pain. We'll show you a way out of the mess you've made. We'll discuss the main problems BDD was intended to solve. We'll show you how to groom your test suite into journey, functional, integration, and unit tests in order to address build times. We'll teach how to mitigate against brittleness and flickers, and how to let your tests reveal the intent of the application and actually become the executable documentation we've been waiting for.

Help us caption & translate this video!

http://amara.org/v/FGdO/

LA RubyConf 2013

00:00:23.670 Hi, I'm Matt. Hi, I'm Robbie. We want to talk to you today about sustainable BDD practices.
00:00:30.340 This talk was originally titled 'It's not your test framework, it's you,' but we thought that made us sound like jerks, blaming you for something.
00:00:35.679 And while we are blaming you, we didn't want to come off as jerks right off the bat, so we decided on a new title.
00:00:41.739 We've really noticed this hype cycle concerning BDD. Initially, there was a very aggressive adoption curve, with people getting on board and excited, lots of tools emerging.
00:00:48.670 Then, interest started to wane, and people became critical of it, starting to blame the tools and questioning the necessity of automated acceptance testing.
00:00:54.969 We want to discuss some of the reasons why people have stopped doing BDD and explore what's really going wrong.
00:01:01.059 Some recurring themes include brittle tests, developers writing unreadable acceptance criteria, flickering tests, and slow test execution times.
00:01:11.530 Before we delve deeper, we need to ensure that everyone is on the same page regarding our discussion. When I first started doing BDD, I had no idea what I was doing.
00:01:22.719 I was just reading blog posts, exploring tools, and trying things out without understanding the deeper intentions behind them.
00:01:28.170 This phenomenon is often referred to as 'cargo culting,' a term you can look up on Wikipedia if you're not familiar.
00:01:36.700 Essentially, it means adopting or copying a methodology, process, or technology without truly understanding it.
00:01:42.630 You've just integrated it into your development environment, trying to run it without asking the right questions or doing it the correct way.
00:01:49.210 Before diving into our current state, let's step back and review how we got into this mess, starting with unit testing and the small talk community's development of it, leading to the creation of various tools.
00:02:06.399 Dan North coined the term 'Behavior Driven Development' (BDD) and proposed a different perspective by flipping our approach on its head.
00:02:30.310 We were still working within a test-driven development (TDD) mindset, while our collaborative practices with product owners were evolving through a methodology called 'Specification by Example.'
00:02:42.090 However, our tooling did not align with our workflow, creating a disconnect we're keen to address.
00:02:56.500 Let's consider an example: as a product manager, I might say, 'Let’s build this web app, we need to implement sign-in.'
00:03:02.130 But that's not the right approach. Instead, we should ask for a specific example that clarifies the requirements.
00:03:10.180 Through discussion with the product manager, we could outline the steps a user would take, like entering a username, password, and email to sign in.
00:03:40.900 This leads us to define a valid set of credentials as well as scenarios for when inputs are invalid or the username isn’t available, ultimately uncovering additional features, such as account recovery.
00:04:07.870 Having these discussions is imperative, as they shape our understanding of the robustness versus speed of our process.
00:04:13.600 If we lack this communication, we may assume the product owner's priorities without confirming their desires.
00:04:19.530 Let's examine several potential use cases: what happens when the password confirmation doesn't match, the user provides an invalid email, or the username is unavailable.
00:04:42.069 Each of these points requires careful consideration, as they lead to multiple scenarios, such as account recovery processes or handling forgotten credentials.
00:05:01.720 The importance of these discussions is to ensure we cover every possible scenario, as the implications can be significant.
00:05:07.270 After discussing a seemingly simple feature request, we end up with four features and numerous scenarios, revealing the complexity that often hides beneath the surface.
00:05:27.820 Dan North's approach wasn't a whim; it arose from a need to clarify hidden complexities, ensuring that developers understood what their product owners truly valued.
00:05:42.090 Yet, at the time, the right tools to support this process were lacking, leading Dan North to create 'JBehave,' one of the first successful BDD tools.
00:05:54.909 JBehave encouraged discussions around important terminology such as 'features' and 'scenarios,' fostering a common language between developers and product owners.
00:06:18.729 Dan North later began working with Ruby, and this led to the development of 'RSpec,' which became a significant moment for BDD, allowing stories to be articulated in plain text files.
00:06:31.539 This provided a less intimidating environment for product owners to collaborate with developers, as it was not hidden in code.
00:06:46.460 Then, 'Cucumber' arose, contributing to the same goals, equipped with a specification language—Gherkin—which helps outline application states and transitions.
00:07:10.000 However, many users adopted Cucumber and Gherkin without fully understanding when and how to use it, missing the opportunity to discuss complexity.
00:07:31.090 The beauty of Gherkin lies in its ability to engage everyone in considering their application as a state machine, which fosters comprehension and dialogue.
00:07:49.990 With this common vocabulary, project teams can integrate their requirements into code seamlessly, enhancing collaboration further amidst different domain concerns.
00:08:05.850 When done effectively, BDD and TDD collectively help create living documentation that exemplifies and validates application functionality.
00:08:16.520 Examples of tools that took inspiration from these frameworks include Spinach, Turnip, and others, illustrating a variety of approaches yielding different results.
00:08:23.200 The cautionary tale here is about asking the right questions; if we start blaming our tools without understanding our approach, we risk a cyclical problem.
00:08:41.560 As developers, we must identify why the pain exists when using BDD: is it brittleness in tests or an inherent misunderstanding of our practices?
00:08:48.590 Brittleness manifests when changes occur in the product that ripple through the test suite, leading to a string of failing tests that can feel daunting.
00:09:04.850 Brittle tests do not clarify intent; they articulate how something operates instead of what it should accomplish.
00:09:18.940 Imagine being on a team building Twitter: if the product owner requests a 'tweet feature,' you might hastily derive tests without contemplating the broader implications or the necessary scenarios.
00:09:51.130 Testing those features can often revolve around the user interface instead of focusing on the desired outcomes, creating a muddled approach.
00:10:03.920 We risk duplicating knowledge in our test base, leading to a brittle structure rife with inefficiencies and challenges in maintaining clarity when requirements evolve.
00:10:17.840 One of the critical lessons is that our framework doesn’t guarantee good outcomes. It’s up to us to apply it properly to achieve a sustainable practice.
00:10:41.270 When writing tests, we should emerge from a mindset focused purely on testing frameworks to one shaped by domain knowledge.
00:10:48.840 By utilizing a domain-specific language (DSL), you can gain clarity and focus, abstracting unnecessary complexities behind clear and concise user stories.
00:11:00.860 Instead, let's write about the outcomes we desire and the behaviors we wish to observe, rather than the intricate pathways that may lead there.
00:11:10.190 It's essential that our acceptance tests, while granular, remain relevant to the user features while also being maintainable.
00:11:20.290 We need to be cautious of not collaborating closely enough with our product owners while defining acceptance criteria.
00:11:36.820 The miscommunication may stem from a lack of understanding about the purpose of BDD, which includes supporting collaboration with customers.
00:11:47.150 Sometimes product owners may become annoyed if they perceive your new testing framework as imposing constraints on their workflow.
00:12:03.690 Moreover, if they are not engaged in writing clear testable requirements, they’ll struggle to align their stories with your framework.
00:12:12.030 As developers, we need to build systems of helper methods that facilitate the inclusion of product owner language into our tests.
00:12:25.940 This way, we can simplify acceptance testing and allow for a seamless flow between the user language and the technical requirements.
00:12:34.490 We should emphasize documentation but keep in mind that living executable documentation is ineffective if no one is reading it.
00:12:50.900 One of the issues leading to unread tests is their unreadability; if they are difficult to comprehend, developers might not revisit them once completed.
00:13:08.370 Another reason is if the tests aren’t easily visible or accessible; exposing them helps bridge the understanding gap.
00:13:19.730 Using tools like Cucumber can allow us to format our output as HTML, and publish it on platforms like Relish for better visibility and collaboration.
00:13:37.790 Consider utilizing repositories such as Wally from the BBC if you have security concerns but still want to ensure documentation aligns with tests.
00:14:01.370 However, something to note is that better documentation doesn't always equate to a need if you’re not at a point where you actively require it.
00:14:15.340 As we explore the issues we've faced, the biggest pain point remains the slow execution of tests.
00:14:25.370 For instance, if your tests take eighteen minutes to run, it's unlikely you’ll run them frequently or before pushing updates.
00:14:41.040 In delaying execution, developers start relying more on continuous integration, with all the inherent pitfalls that can lead to.
00:14:52.540 Moreover, slow tests can encourage practices like adding sleep statements, which signal a developer has surrendered to fragility within the tests.
00:15:10.200 Capybara has previously offered solutions for dealing with asynchronous issues, but it requires developers to be diligent with their assertions.
00:15:25.950 In the recent updates, the polling mechanism has improved, minimizing the risk of introducing unnecessary sleep into the system.
00:15:36.760 With recent improvements, you can often upgrade Capybara and remove some of the sleeps while witnessing an eventual decrease in test execution time.
00:15:50.370 If flickering tests remain an issue even after adjusting sleep times, a solution could be to quarantine those tests, tagging them accordingly.
00:16:03.120 With a targeted approach, you can redesign your testing pipeline to effectively address these flickering tests and better manage your overall testing strategy.
00:16:19.810 You may need to designate a 'build nanny' within your team to oversee investigations into flickering tests to enhance accountability.
00:16:35.670 If they can’t resolve the issues, it may signal a need to rethink, rewrite, or even delete those tests to prevent further project drag.
00:16:50.340 Adam Milligan, a senior engineer at Pivotal, discussed how we should not fear deleting tests that are no longer providing value.
00:17:03.930 We should recognize when tests convert into project burdens and prioritize the overall health of our codebase instead.
00:17:19.480 After tackling time-consuming test suites, an essential question arises: Why are we testing the same sequences over and over again within our application?
00:17:33.090 You might achieve greater efficiency by establishing a single comprehensive test for functionality instead of repeating login tests throughout the suite.
00:17:44.290 Journey testing, focusing on user flows and critical paths, should encompass the core functionality, while most tests should lean more towards white-box testing.
00:18:00.370 This approach allows for quick access to necessary states without extraneous navigation through the application interface.
00:18:12.470 By efficiently isolating critical functionality, we can significantly reduce build times.
00:18:24.050 After streamlining the application tests, we can aim for a target of fewer than five minutes for the entire suite.
00:18:36.490 Continuous improvement should remain the goal; you should never allow test times to creep back up.
00:18:47.030 In summary, identifying and addressing the pain points is crucial; if your testing framework causes suffering, thoughtfully investigate why.
00:19:01.200 Remember that well-maintained tests should be treated as equal citizens within your codebase, deserving the same care as production code.
00:19:12.960 Lastly, embrace BDD wholeheartedly. Put in the effort required for every step of the process, thereby reaping the rewards of your work.
00:19:22.470 Thank you very much for your attention.
RC
Robbie Clutton
3 talks
MP
Matt Parker
2 talks
Explore all talks recorded at LA RubyConf 2013
+6