Suggest modification to this talk

Title

Description

RailsConf 2019 - Fixing Flaky Tests Like a Detective by Sonja Peterson
_______________________________________________________________________________________________
Cloud 66 - Pain Free Rails Deployments
Cloud 66 for Rails acts like your in-house DevOps team to build, deploy and maintain your Rails applications on any cloud or server.

Get $100 Cloud 66 Free Credits with the code: RailsConf-19
($100 Cloud 66 Free Credits, for the new user only, valid till 31st December 2019)

Link to the website: https://cloud66.com/rails?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
Link to sign up: https://app.cloud66.com/users/sign_in?utm_source=-&utm_medium=-&utm_campaign=RailsConf19
_______________________________________________________________________________________________
Every test suite has them: a few tests that usually pass but sometimes mysteriously fail when run on the same code. Since they can’t be reliably replicated, they can be tough to fix. The good news is there’s a set of usual suspects that cause them: test order, async code, time, sorting and randomness. While walking through examples of each type, I’ll show you methods for identifying a culprit that range from capturing screenshots to traveling through time. You’ll leave with the skills to fix any flaky test fast, and with strategies for monitoring and improving your test suite's reliability overall.

Date

Summarized using AI?

If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.

Show "Summarized using AI" badge on summary page

Summary

Markdown supported

In the talk "Fixing Flaky Tests Like a Detective," presented by Sonja Peterson at RailsConf 2019, the speaker delves into the pervasive issue of flaky tests in software development. Flaky tests are those that pass sometimes and fail at other times without any changes to the underlying code. Sonja emphasizes the importance of not only fixing these by identifying their root causes but also preventing them from being introduced in the first place. She shares a structured approach that parallels detective work to diagnose and resolve flaky tests efficiently.

Key Points Discussed:

- **Challenges of Flaky Tests**: They can significantly impede the development process, leading to wasted time and lost trust in automated tests.
- **Categories of Flaky Tests**: Sonja identifies typical culprits behind flaky tests:
  - **Async Code**: Tests influenced by the order of asynchronous events, particularly in feature tests using Capybara.
  - **Order Dependency**: This arises when tests' outcomes change based on the state influenced by previous tests, emphasizing the need to isolate test states.
  - **Time Issues**: Tests failing due to date and time calculations, potentially fixed by using libraries like Timecop.
  - **Unordered Collections**: Ensure that database queries have predictable outcomes; ordering results can avoid flaky failures.
  - **Randomness**: Reducing reliance on randomness in tests increases reliability.
- **Information Gathering**: Before attempting fixes, developers should gather data on flaky tests such as error messages, timing of failures, and the running order of tests. This can be managed using a bug tracking system.
- **The Detective Method**: Adopt a systematic method similar to investigating a crime: gather evidence, identify suspects, form hypotheses, and test fixes rather than relying on trial and error.
- **Team Collaboration**: The collective responsibility is emphasized to handle flaky tests. Designated individuals should monitor and fix these tests while ensuring continuous communication within the team.

Conclusions and Takeaways:
- Flaky tests are an inherent part of a developer's journey, but they can also be opportunities for growth and learning. Fixing them leads to improved understanding of both code and testing frameworks.
- It is vital to maintain a healthy test suite with a focus on high reliability and effectiveness, which ultimately leads to better software quality and development speed. The overarching message is that addressing flaky tests is a valuable investment in the stability of software development processes.

Cancel