Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
When talking about performance, most developers think application speed, faster algorithms or better data structures. But what about your test suite? CI time is developer waiting time! At Shopify we have more than 170,000 Ruby tests and we add 30,000 more annually. The sheer amount of tests and their growth requires some aggressive methods. We will illustrate some of our techniques including monitoring, test selection, timeouts and the 80/20 rule. If you have experience in writing tests and want to learn tricks on how to speed up your test suite, this talk is for you!
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
In the talk titled "Keeping Developers Happy with a Fast CI" at RubyConf 2021, Christian Bruckmayer, a member of the test infrastructure team at Shopify, explores best practices for optimizing continuous integration (CI) workflows, particularly with regards to managing an extensive test suite. With over 210,000 Ruby tests that take roughly 40 hours to execute, the team's focus is on reducing CI time to improve developer experience and efficiency. **Key Points Discussed:** - **Data-Driven Development:** Bruckmayer emphasizes the significance of data over intuition in driving development activities. The CI system at Shopify utilizes a pipeline that records detailed metrics such as build times and individual test durations which are then analyzed to identify bottlenecks and areas for improvement. - **The Scale of Shopify's Testing:** With the expectation to grow to approximately 270,000 tests next year and with 100 million test runs per day, scalability is crucial. The CI environment is run on Buildkite, which features scalable test agents that adapt to daily load changes. - **Test Optimization Techniques:** The speaker highlights the use of aggressive timeouts to prevent tests from hanging and ascending the average build duration. By implementing a timeout wrapper around tests, they managed to enhance CI run times significantly, halving the average build time with the deactivation of just 10 poorly performing tests. - **Efficient Test Selection:** Bruckmayer describes a strategy for selective test execution based on the changes detected in pull requests. By running only relevant tests, they optimize CI time while also improving overall system stability and reducing the chances of flaky tests affecting results. - **Test Prioritization:** The focus on executing tests that yield the highest assurance of catching failures first was detailed. By analyzing historical test performance metrics, tests that had failed previously were prioritized to run first, enhancing the speed at which critical failures are detected. **Conclusion:** Bruckmayer concludes that evolving a CI system to prioritize data-driven insights leads to substantial improvements in efficiency. The team's philosophy treats CI as a production-level system. By collecting and analyzing data regarding their CI operations, they have continually refined their processes, ultimately ensuring better developer experiences and faster feedback on code changes. The presentation not only delves into the technical workings of this system but also illustrates the outstanding benefits of practical adjustments to testing approaches at scale. This talk is an insightful resource for developers looking to enhance their testing frameworks and CI pipelines.
Suggest modifications
Cancel
