Extending a Flakiness Score for System-Level Tests

Extending a Flakiness Score for System-Level Tests

연구 분야: Software Development

학회: IFIP International Conference on Testing Software and Systems

Flaky tests (i.e. automated tests with a non-deterministic test outcome) undermine the trustworthiness of today’s DevOps build-pipelines, and recent research has investigated ways to detect or even remove flaky tests. In contrast, others proclaim that test engineers should “Assume all Tests Are Flaky” because, in today’s build-pipelines, one can never fully control all components of the system under test. Test engineers then capture the randomness of test results via what is called a flakiness score. In this paper, we extend an existing flakiness score to deal with system-level tests. We illustrate, via simulated test outcomes, how this refined score can support three different strategies for dealing with flaky tests—(i) Rerun, (ii) Fix and (iii) Monitor.