Abstract

This talk reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy as the acceptance metric. A reference point to current best practice is provided by using hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way adequate test sets are not adequate as measured by the mutants kill rate compared with hand generated test sets of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.