Effective Test Generation for Combinatorial Decision Coverage

Complex software systems present significant challenges to existing software testing techniques. Simply applying exhaustive testing will lead to the execution of a prohibitively large number of test cases. Furthermore, many testing techniques today provide neither promising coverage achievement nor reliable fault detection strength. In this paper, we propose a technique, which represents an innovative synthesis of combinatorial testing and symbolic execution, to generate test cases based on a novel coverage criterion, namely combinatorial decision coverage (CDC). Strength t (or t-way) CDC requires each t-tuple of decision outcomes to be executed by at least one test case. Given a program, our CDC-based technique first uses a revised version of a symbolic executor, S2E, to collect all program decisions with symbolic variables as well as their corresponding constraints and then applies a combinatorial test generation tool, ACTS, to generate t-way combinations for the outcomes of these decisions. A test case can be generated with respect to each combination that represents a single path-condition of the program. Case studies were conducted on three versions of photo editing applications. Our results indicate that a test set generated using the proposed technique has higher statement, decision, and all use coverage as well as better fault detection strength than a test set of the same size generated by random testing and genetic algorithm-based test generation.