Development of interaction test suite generation strategy with input-output mapping supports

,


INTRODUCTION
Software can fail in many unexpected ways (Kimoto et al., 2008;Maity and Nayak, 2005).Thus, in order to ensure quality and conformance to specifications, there is a need to exhaustively test them.Yet, exhaustive testing is practically impossible.Addressing this issue, many test case design strategies have been developed in the literature (for example boundary value analysis, equivalence partitioning, decision tables and random testing (Basili and Selby, 1987;Beer and Mohacsi, 2008;Kuhn et al., 2004;Reid, 1997) to help sample out test data into manageable ones.Although useful, these strategies do not sufficiently cater for faults due to interaction.For this reason, t-way strategies have started to emerge.Numerous efficient t-way testing strategies have been proposed in the past literatures (Ahmed et al., 2011;Baowen et al., 2003;Jangbok et al., 2007;Klaib et investigate the usefulness of automated mapping based on input-output relationship as well as its flexible iteration control for constructing t-way test suite.Benchmarking results demonstrate that AURA strategy gives competitive results against most existing strategies.The rest of the paper is structured as follows: Subsequently the mathematical background on interaction testing is given after which the study discusses the recent related work on input-output based relationship interaction testing; whereas, the development of AURA strategy is further illustrated and clarified.Then, we compare and discuss our results with other existing strategies.Lastly, we conclude our work.

MATHEMATICAL BACKGROUND
Mathematically, interaction testing can be abstracted to a covering array (CA).CA is a combinatorial object that been extensively used to generate interaction test cases in software systems when all factors (parameters) have equal number of levels (options or values).A covering array, CA (N; t, k, v), is an array with N rows and k columns that satisfies the criteria that each t-tuple occurs at least once within these rows (Cemal et al., 2006;Myra, 2003).When N is unknown or unspecified, the notation CA (t, k, v) can be used (that is t is interaction strength, k is the number of factors and v is the number of options associated with each factor).For covering array, the value of v is the same for all k.Meanwhile, mixed-level covering array is a generalization of covering array that allows for different alphabet sizes for different rows.The mixed-level covering array is denoted as MCA (N; t, k, (v 1 , v 2 , …, v k )), an N x k array on v symbols (Bryce and Colbourn, 2007;Yan and Jian, 2006), where , with the following properties: 2) The rows of each N x t sub-array cover all t-tuples of values from the t columns at least once.
A shorthand notation can be used to describe MCA (also for CA, VCA and IOR) by combining the same v i 's and representing this number as a superscript (Yan and Jian, 2006).For instance, three v i 's each with two options is written as 2 3 .In this manner, an MCA (N; t, k, (v 1 , v 2 , …, v k )) can also be written as an MCA (N; t, (s 1 p1 , s 2 p2 , …, s r pr )) where covering array, of strength t containing C, a vector of covering arrays each of strength greater than t and defined on a subset of the k columns.Ordering of the columns in the representation of a VCA is important since the columns of the covering arrays in C are listed consecutively from left to right (Myra et al., 2003b;Ziyuan et al., 2008).Unlike CA, MCA and VCA, input-output based relationship covering array needs not generate test cases to cover all t-way interactions but only required to cover all actual interactions.This covering array can be denoted as IOR (N; (v 1 , v 2 , …, v k ), R), an N x k mixed level covering array which covers interaction relationship, R, of a typical software SUT.R is consisted of w number of interaction coverage requirement, r, which specified the actual interactions for that SUT and is defined as } ,..., , { 2 1 w r r r R = (Patrick and Bogdan, 2000;Wang et al., 2007).Each r indicates a set of inputs (factors) that are interacting and is constitute to a specified interaction coverage requirement.

RELATED WORK
In the last 15 years, many t-way strategies have been proposed in the literature including automatic efficient test generator (AETG) (Cohen et al., 1997), pairwise independent combinatorial testing (PICT) (Czerwonka, 2006), in parameter order (IPO) and its variants (Lei et al., 2007;Reussner et al., 2005;Younis and Zamli, 2010;Yu and Kuo-Chung, 1998;Yu et al., 2008), genetic algorithm (GA) (McCaffrey, 2009a), simulated bee colony algorithm (SBC) (McCaffrey, 2009b), simulated annealing (SA) (Myra et al., 2003b) and ant colony system (ACS) (Xiang et al., 2009).All aforementioned strategies are found useful and become the pioneers in tway and variable strength interaction testing.Moreover, a comprehensive survey of interaction testing has been published by Nie and Leung (2011) recently.However, in line of the scope of this paper, the further discussions shall be drawn on the recent works in input-output based relationship interaction testing.Considering the support of input-output based relationship interaction testing, much useful effort is also emerging.Patrick (2001) proposed the model of input-output based relationship testing method and gave three different test generation algorithms (Patrick, 2001;Patrick and Bogdan, 2000;Patrick et al., 2002) to solve the problem of test cases generation for software with complex input-output relationship.Their first approach implemented a brute force algorithm to explore all possible combinations of test to discover the correct minimal test suite (Patrick and Bogdan, 2000).Although straightforward, brute force algorithm tends to consume time especially involving large test data.Next, they proposed Union algorithm (Patrick, 2001) by generating a serial of test suite (that is, sets of test cases) for output variables to cover the interaction that is corresponding to their associative inputs variables, and then taking the union of them to obtain a final test suite.
The implementation of the Union algorithm is straightforward with low time complexity but generally not producing optimal test suite.Lastly, Patrick et al. (2002) proposed Greedy algorithm.The algorithm works by selecting an unused test case that covers the greatest number of uncovered combinations of input values each times until all interactions have been covered by the selected test suite.It indeed generate a much smaller test suite than Union algorithm, but with a bad time and space performance since this method has to check all test cases in a huge search space.To overcome this constraint, Cheng et al. (2003) implemented a problem reduction method which is based on color graph in their later work.This method only shows significant gain in efficiency (both time and space performance) when the number of colors used is small relative to the number of nodes in a complete graph.In other words, this reduction method is merely applicable for simple relationships between the input parameters based on their occurrences in the output parameters.Later on, Wang et al. (2007) analyzed and improved Union algorithm (Patrick, 2001).They suggested that all the positions corresponding to each "do not care" factor shall not be assigned until a coverage requirement which include that factor is dealt.With this perception, the improved Union algorithm (termed as ReqOrder) generated better results in term of test suite size reduction as compared with previous work.Despite of this, they also implemented input-output interaction testing by adopting in-parameterorder strategy (Yu and Kuo-Chung, 1998), which is known as ParaOrder (Wang et al., 2007;Ziyuan et al., 2008).For this strategy, an initial test suite will be constructed for a sub-system with small number of factors.The system is then extended by adding a new factor to get a test suite for the new sub-system.The extending process is repeated until all factors have been added into that system.As far as the input-output interaction test suite size is concerned, ParaOrder gave comparable results against Greedy algorithm and better test suites than Union algorithm and ReqOrder.
Meanwhile, Ziyuan et al. (2008) utilized one-test-at-atime strategy (Cohen et al., 1997) and density concept (Patrick et al., 2002) to generate test suite.This approach is generally consumed longer computational time against others (Union, ReqOrder and ParaOrder) although it mostly produced better test suite than them.In addition, there is an interaction testing tool; test vector generator (TVG) (Arshem, 2009) which is capable to generate test suite based on input-output relationship, uniform strength and variable strength t-way coverage as well as random manner.However, its implementation details are unknown.While most of the strategies produce test cases in symbolic values, automatically maps these test cases back into actual data form is therefore another practical feature shall be taken into account.A recent strategy, GTWay (Zamli et al., 2011) started to address this automated mapping need.At a glance, GTWay implemented a preprocessing automated mapping system by employing Parser algorithm to capture the actual values from the fault file and map them into symbolic representations before they can be used for t-  way test cases generation.Upon completion of test suite generation, GTWay remaps these symbolic data representations into actual data values.Nevertheless, GTWay consumed a significant portion of execution time to read and convert the actual data from the fault file and this lead to an overhead penalty incurred during the preprocessing mapping process.

OVERVIEW OF AURA STRATEGY
As input-output based relationship interaction testing is more Ong and Zamli 3421 flexible to accommodate the actual interactions for a typical SUT, we have opted to develop AURA strategy which supports inputoutput interaction test suite generation.Moreover, the urge of automated input-output mapping support in the sense of generating actual value test suites encourages AURA strategy to adopt this mapping automation feature as well.Throughout this section, the development of AURA strategy with input-output mapping supports will be highlighted accordingly.Basically, this strategy is composed of three algorithms: "interaction pair generation algorithm, test suite construction algorithm and actual data mapping algorithm".The overview of the AURA strategy has been summarized in Figure 1.
Referring back to Figure 1, a software tester clarifies and summarizes the inputs (in terms of actual data form) into a look-up table for typical software SUT.In order to generate intended test suite, the software tester then keys in the symbolic values of inputs into AURA strategy.Based on these inputs, 'interaction pair generation' algorithm will then be triggered to generate all possible interaction pairs, which are needed for successive operations.
After that, 'test suite construction' algorithm starts to construct test suite by exploiting the interaction pairs that are previously generated.Upon completion, the results are loaded into 'actual data mapping' algorithm for generating final test suite (in actual data form) as specified at the predefined look-up table file.Lastly, Figure 2 shows a typical look-up table that AURA strategy adopts.

Interaction pair generation algorithm
In AURA strategy, test suite generation is based on its corresponding interaction pairs.The interaction pairs are used to ensure the completeness of interaction coverage for specified coverage requirements.Hence, the details on generating the interaction pairs shall be discussed here.In order to generate interaction pairs, AURA strategy first needs to enumerate the corresponding parameter interactions groups.As mentioned earlier, AURA strategy is developed to support input-output based relationship, uniform strength and variable strength interaction testing.Therefore, one would ask how the AURA strategy enumerates the parameter interactions groups and also the interaction pairs with these different kinds of considerations.Indeed, AURA strategy generalizes and converts these parameter interactions groups into binary representations (that is, 0 and 1 s).After the conversion, for each parameter interactions group, 1 s (in binary form) is representing the parameters that involved in the parameter interactions whereas 0 s (in binary form) are the complementary parameters of 1 s.It is noted that parameter involves in parameter interactions is termed as interaction element throughout this paper.As illustration, consider the input parameters as shown in Figure 2. In fact, these inputs are summarized as symbolic data (Table 1) which AURA strategy adopts.Supposed that we intent to generate an input-output based relationship interaction test suite that comprised of 2 coverage requirements, R = (ABC, DE).As far as the parameter interactions groups are concerned, AURA strategy will generalize these as 11100 (for ABC) and 00011 (for DE) respectively.The similar mechanisms can be applied to both uniform and variable strength interaction testing as well.For instance, assumed that CA (2, 2 5 ) and VCA (2, 2 5 CA (3, 2 3 )) in this case.For all of these, the parameter interactions groups and their subsequent binary representations are summarized in Table 2.
Upon generation of aforementioned parameter interactions groups in binary representations, 'interaction pair generation' algorithm proceeds to generate all possible interaction pairs.To do so, for each parameter interactions group, exhaustive combinations will be formed within its interaction elements whereas the do not care ('X') values are assigned on the corresponding non interacting elements.Table 3 depicts the resultant interaction pairs for ABC and DE and Figure 3 shows the pseudo-code of 'interaction pair generation' algorithm.As could be seen in Figure 3, AURA strategy adopts dynamic partitioned base data structure, IE, in Interaction Pair Generation algorithm to hold the generated interaction pairs since it offers systematic and well organized space search rather than the unpartitioned base data structure, in which every search always begin from the first data of the bulky irregular data structure.This data structure is termed as dynamic since the number of partition for data structure relies on the number of parameter interactions group involved.In this example, AURA strategy used 2 partitioned data structure to store the resultant interaction pairs accordingly.

Test suite construction algorithm
First of all, AURA strategy is implemented based on one-test-at-a time basis (Ziyuan et al., 2008).In this basis, the test suite generation process begins with an empty test suite.Besides that, the interaction pairs sets that corresponding to a SUT input specifications are generated.Then, test cases are generated and added into that empty test suite one by one, until all interaction pairs are covered.Meanwhile, AURA strategy also gives nondeterministic output since the random selections are used to construct each test case.For this reason, this strategy not always produce similar test suite for every run; though, the generated test suite size is compromised.In addition, AURA strategy user can decide on the number of iterations, n.With this customizable looping system, if the user chooses a larger value of n for typical input, then there is higher chance for AURA strategy to give more optimized test suite.This is due to the fact that, by having more iterations (corresponding to higher value of n) on assigning additional set of random values and checking corresponding uncovered interaction pairs (in IE) process, AURA strategy gets higher possibility to obtain the test case which have more uncovered interaction pairs (in IE).Though, AURA strategy consumes extra computational time for these.In other side of coin, AURA strategy generates less optimized test suites with smaller n value but less execution time.With the aforementioned considerations, AURA strategy summons 'test suite construction' algorithm to generate test suite once the interaction pairs are generated.Figure 4 illustrates the pseudo-code of 'test suite construction' algorithm.In order to generate test suite, a test case, α is proposed by forming the first combination of exhaustive combinations (for interaction elements) and assigning random values to those non interacting elements from the first group of parameter interactions.As assigning random values are nondeterministic in nature, one may also get different kind of results for each proposed α.Then, based on IE as well, AURA strategy subsequently verifies the number of interaction pairs, β, which could be covered by α.
Next, α is promptly sent to final test suite, τ if only if it covered all of its corresponding interaction pairs (indicating an optimized test case has been found).Otherwise, AURA strategy will reassign another set of random values and check for its covering interaction pairs.As long as the optimized test case has not been discovered, AURA strategy will repeat the assigning and checking process for n (number of iterations) times.Among the n test cases been proposed, AURA strategy will select a test case greedily (that is, with the greatest number of uncovered interaction pairs in IE).The selected test case is then added into τ.The corresponding β in IE are then eliminated before the system proceeds to propose next α.The τ is considered completely formed as all interaction pairs in IE are covered, indicating all interaction coverage from coverage requirements are included.

Actual data mapping algorithm
As existing strategies are focused on test suite minimization efforts; therefore, less intention has been drawn into the input-output mapping automation supports to map the test data from symbolic value to actual data form.Consequently, test engineers will have to manually perform the mapping process before the test suite can be used for testing execution phase or else some automated mapping efforts are still be required.In this scenario, AURA strategy has incorporated with this automated mapping concern as far as the test suite practicality is concerned.Specifically, the automated mapping support in AURA strategy is known as post-processing mapping system.In this case, the term of post-processing refers to the test cases generation process is prior to the automated input-output mapping process, which allows symbolic inputs to be executed on AURA strategy (instead of real values) and generates real value outputs via a predefined look-up table.It is noted that the look-up table is by test engineers before they specify inputs (in symbolic values) to AURA strategy.Thus, this feature enhances the mapping automation support while maintaining the ease of data manipulation in test suite construction.In order to implement the post-processing automated input-output mapping system, AURA strategy employed Actual Data Mapping algorithm to support both symbolic values as well as actual data output generation.Figure 5 depicting the pseudo-code of 'actual data mapping' algorithm.In this case, AURA strategy generates symbolic values results which are similar output format as some other strategies possessed.Moreover, with this feature, AURA strategy is also capable to map back the test cases with their actual data based on the corresponding look-up table.
To accomplish this aim, the actual data information from the look-up table is stored in σ as shown in Figure 5.Then, for the first test case in the τ, AURA strategy maps that test case, which is in symbolic values back into actual data form based on σ.After that, the test case (in actual data form) is then written into an output file, 'ofile'.This mapping process is repeated until all test cases in the final test suite are converted.For instance illustration, Figure 6 shows a typical constructed test suite in terms of symbolic values and also the conversion from that to actual data based on our previous example.

RESULTS AND DISCUSSION
Here, we evaluate AURA strategy with current benchmarking input-output based relationship interaction testing strategies.As mentioned earlier, AURA strategy can support three distinctive types of interaction testing: input-output based relationship interaction testing, uniform and variable strength interaction testing.Therefore, three experiments are conducted accordingly.For the first experiment, 2 set of factors are chosen to represent the systems with fixed-level factors and mixed-level factors respectively in order to evaluate the effectiveness of AURA strategy on input-output based relationship interaction testing.The fixed-level factors are consisted of 10 factors (parameters) where each factor have 3 levels (options) and defined as F 1 = (3 10 ) whereas mixed-level factors are 3 factors of 2 options, 3 factors of 3 options, 3 factors of 4 options and a factor with 5 options and denoted as F 2 = (2 3 × 3 3 × 4 3 × 5).AURA strategy is then required to generate test cases for these factor sets based on predefined input-output interactions.The input-output interactions are built by selecting coverage requirements from a pool of coverage requirements in Wang et al. (2007); they are also attached in appendix of this paper.Based on these, there are 6 iterations in the experiment for each factor set.The first 10 coverage requirements are included for the first iteration and defined as |R| = 10.The input-output interactions are then been added with another 10 following coverage requirements for consecutive iteration.Tables 4 and 5 have shown the generated test suites size from AURA strategy as well as other published strategies.Meanwhile, there are eight different input specifications (S 1 to S 8 ) of uniform strength interaction testing that commonly applied in other published strategies.Hence, AURA strategy is executed based on these and gave the test suite sizes as summarized in Table 6.As far as the benchmarking is concerned, existing variable strength interaction testing strategies tend to refer on the input specifications proposed by Cohen et al. (1997) and Myra et al. (2003b).Therefore, AURA strategy adopted these as the inputs for the last experiment.
GA, ACA, GA-N, IPO-N, IPO, Jenny, SA and ACS) are collected from published articles in Myra et al. (2003b), Xiang et al. (2009) and Ziyuan et al. (2008).Since ACS strategy (Xiang et al., 2009) possesses randomness algorithm, and its results are obtained based on 200 iterations; hence, AURA strategy also performed 200 iterations throughout the experiments.As AURA strategy is non-deterministic in nature, 20 independent runs have been performed and the best results (with minimal test cases) of these runs are reported for each input.Lastly, no fair comparison for test cases generation time is able to be performed due to the differences or unspecified of computing environments in the published literatures.Thus, this aspect is excluded in our discussion.Based on Tables 4 and 5, 'density' produced smallest test suites for almost all inputs.However, AURA strategy also produced considerable results in this case.For instance, AURA strategy gave optimized test suite at |R| = 10 for F 2 = (2 3 x 3 3 x 4 3 x 5) and produced better results than ParaOrder as well as TVG in some other cases.Particularly, AURA strategy always generates smaller test suites than Union for each input.By referring to Table 6, none of the reported strategies is accomplished to give the best results for all input configurations of uniform strength interaction testing.In addition, we discovered that there is no big gap in terms of test suites sizes constructed by AURA strategy with some classic algorithms.For some inputs, AURA strategy even produced better output than Density, ParaOrder, PICT, GA-N, IPO and Jenny.Meanwhile, AURA strategy generated smaller size than TVG for most of the inputs as well.Despite of the test suite optimality has been concerned, most of the existing uniform strength interaction testing strategies (AETG, GA, ACA, GA-N, IPO-N, IPO and Jenny) do not address capability of input-output interaction testing as well as variable strength interaction testing.Concerning variable strength interaction testing test suites as shown in Table 7, simulated annealing (SA) strategy is often effective in constructing small test suite size but it does not support input-output interaction testing.Besides that, it is worth to mention that AURA strategy frequently performed well against PICT as far as the size is concerned.In addition, AURA strategy is comparable in certain cases as compared to Density, ParaOrder and TVG, by generating smaller test suites.Moreover, there comprised of four input specifications at which AURA strategy produced optimized results that as optimized as SA strategy.They are namely: CA (3, 5 3 ), CA (3, 4 3 ) + CA (3, 5 3 ) and CA (3, 5 1 6 2 ) from VCA [m; 2, 4 ).Apart from supporting all forms of t-way testing possibilities (for example uniform strength, variable strength, and inputoutput based relation) and generating competitive test size, the main contribution of AURA strategy is the fact that it provides seamlessly integration of input-output mapping of actual data values as part of the strategy itself.Comparatively, AURA strategy can be put side-byside with GTWay (Zamli et al., 2011).GTWay takes the preprocessing approach through the use of fault file to address its input-output mapping of actual data.Although useful, this preprocessing approach introduces some timing overhead to parse the actual parameter values and to directly manipulate actual data for test generation.Particularly, it is reported that the overhead penalty incurred in GTWay is directly proportional to the number of defined base test cases (approximately 50 us per additional defined test).Unlike GTWay, AURA strategy adopts post-processing approach, that is, the mapping to actual values is achieved after generation via look-up tables.Thus, no time is wasted on loading fault file and parsing parameter.In this case, AURA strategy provides better mapping automation feature than GTWay.
Furthermore, unlike existing strategies, the iterations of assigning additional set of random values and checking corresponding uncovered interaction pairs in AURA strategy can be controlled (that is the number of iterations is customizable).On one hand, when the optimality of test suite size is preferred than generation time, AURA strategy can increase the iterations looping number in order to generate more optimized test suite.On the other hand, AURA strategy can decrease the iterations looping to get fast test suite construction.This notable customization is useful to provide the flexibility for the users to decide their Sci.Res.Essays preference (either the optimality of test suite or fast generation time).

CONCLUSIONS
This paper has discussed the development of a flexible input-output based t-way strategy called, AURA.Experimental data shown AURA strategy generally produces competitive test suites for all these inputs.Moreover, AURA strategy that implemented the postprocessing automation on converting back test suites into actual value form has alleviated the burden of software tester from mapping the test suites manually.In future work, more practical features such as seeding and constraint implementations shall be incorporated into AURA strategy to further enhance its reliability and usability.

Figure 2 .
Figure 2. A typical look-up table.

Figure 3 .
Figure 3. Pseudo-code of interaction pair generation algorithm.

Figure 4 .
Figure 4. Pseudo-code of test suite construction algorithm.

Figure 5 .
Figure 5. Pseudo-code of actual data mapping algorithm.

Figure 6 .
Figure 6.Conversion from symbolic values to actual data for a typical test suite.

Table 2 .
Summary of parameter interactions groups.

Table 3 .
The resultant interaction pairs for coverage requirements ABC and DE.

Table 4 .
Comparison of different strategies for F1 = {3 10 } and different size of R.

Table 7 .
Sizes of variable strength interaction testing test suites.