We know that a very simple product or system can generate a vast number of potential test cases. With more complex systems, this number becomes astronomical. This is the result of a factorial calculation! One technique we use to get around this problem originates with designed experiments. Many designed experiments are based on orthogonal arrays of test values. Two-level testing for each input and output is often represented with a matrix of zeros and ones or of pluses and minuses to indicate the levels. More levels are possible, but increasing the levels increases the number of test cases exponentially. With digital inputs, two levels are adequate.

            In essence, we will stimulate the inputs according to the recipe from any given row of our matrix and observe the behavior of the outputs. This situation means we must understand what the correct outputs are and be able to measure them. Furthermore, for each set of specified inputs, we must also understand which behaviors we expect to see—any deviation from these is an anomaly that prudence necessitates must be explored.

            One alternative to the orthogonal array is pairwise testing, perhaps best exemplified by James Bach’s gratis Allpairs program. Allpairs takes input from a test file that is itself generated most commonly from a spreadsheet and creates a sequence of test cases that will exercise every pair. The amazing thing about Allpairs is that it holds up fairly well against commercial pairwise programs while remaining completely free! The pairwise approach has some defects, for example:

• It will only see two-way issues
• Interactions are generally invisible
• It is not complete

Even with the issues that pairwise testing exhibits, we consider it to be useful if for no other reason than it provides us with a rational basis for more test cases.