Pairwise Testing - Orthogonal Array Explained

• Example 1
  • Conventional Method
  • Orthogonal Array Method
  • Rules of writing an orthogonal array
• Applying orthogonal array rules to Example 1
  • Applying Rule 1
  • Applying Rule 2
  • Applying Rule 3
  • Test combinations derived with orthogonal array method
• Orthogonal Array - Naming Conventions
  • Sample 1
  • Sample 2
• Example 2
  • Step 1
  • Step 2

Pairwise Testing is a black-box testing technique used in situations where the interaction of the factors cannot be so easily determined, or even when there is no interaction of the factors.

For example, consider the testing of a web application that has four features (factors) say OS, speed, browser types and anti-virus software and each of these factors has five values (options).

In order to have full coverage for the application, we need to test 5 x 5 x 5 x 5 = 625 combinations, which is not easily possible.

We may test all these combinations, provided we have enough time and budget. However, what if we do not have enough time and budget?

Pairwise testing is the technique we can use in such a situation.

In pairwise testing, we make sure that all possible pairs of parameter values are covered by at least one test. However, not all possible higher-order combinations such as triples quadruple and so forth will be covered.

There are two basic models in pairwise testing

• Orthogonal Array
• All-pair Table

Both are tables that tell us which particular options to include in a given test configuration. The tables are guaranteed to contain all options for every factor at least once, and every pair of options across all pairs of factors.

Let us discuss more on “Orthogonal Array”.

Before going into the details of the orthogonal array, let us write down the test case by applying the conventional/exhaustive testing technique.

Example 1

Consider a function having three factors X, Y and Z and each factor having two options as 0 and 1.

Now, let us list down all the possible combinations without considering the orthogonal array

Conventional Method

Here, as you can see there are 8 test cases/test combinations

Let us move on to the details of the orthogonal array and finally will derive the test cases using the orthogonal array technique for the same example.

Orthogonal Array Method

Definition of orthogonal array: A two-dimensional array constructed with special mathematical properties such that choosing any two columns in the array provides every pair combination of each number in the array.

If we explain it with Example 1, we need to consider the following.

• Consider the factors X & Y and ensure that the combinations of X & Y are covered.
• Consider the factors X & Z and ensure that the combinations of X & Z are covered.
• Consider the factors Y & Z to ensure that all pairs of combinations are covered.

Orthogonal array testing is a systematic way of testing all-pair combinations of variables using orthogonal arrays. It significantly reduces the number of all combinations of variables to test all pair combinations

Rules of writing an orthogonal array

Let us get to know the rules of writing an orthogonal array

There are three rules for selecting an orthogonal array

1. There must be at least as many columns as factors. i.e. Number of columns = Number of factors
2. There must be at least enough numbers in the columns to hold the options for each factor. i.e. If you have two options for a factor, then test cases should be there for both the values and if there are three options for a factor, then test cases should be there for all the three values. If we have spare numbers that don’t map to any option for that factor can be represented by a tilde (~).
3. There must be at least as many rows or test cases as the product of the two largest numbers of options. For example, if one factor has 4 options and another has 3 options and yet another has 2 options, then 4 x 3 = 12 row.

Applying orthogonal array rules to Example 1

Now, let us apply these rules and the definitions of the orthogonal array to example 1 and try to find the tests involved.

Applying Rule 1

Consider the two factors X & Y. Now list down all possible combinations for the pair which will be 0 0, 0 1, 1 0 and 1 1

Applying Rule 2

Consider the factors X & Z and list down all possible combinations for the pair and here notice to shuffle the order while writing the pairs i.e. 0 0, 0 1, 1 1 and 1 0. Note that here we changed the order of the values of Z for the 3rd and 4th pair.

We can explain this in another way.

• Write down all the values of X similar to how it is written for the pairs X & Y.
• For the first two pairs, write values of Z as same as Y in pairs of X & Y.
• To cover all pair combinations of Y & Z, interchange the values of Z in the 3rd and 4th pairs as highlighted in the table below.

Applying Rule 3

Now let us consider the factors Y & Z. Here note that all the combinations for the pairs are now listed. If we had not changed the order for the pairs X & Z, all combinations would not have been covered for pairs Y & Z.

Now we have to write down the values of Y from the X & Y factor table and values of Z from the X & Z factor table. If we combine this we will get the order of combinations for Y & Z

Test combinations derived with orthogonal array method

Now the table below shows the test cases or test combinations developed using the orthogonal array method.

Note that there are only four test cases after applying the orthogonal array technique.

Thus the number of test cases by using the orthogonal array technique is reduced to four while in conventional technique the number of test cases is eight.

I just saved 4 test cases by using the orthogonal technique. Did I miss something? I should have considered the other 4 too.

Someone from your head.

This might be what you are thinking when you derived test cases for example 1. However, if we consider a large example wherein testing all combinations is impossible then we will come to know the strength of the orthogonal array technique.

Orthogonal Array - Naming Conventions

Now, let us move on to the naming convention followed in orthogonal arrays

Sample 1

Consider an orthogonal array having 4 factors and each factor having 3 values each.

No: of test cases as per the rules of orthogonal array = 3 x 3 = 9

Sample 2

Consider an orthogonal array having six factors and these six factors are having 3 options each and yet another factor having 6 options.

No: of test cases = 3 x 6 = 18

Example 2

Company A has designed a web page with three distinct sections (Top, middle, and bottom). These sections can be individually shown or hidden by the user. The browsers on which the webpage can be viewed are internet explorer and Firefox. The server used for launching the application can be IIS, Apache and web logic. Write down the number of test cases required by using the orthogonal array technique.

Step 1

First lets list down the factors and the options for each factor.

In this scenario, we have 4 factors.

• Section
• Visibility
• Browser
• Server

Options for these factors are as below.

Step 2

Apply the rules of the orthogonal array technique and derive the number of test cases.

The above orthogonal array is having 4 factors, out of which two factors are having 2 options each and the other two factors are having 3 options each.

So, No: of test cases = 3 x 3 = 9