Pairwise Coverage

Pairwise Coverage (ISTQB CTAL-TA): What It Is, How It Works and How to Answer Exam Questions

Introduction
Pairwise coverage belongs to combinatorial testing, one of the data-based test techniques in the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus. It is used when a test object has several input parameters, configuration options or environment settings, each with several possible values. Testing every possible combination is usually impossible, so pairwise testing gives a systematic, measurable and much smaller set of tests that still finds most interaction defects.

Why Pairwise Coverage Is Important
1. Combinatorial explosion: The number of full combinations is the product of the number of values of every parameter. For example, 10 parameters with 4 values each give 4^10 = 1,048,576 combinations. Pairwise testing can often cover every pair of values in roughly 25 to 30 tests.
2. Defect evidence: Industry studies, notably by NIST, show that most field failures are triggered by a single parameter value or by the interaction of two parameter values. Fewer failures need three or more parameters to interact. Pairwise testing therefore targets the place where most interaction defects occur.
3. A measurable exit criterion: Pairwise coverage gives a clear percentage that can be reported, compared and used as a completion criterion. This helps the Test Analyst justify how thorough the tests are.
4. Efficiency and risk-based testing: It offers a good balance between cost and thoroughness. It can be strengthened (3-wise, n-wise) for high-risk areas or weakened (each choice) for low-risk areas.
5. Typical uses: configuration testing (operating system, browser, database, language), product options and feature toggles, forms with many fields, and system or API parameters.

What Pairwise Coverage Is
In combinatorial testing, the test object is modelled as a set of parameters, and each parameter has a finite set of values. The values are often equivalence partitions or boundary values chosen beforehand.
- A pair, also called a 2-tuple, is a combination of one value of one parameter with one value of a different parameter. An example is (Browser=Chrome, OS=Linux).
- Coverage items for pairwise testing are all the feasible pairs of parameter values.
- Pairwise coverage is calculated as:
Pairwise coverage = (number of distinct pairs exercised by the tests / total number of feasible pairs) x 100%
- One test case is a complete assignment of a value to every parameter. Each test case covers several pairs at the same time. With n parameters, one test covers n(n-1)/2 pairs. This is why pairwise testing is efficient.

The Family of Combinatorial Coverage Levels
- Each choice coverage (1-wise): every value of every parameter appears in at least one test. The minimum number of tests equals the largest number of values of any single parameter.
- Pairwise coverage (2-wise): every pair of values from any two parameters appears in at least one test.
- n-wise coverage (for example 3-wise): every combination of values of any n parameters appears in at least one test.
- Full (exhaustive) combinatorial coverage: every combination of all parameters is tested. The number of tests equals the product of all value counts.
Subsumption: Full coverage subsumes n-wise coverage. n-wise subsumes (n-1)-wise. Pairwise subsumes each choice. So 100% pairwise coverage always gives 100% each choice coverage, but the reverse is not true.

How It Works: Step by Step
Step 1: Identify parameters and values. Use the test basis to decide which inputs or configuration items interact. Reduce each parameter to a small set of representative values, for example by using equivalence partitioning.
Step 2: Identify constraints. Some combinations are impossible. For example, Safari may not run on Linux in the model. Infeasible pairs are removed from the coverage items and must not appear in tests.
Step 3: Count the coverage items. For every pair of parameters, multiply their value counts. Then add the results together and subtract any infeasible pairs.
Step 4: Build the test set. Use a covering array, an orthogonal array, a tool (such as PICT, ACTS or Hexawise), or work it out by hand for small models. Choose tests so that each new test covers as many still-uncovered pairs as possible.
Step 5: Measure coverage. Check which pairs are covered, calculate the percentage and add tests for any missing pairs.
Step 6: Add expected results. Combinatorial techniques produce input combinations only. Expected results must come from the test basis or a test oracle.

Worked Example
Parameters:
- A (Payment): a1 = Card, a2 = PayPal. That is 2 values.
- B (Browser): b1 = Chrome, b2 = Firefox, b3 = Edge. That is 3 values.
- C (Language): c1 = EN, c2 = DE. That is 2 values.

Full combinations: 2 x 3 x 2 = 12 tests.
Total pairs: A-B = 2x3 = 6, A-C = 2x2 = 4, B-C = 3x2 = 6. The total is 16 pairs.
Lower bound on the number of tests: At least 6 tests are needed. One reason is that the largest single pair group (A-B or B-C) has 6 pairs, and each test covers only one pair from each group. Another reason is that 16 pairs divided by 3 pairs per test, rounded up, also gives 6.

A pairwise test set with 6 tests:
T1: a1, b1, c1
T2: a2, b1, c2
T3: a1, b2, c2
T4: a2, b2, c1
T5: a1, b3, c1
T6: a2, b3, c2

Check:
- A-B pairs: a1b1, a2b1, a1b2, a2b2, a1b3, a2b3. All 6 are covered.
- A-C pairs: a1c1, a2c2, a1c2, a2c1. All 4 are covered.
- B-C pairs: b1c1, b1c2, b2c2, b2c1, b3c1, b3c2. All 6 are covered.
Result: 100% pairwise coverage with 6 tests instead of 12.

Partial coverage example: If only T1 to T4 were run, the uncovered pairs would be a1b3, a2b3, b3c1 and b3c2. That means 12 of 16 pairs are covered, so coverage = 12/16 = 75%.

Constraint example: Suppose PayPal (a2) is not supported on Edge (b3). Then the pair a2b3 is infeasible. The coverage items drop to 15, and T6 must be changed, for example to a1, b3, c2.

Strengths and Limitations
Strengths:
- It drastically reduces the number of tests.
- It systematically detects defects caused by single values and by two-way interactions.
- Its coverage is measurable.
- It is well supported by tools.
Limitations:
- It can miss defects that need three or more specific values together.
- Its results depend on choosing the right parameters and values.
- It does not provide expected results.
- Constraints must be modelled correctly.
- It is not suitable when the order of inputs matters. For sequences, use state transition testing instead.

Exam Tips: Answering Questions on Pairwise Coverage
1. Know the formula exactly. Coverage = covered feasible pairs / total feasible pairs x 100%. Questions often ask for a percentage after showing a partial test table.
2. Count total pairs correctly. Multiply the value counts for every pair of parameters and add them. With 4 parameters there are 6 parameter pairs: AB, AC, AD, BC, BD and CD. A common trap is to multiply all values together, which gives the full combinations, not the pairs.
3. Find missing pairs methodically. Write out each parameter-pair group as a small grid. Tick off the pairs that each test covers. The empty cells are the answer.
4. Minimum number of tests. For pairwise testing, the minimum is at least the product of the two largest value counts. For each choice testing, the minimum is the largest single value count. For full coverage, it is the product of all value counts. Exam options often include all three numbers as distractors.
5. Remember subsumption. 100% pairwise coverage implies 100% each choice coverage. It does not imply 3-wise or full coverage.
6. Handle constraints. Exclude infeasible pairs from the denominator. A test containing an impossible combination is invalid and should not be counted.
7. Each test covers many pairs. One test with n parameters covers n(n-1)/2 pairs. Use this to sanity-check answers. Remember that duplicate pairs across tests count only once.
8. Recognise when to apply it. Scenario questions about many configurations, options or independent input fields point to combinatorial or pairwise testing. Questions about ranges point to domain testing. Questions about sequences of events point to state transition testing.
9. Think about risk. If a scenario mentions high risk or known three-way interactions, the best answer may be 3-wise coverage rather than pairwise.
10. Do not confuse terms. Orthogonal arrays and covering arrays are ways to achieve pairwise coverage. Pairwise coverage is the measure. Expected results still come from the test basis.
11. Time management. For calculation questions, write the parameter-pair grids quickly on scratch paper. Most CTAL-TA pairwise questions can be solved in 2 to 3 minutes if you work methodically.

Summary
Pairwise coverage measures how many of all feasible two-value combinations of input parameters a test set exercises. It is a powerful, efficient way to tame combinatorial explosion while targeting the interactions that cause most defects. For the exam, master four things: counting pairs, computing the coverage percentage, finding missing pairs and estimating the minimum number of tests. Also remember its place in the hierarchy between each choice coverage and n-wise coverage.