Combinatorial Testing: Parameters, Values and Interaction Failures

Combinatorial Testing: Parameters, Values and Interaction Failures (ISTQB CTAL-TA)

Overview
Combinatorial testing is one of the data-based test techniques in the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus. Domain testing and random testing are covered in the same area. Combinatorial testing deals with a common problem: a test object has several parameters, each parameter has several values, and there are far too many combinations to test them all.

The technique selects a small, systematic subset of combinations. That subset still detects most failures caused by parameters interacting with each other. For the exam you must understand the vocabulary, the coverage criteria, the basic calculations, constraints and the limitations of the technique.

1. Why Combinatorial Testing Is Important
Combinatorial explosion: The number of possible combinations is the product of the number of values of every parameter.
Example: 10 parameters with 4 values each give 4^10 = 1,048,576 combinations. Exhaustive testing is impossible in practice.

Most failures involve few parameters: Empirical research, notably NIST studies by Kuhn and colleagues, found the following.

• Most field failures are triggered by a single parameter value or by the interaction of two parameters.
• A smaller share needs three or more parameters.
• Almost none needs more than about six.

Covering all pairs (or all t-way combinations) therefore gives a very good return on test effort.

Typical application areas:

• Configuration testing: browsers, operating systems, devices, databases, locales.
• Option and feature-flag settings.
• Input forms with many fields.
• Product variants and settings menus.
• Interoperability and compatibility testing.

Efficiency and traceability: The technique gives a measurable coverage criterion. It replaces ad hoc guessing about which combinations matter.

2. What It Is: Key Concepts and Terminology
Parameter: A factor or variable of the test object that can vary independently. Examples are an input field, a configuration option or an environment attribute.

Value (parameter value): One of the options a parameter can take.

• Values often come from equivalence partitions or boundary values.
• For example, the parameter Age could have the representative values child, adult and senior rather than every possible number.
• Combinatorial testing is therefore often combined with other techniques that reduce each parameter to a manageable set of values.

Combination / interaction: A set of values taken from different parameters. A pair (2-way combination) is one value from each of two parameters. A t-way combination involves t parameters.

Interaction failure: A failure that appears only when specific values of two or more parameters occur together. Each value works correctly on its own.

• Example: a page renders correctly in Firefox and correctly on macOS, but fails only with Firefox on macOS.
• Lee Copeland calls these double-mode faults (two parameters) or multi-mode faults. Failures caused by one value regardless of the others are single-mode faults.

Coverage criteria (strength t):

• Each choice coverage (1-wise): every value of every parameter appears in at least one test case. This finds single-mode faults only.
• Pairwise coverage (2-wise): every possible pair of values for every pair of parameters appears in at least one test case. This is the most commonly used criterion.
• P-wise / t-wise coverage: every possible combination of values for every set of P (or t) parameters is covered. P = 3 is 3-wise, and so on.
• Full (exhaustive) combinatorial coverage: all combinations of all parameters. This is equivalent to t = number of parameters.

Coverage measurement: Coverage = (number of t-wise combinations covered by the tests) / (total number of feasible t-wise combinations), usually expressed as a percentage. Pairwise coverage = covered pairs / all feasible pairs.

Covering arrays and orthogonal arrays:

• A covering array is a table in which every t-way combination appears at least once. Tools generate these; examples are PICT, ACTS and AllPairs.
• An orthogonal array is a stricter mathematical structure in which every pair appears the same number of times.
• Orthogonal arrays are balanced but often need more tests. Covering arrays need only 'at least once', so they are usually smaller.

Classification tree method: This method is closely related. Parameters (classifications) and their values (classes) are modelled as a tree, and combination rules such as minimal, pairwise or full are then applied.

Constraints: Some combinations are impossible, invalid or irrelevant.

• Example: Safari on Windows, or the payment method invoice for a guest user.
• Constraints exclude these combinations from the test set.
• They also reduce the number of combinations that must be covered: coverage is measured against feasible combinations only.
• Constraints complicate manual test design, which is a strong reason to use tools.

3. How It Works: Step by Step
Step 1 – Identify the parameters. Analyse the test basis for factors that vary, such as inputs, configurations and environments.

Step 2 – Identify the values for each parameter. Use equivalence partitioning, boundary value analysis or domain knowledge to keep the number of values small.

Step 3 – Identify constraints. Determine combinations that are impossible or invalid, and dependencies between parameters.

Step 4 – Choose the coverage strength. Each choice, pairwise or higher t-wise, based on risk. Higher-risk systems justify 3-wise or more.

Step 5 – Generate the combinations. Use a tool, an orthogonal array or manual construction for small problems.

Step 6 – Complete the test cases. Add preconditions and, crucially, expected results. Combinatorial techniques select input combinations but do not provide expected results; a test oracle is still needed.

Step 7 – Measure coverage. Check which t-wise combinations are covered and add tests if needed.

Worked example
A web shop checkout has four parameters:

• Browser: Chrome, Firefox, Edge (3 values)
• OS: Windows, macOS (2 values)
• Language: EN, DE, FR (3 values)
• Payment: Card, PayPal (2 values)

Full combinations: 3 x 2 x 3 x 2 = 36 test cases.

Each choice coverage: The minimum number of tests equals the largest number of values of any single parameter, which is 3. For example:

• Chrome / Windows / EN / Card
• Firefox / macOS / DE / PayPal
• Edge / Windows / FR / Card

Pairwise coverage: number of pairs to cover. Add the product of the value counts for every pair of parameters.

• Browser-OS: 3x2 = 6
• Browser-Language: 3x3 = 9
• Browser-Payment: 3x2 = 6
• OS-Language: 2x3 = 6
• OS-Payment: 2x2 = 4
• Language-Payment: 3x2 = 6

Total = 37 pairs.

Pairwise coverage: lower bound. The minimum number of pairwise tests is at least the product of the two largest value counts, here 3 x 3 = 9.

• Every Browser-Language pair (9 of them) must appear, and one test case can contain only one such pair.
• In this example 9 tests are achievable, a 75% reduction compared with 36.

Effect of a constraint: Suppose Edge is not supported on macOS.

• The pair Edge-macOS is excluded.
• Feasible combinations and pairs are counted without it.
• The generated set must never contain Edge with macOS.

Number of parameter pairs: With n parameters there are n x (n-1) / 2 parameter pairs. For 4 parameters that is 4 x 3 / 2 = 6, matching the list above.

4. Strengths and Limitations
Strengths:

• Drastically reduces the number of test cases compared with exhaustive testing.
• Systematic, with measurable coverage.
• Effective against single-mode and double-mode interaction faults.
• Well supported by tools.
• Works well together with equivalence partitioning and boundary value analysis.

Limitations:

• Faults that need specific combinations of more than t parameters can be missed. Pairwise coverage does not guarantee finding 3-way faults.
• Effectiveness depends heavily on choosing the right parameters and representative values. Poor partitioning means poor tests.
• Expected results must still be determined separately.
• Constraints and dependencies can make the model hard to build, and can produce invalid tests if they are overlooked.
• Value combinations are treated as equally important. Risk-critical combinations may need to be added explicitly.
• Less suitable when parameters are heavily dependent on each other or when the sequence of actions matters. State transition testing suits sequences better.

Exam Tips: Answering Questions on Combinatorial Testing: Parameters, Values and Interaction Failures
Tip 1 – Know the definitions precisely.

• Each choice = every value appears at least once.
• Pairwise = every pair of values from every pair of parameters appears at least once.
• Many distractors swap these or say 'every pair of parameters' without mentioning values.

Tip 2 – Memorise the quick calculations.

• Full combinations = product of all value counts.
• Minimum tests for each choice = maximum number of values of any parameter.
• Minimum tests for pairwise is at least the product of the two largest value counts.
• Total pairs = sum of value-count products over all parameter pairs.
• Number of parameter pairs = n(n-1)/2.
• Coverage % = covered combinations / feasible combinations x 100.

Tip 3 – Remember constraints. If a question mentions impossible or unsupported combinations, exclude them before counting. Coverage is measured against feasible combinations only.

Tip 4 – Check a given test set for pairwise coverage.

• When asked whether a table achieves pairwise coverage, or how many tests to add, go through each parameter pair systematically.
• List the required value pairs and tick them off.
• Look for the missing pair. The question usually turns on a single one.

Tip 5 – Recognise the scenario. Keywords such as 'many configurations', 'options', 'compatibility', 'too many combinations', 'interactions between settings' or 'browsers and operating systems' point to combinatorial testing. Keywords such as 'boundaries', 'ranges' or 'sequence of events' point to other techniques.

Tip 6 – Understand interaction failures.

• A failure that occurs only when two specific values coincide is a pairwise (double-mode) interaction failure. Pairwise testing is designed to detect it.
• A failure that requires three specific values may not be found by pairwise testing.
• The correct answer often states that higher strength (3-wise) is needed.

Tip 7 – Watch for false claims. Typical wrong options:

• 'Pairwise testing finds all defects.'
• 'Combinatorial testing determines expected results.'
• 'Orthogonal arrays always need fewer tests than covering arrays.'
• 'Pairwise testing is equivalent to exhaustive testing.'

Tip 8 – Link to other techniques. Values usually come from equivalence partitioning or boundary value analysis. An answer that first reduces values using partitions and then applies pairwise coverage is often the best choice.

Tip 9 – Justify the choice of strength by risk. Questions may ask which coverage level is appropriate. Higher risk or safety-critical contexts point to higher t. Limited time with low risk points to each choice or pairwise.

Tip 10 – Use elimination and stay calm with numbers. For calculation questions, compute the lower bound first. Any option below the minimum, such as fewer than 9 tests in the example above, is automatically wrong. Options equal to the full product usually describe exhaustive testing, not pairwise testing.

Summary
Combinatorial testing models the test object as parameters with values. It selects combinations so that every t-wise interaction, most commonly every pair, is covered at least once. It exploits the observation that most interaction failures involve only a few parameters. It cuts test effort dramatically while keeping coverage measurable.

For the exam, master these points:

• The terminology: parameter, value, interaction, each choice, pairwise, t-wise.
• The simple counting rules.
• The handling of constraints.
• The limitations: no expected results, and higher-order faults can be missed.