Constraints and Infeasible Combinations in Combinatorial Testing (ISTQB CTAL-TA)
Introduction
Combinatorial testing techniques, such as pairwise testing, each-choice testing and classification tree based combinations, are covered in the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus as part of data-based test techniques. They help a test analyst handle the combinatorial explosion that occurs when a system has many parameters, each with several possible values. In real systems, however, not every combination of parameter values is allowed or even possible. This is where constraints and infeasible combinations come into play. Understanding them is essential both for designing realistic test suites and for answering exam questions correctly.
Why It Is Important
1. Realism of test cases: A combinatorial tool that ignores constraints may generate test cases that cannot be executed. An example is choosing the operating system 'macOS' together with the browser 'Internet Explorer 11'. Such test cases waste effort and can confuse test execution and reporting.
2. Coverage accuracy: If infeasible pairs are counted as part of the coverage target, the stated coverage figure is misleading. Coverage should be measured only against the feasible combinations.
3. Efficiency: Removing impossible combinations reduces the number of test cases and focuses effort on valid scenarios.
4. Risk of losing valid combinations: If you simply delete a test case that contains an infeasible combination, you may also lose valid pairs that appeared only in that test case. These must be recovered in another test case, or coverage drops.
5. Negative testing considerations: Some combinations are 'invalid' rather than physically impossible. The software is supposed to reject them. These may deserve separate negative tests rather than simple removal.
6. Exam relevance: The CTAL-TA syllabus explicitly mentions that constraints among parameters must be taken into account, and that tools or manual adjustments may be needed. Questions often test whether you know how to handle such constraints.
What It Is
Parameters and values: In combinatorial testing, the test object is modelled as a set of parameters (factors), each with a finite set of values (levels). Examples are Browser {Chrome, Firefox, Safari}, OS {Windows, macOS, Linux} and Language {EN, DE}.
Constraint: A rule that restricts which values of different parameters can appear together in a single test case. Constraints come from business rules, technical limitations, physical impossibility or dependencies. Typical forms are:
- Exclusion (forbidden pair): Safari cannot run on Linux.
- Implication / dependency: If Payment = 'Invoice', then Customer Type must be 'Business'.
- Conditional parameter: The parameter 'Card Type' is only relevant if Payment = 'Credit Card'. Otherwise it is 'not applicable'.
Infeasible combination: A combination of parameter values that cannot occur, or cannot be executed, because it violates a constraint. Infeasible combinations should not appear in the final executable test suite. They should also not be counted in the coverage denominator.
Infeasible vs. invalid: These two ideas are related but distinct.
- Infeasible (impossible): The combination cannot be set up at all. An example is a physical device that does not exist in that configuration.
- Invalid (not allowed by rules): The combination can be entered, but the system should reject it. These combinations can be useful for negative testing and error-handling tests.
How It Works
Step 1: Identify parameters and values. Use equivalence partitioning, classification trees or domain analysis to define the parameters and their representative values.
Step 2: Identify constraints. Analyse the requirements, business rules, technical specifications and domain knowledge. Document each constraint explicitly, for example 'IF OS = Linux THEN Browser != Safari'.
Step 3: Generate combinations while respecting constraints. There are two main approaches.
- Constraint-aware generation: Modern pairwise tools (for example PICT) accept constraint definitions. They generate only feasible test cases while still covering all feasible pairs. This is the preferred approach.
- Post-generation adjustment (manual handling): Generate the pairwise set first. Then find test cases containing infeasible combinations and handle them in one of these ways:
a) Replace the offending value with another value that makes the test case feasible, while keeping the other valid pairs.
b) Remove the test case, then add new test cases to cover any valid pairs that would otherwise be lost.
c) Use 'don't care' values flexibly to resolve conflicts.
Step 4: Recalculate coverage. Coverage = (number of feasible combinations covered) / (total number of feasible combinations). Infeasible combinations are excluded from the denominator.
Step 5: Consider negative tests. For invalid (but enterable) combinations, decide whether to design separate tests that verify the system rejects them properly. Do this outside the pairwise set, so that error handling does not mask other pairs.
Worked Example
Parameters:
- OS: Windows, macOS, Linux
- Browser: Chrome, Safari
- Mode: Light, Dark
Constraint: Safari is available only on macOS.
Total pairs without constraints are OS-Browser (6), OS-Mode (6) and Browser-Mode (4), which gives 16. The infeasible pairs are Windows-Safari and Linux-Safari. That leaves 14 feasible pairs to cover.
A constraint-respecting pairwise set could be:
1. Windows, Chrome, Light
2. Windows, Chrome, Dark
3. Linux, Chrome, Light
4. Linux, Chrome, Dark
5. macOS, Safari, Light
6. macOS, Chrome, Dark
7. macOS, Safari, Dark
Every one of the 14 feasible pairs is covered, and no test contains an infeasible pair. Notice two effects:
- Constraints can increase the number of test cases compared with the unconstrained minimum, which here would be 6.
- In a different model, a constraint can decrease the number of test cases by removing combinations.
Common Pitfalls
- Deleting an infeasible test case without checking which valid pairs are lost.
- Counting infeasible pairs in the coverage target, which makes 100% coverage seem impossible.
- Failing to document constraints, so the model becomes unmaintainable.
- Mixing invalid values into pairwise tests. One invalid value may cause early rejection, which masks the other pairs in that test case (input masking).
- Over-constraining the model, which removes legitimate combinations that real users could encounter.
Exam Tips: Answering Questions on Constraints and Infeasible Combinations in Combinatorial Testing
1. Know the definitions: A constraint is a dependency or restriction between parameter values. An infeasible combination is one that cannot exist or be executed. Be ready to distinguish 'infeasible' from 'invalid'.
2. The correct handling is never simply 'delete and forget': If an option says to just remove test cases with infeasible combinations, be cautious. The best answer usually mentions preserving coverage of the remaining valid pairs, either by modifying values or by adding test cases.
3. Prefer constraint-aware tools: When asked about the best approach, answers involving defining constraints in the combinatorial tool before generation are typically correct.
4. Coverage calculations: Exclude infeasible pairs from the total. If a question asks for the number of pairs to be covered, subtract the forbidden pairs first. Practise counting pairs: for parameters with n1, n2, n3 values, total pairs = n1*n2 + n1*n3 + n2*n3.
5. Expect test count changes: Constraints can change the minimum number of test cases. Do not assume the theoretical minimum (product of the two largest value counts) always applies.
6. Watch for masking: If a question involves invalid values in combinatorial tests, remember that invalid values should typically be tested separately. This avoids masking the effect of other parameter values.
7. Read scenarios carefully: Exam questions often hide constraints in the business rules of a scenario, such as 'Express delivery is only available for domestic addresses'. Identify these before choosing an answer.
8. Look for 'not applicable' values: Conditional parameters may need an 'N/A' value. Recognise when a parameter becomes irrelevant under certain values of another parameter.
9. Eliminate extreme answers: Options claiming that constraints make combinatorial testing unusable, or that constraints should always be ignored, are usually wrong.
10. Link to classification trees: The syllabus connects combinatorial techniques to classification trees, where constraints between classes are also captured. Questions may combine these ideas.
Summary
Constraints describe which parameter value combinations are allowed. Infeasible combinations are those that violate them. A competent test analyst identifies constraints early and applies them during generation, preferably with tools. When manual adjustments are needed, the analyst makes sure valid combinations are still covered and measures coverage against feasible combinations only. In the exam, focus on correct handling that preserves coverage, accurate pair counting, and the distinction between infeasible and invalid combinations.
