Source Test Cases and Follow-Up Test Cases in Metamorphic Testing (ISTQB CTAL-TA)
Overview
In the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, metamorphic testing (MT) is one of the rule-based test techniques. It is built on two kinds of test cases: source test cases and follow-up test cases. They are linked by a metamorphic relation (MR). Many exam questions ask you to identify which test is the source and which is the follow-up. Others ask you to derive a follow-up from a given MR, or to judge whether a test passed or failed. To answer them, you need to understand how these three elements fit together.
Why It Is Important
Metamorphic testing addresses the test oracle problem. This is the situation where the tester cannot easily determine the correct expected result for a given input. Typical examples include:
- scientific simulations;
- machine learning and AI models;
- search engines and recommendation systems;
- compilers;
- complex numerical algorithms;
- image processing;
- big-data analytics.
In these situations, traditional tests that compare an actual result with an exact expected result are difficult or impossible to design. Metamorphic testing does not ask whether an output is exactly correct. It asks whether the outputs of related executions are consistent with each other, according to a known rule.
This brings several benefits:
- Testing is possible even without a full oracle.
- Many new test cases can be generated from a small set of existing ones, which supports automation and scaling.
- Existing test cases can be reused as source test cases.
- The technique can reveal subtle defects that single tests often miss.
- It suits systems with non-deterministic or hard-to-predict behaviour.
What It Is: Key Definitions
Metamorphic relation (MR): a property of the test object that describes how a change to the inputs should affect the outputs. An MR has two parts:
- an input transformation, which describes how to create new inputs from existing ones;
- an output relation, which describes how the outputs of the related executions are expected to relate.
Source test case: the original test case, with its input and the output actually observed when it is executed. It can come from any source:
- an existing regression test;
- a test designed with another technique, such as equivalence partitioning or boundary value analysis;
- randomly generated input;
- production-like data.
Importantly, the exact correct output of the source test case does not need to be known.
Follow-up test case: a new test case derived by applying the MR's input transformation to the source test case's input. After it is executed, its output is compared with the source output using the MR's output relation, not with a fixed expected value.
How It Works: Step by Step
1. Identify metamorphic relations from the specification, the domain, mathematical properties or user expectations. Examples: symmetry, invariance, monotonicity, permutation, scaling, addition or removal of data.
2. Create or select source test cases. Execute them and record the outputs.
3. Derive follow-up test cases by applying the input transformation of each MR to the source inputs.
4. Execute the follow-up test cases and record their outputs.
5. Check the output relation between the source output and the follow-up output.
- If the relation holds, the test passes for that MR.
- If the relation is violated, a failure has been revealed. Something in the test object is defective, or the MR itself was wrongly defined.
Worked Examples
Example 1: Mathematical function
- MR: sin(x) = sin(pi - x).
- Source test case: input x = 0.7, output sin(0.7) recorded. You do not need to know the exact value.
- Follow-up test case: input pi - 0.7.
- Check: both outputs must be equal, within tolerance.
Example 2: Online search with filters
- MR: adding a filter must return a subset of the original results.
- Source test case: search 'running shoes', which returns N results.
- Follow-up test case: search 'running shoes' with the filter 'size 42'.
- Check: every follow-up result must also appear in the source results, and the count must be less than or equal to N.
Example 3: Shortest-path routing
- MR: the shortest distance from A to B equals the shortest distance from B to A on an undirected map.
- Source test case: route A to B.
- Follow-up test case: route B to A.
- Check: both distances must be equal.
Example 4: Sorting or averaging
- MR: permuting the input list does not change the average or the sorted output.
- The follow-up test case is the shuffled list.
Important Characteristics to Remember
- One source test case can produce many follow-up test cases, for example one per MR or several per MR.
- A follow-up test case can itself act as a source for further follow-ups, creating chains.
- Several MRs can be combined.
- A passed MR check does not prove correctness. It shows only that the outputs are consistent. For example, both outputs could be wrong in the same way.
- A failed check reveals a failure, but further analysis is needed to locate the defect, or to find out that the MR was invalid.
- Coverage is usually considered in terms of the MRs exercised and the number of source and follow-up pairs. The effectiveness of the technique depends heavily on the quality and variety of the MRs.
- MT is highly suitable for automation, because generating follow-ups and checking relations can be scripted.
- MT complements other techniques. Source test cases are often produced with other techniques.
Common Confusions
- The source test case needs a known expected result. False. Only the relation between outputs needs to be known.
- The follow-up test case is compared with a predefined expected value. False. It is compared with the source output through the output relation.
- The follow-up test case is a retest or confirmation test. False. It is a new, transformed input, not a re-execution after a fix.
- The MR is the same as a test case. False. The MR is the rule. Source and follow-up test cases are its concrete instances.
Exam Tips: Answering Questions on Source Test Cases and Follow-Up Test Cases
1. Spot the trigger words. Phrases such as 'no reliable oracle', 'expected results are hard to determine', 'AI/ML system', 'simulation' or 'search engine' strongly suggest that metamorphic testing is the right answer.
2. Identify the roles. The source is the original input and its observed output. The follow-up is the input after the MR transformation has been applied. If a question lists pairs of tests, ask yourself which one was derived from the other.
3. Separate the two parts of the MR. First apply the input transformation to the source input to get the follow-up input. Then use the output relation to decide pass or fail. Distractors often mix the two up.
4. Judge pass or fail only through the relation. If the follow-up output satisfies the relation with the source output, the test passes. This is true even if you cannot tell whether either output is 'correct'. If the relation is violated, the answer is 'a failure has been detected'.
5. Watch for overclaiming. Reject answer options that say MT 'proves the software is correct' or 'eliminates the need for other techniques'. Also reject options that require 'exact expected results for the source test case'.
6. Derive follow-ups carefully. In scenario questions, apply the stated transformation exactly. For example, if the MR says 'double all prices, then the total doubles', the follow-up input has every price doubled, and the check is total_followup = 2 x total_source.
7. Recognise multiplicity. Remember that one source can produce several follow-ups, and that follow-ups can become sources. Options stating 'exactly one follow-up per source' are usually wrong.
8. Know where it sits in the syllabus. Metamorphic testing belongs to the rule-based techniques, alongside decision table testing. It is not experience-based or data-based.
9. Eliminate distractors systematically. Remove options that describe the following, because none of them is a follow-up test case:
- confirmation testing;
- regression testing;
- equivalence partitions;
- boundary values.
10. Practise with simple MRs. Good practice relations include symmetry (A to B equals B to A), permutation (order does not matter), subset (adding a filter reduces results), scaling (doubling the input doubles the output) and invariance (adding zero changes nothing). Most exam scenarios are built on one of these patterns.
Summary
Metamorphic testing uses a metamorphic relation to turn a source test case, whose correct output need not be known, into one or more follow-up test cases. A failure is detected when the outputs of the source and follow-up test cases violate the expected relation. In the exam, follow these four steps:
- Identify the oracle problem.
- Find the source and the follow-up.
- Apply the input transformation.
- Judge the result only through the output relation.
