Combining Metamorphic Testing with Random Testing

Combining Metamorphic Testing with Random Testing (ISTQB CTAL-TA): A Complete Guide

Introduction
Metamorphic Testing (MT) is one of the rule-based test techniques covered in the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus. It is often combined with random testing, and the two work well together. Random testing can produce large numbers of inputs cheaply, but it usually cannot tell whether each output is correct. Metamorphic testing provides that missing check. This guide explains why the combination matters, what it is, how it works and how to answer exam questions about it.

1. Why It Is Important
The test oracle problem: Many systems produce outputs whose correctness is hard or impossible to verify. Examples include scientific simulations, machine learning models, search engines, compilers, optimisation algorithms, image processing and complex financial calculations. For a single input, the tester often does not know the exact expected result. This is called the test oracle problem.
The weakness of pure random testing: Random testing can generate thousands or millions of inputs automatically. Without an oracle, however, the only failures it can detect are crashes, exceptions, timeouts or obviously invalid outputs. Subtle wrong results go unnoticed.
The strength of the combination: Metamorphic testing does not need the expected output of an individual test. It checks relationships between the outputs of several related executions. When random testing creates the source test cases and metamorphic relations create the follow-up test cases, you get:
- Large-scale, automatable test execution.
- An automated partial oracle that detects incorrect results, not only crashes.
- Low test design cost per test case, because one relation can be applied to unlimited random inputs.
- Better fault detection in domains where traditional expected-result testing is impractical.

2. What It Is
Metamorphic Relation (MR): A property of the test object stating how a change to the input should affect the output. For example, for a sine function the MR could be: sin(x) = sin(π - x). For a search engine, it could be: adding a more restrictive filter must never return more results than the original query.
Source test case: The original test input, executed first. Its output is recorded even though its correctness may be unknown.
Follow-up test case: A new test input derived from the source test case by applying the transformation defined in the MR.
Random testing: A black-box approach in which inputs are selected randomly, often following an operational profile or a defined input distribution, usually with tool support.
The combination: Random testing is used to generate the source test cases. Each MR is then applied to create follow-up test cases automatically. The outputs of source and follow-up executions are compared against the MR. A violation of the MR shows that a defect exists, even though nobody knows the exact correct answer for any single input.

3. How It Works (Step by Step)
Step 1 - Analyse the test basis and identify metamorphic relations. The Test Analyst uses domain knowledge, specifications and user stories to find properties that must hold. Common MR patterns include:
- Equivalence: permuting input order gives the same output (for example, the sum of a list is unchanged when the list is shuffled).
- Additive or multiplicative: multiplying all inputs by a constant scales the output predictably.
- Inclusion or exclusion: adding an element to a set can only increase or keep the same count.
- Inversion: applying an operation and then its inverse returns the original value.
- Subset or superset: narrowing a search query returns a subset of the original results.
Step 2 - Define the input domain and random generation strategy. Decide the input ranges, data types and distribution (uniform, operational profile, or adaptive random testing for better spread). Make sure the generated inputs are valid for the MR's preconditions.
Step 3 - Generate source test cases randomly. A tool or script produces N random inputs.
Step 4 - Derive follow-up test cases. For each source input, apply the MR transformation, such as x becomes π - x, the list becomes a shuffled list, or the query becomes the query plus a filter.
Step 5 - Execute both source and follow-up test cases. Record all outputs.
Step 6 - Check the relation. Compare the outputs using the MR, taking tolerances into account for floating-point results. Any violation is a failure that needs investigation.
Step 7 - Analyse, report and iterate. Investigate violations. Remember that a violation could also come from an incorrect MR. Add or refine MRs to improve fault detection.

Worked example:
Test object: a function that computes the shortest path distance between two nodes in a graph.
MR: the shortest distance from A to B equals the shortest distance from B to A in an undirected graph.
Random testing: generate 10,000 random undirected graphs and random node pairs (source tests).
Follow-up tests: swap the start and end nodes.
Check: dist(A,B) = dist(B,A). Any mismatch reveals a defect, even though the true shortest distance for each random graph was never calculated by hand.

4. Benefits and Limitations
Benefits:
- Addresses the test oracle problem.
- Highly automatable and scalable.
- Reuses one MR across countless random inputs.
- Finds subtle computational defects that crash-only random testing misses.
- Useful for AI/ML systems, scientific software and non-testable programs.
Limitations:
- MRs are only a partial oracle. Satisfying an MR does not prove the outputs are correct, because a program can be consistently wrong.
- Identifying good MRs requires domain expertise and creativity.
- Random inputs may not exercise specific boundaries or rare paths, so coverage can be uneven.
- Floating-point tolerances and non-determinism complicate the comparison.
- A failure might come from an invalid MR rather than a defect in the code.
- Many test executions require tool support and computing resources.

5. Relationship to Other Techniques
- Equivalence partitioning and boundary value analysis can complement random generation by making sure important partitions and boundaries are included as source tests.
- Adaptive random testing spreads inputs more evenly across the input domain and is often paired with MT.
- Back-to-back testing is a different oracle approach that compares against an alternative implementation. MT instead compares the same implementation against itself under related inputs.
- Experience-based techniques help identify good metamorphic relations.

Exam Tips: Answering Questions on Combining Metamorphic Testing with Random Testing
Tip 1 - Recognise the trigger words. Look for phrases such as no expected results available, difficult to determine correct output, test oracle problem, large number of automatically generated inputs, machine learning or scientific calculation. These point to MT, often combined with random testing.
Tip 2 - Know who does what. Random testing generates the source test cases. The metamorphic relation generates the follow-up test cases and acts as the oracle. Questions often test whether you can tell these roles apart.
Tip 3 - Remember that MT is a partial oracle. If an option claims MT proves correctness or guarantees that all defects are found, it is wrong. Satisfying an MR only shows consistency.
Tip 4 - Identify valid MRs. You may be given a scenario and asked to choose the best metamorphic relation. A valid MR must follow from the specification or domain and must state a checkable relationship between the outputs of related inputs. Reject options that require knowing the exact expected output, because that is a conventional oracle, not an MR.
Tip 5 - Distinguish MT from similar techniques. Back-to-back testing compares two implementations. Regression testing compares against previous versions. MT compares multiple executions of the same system linked by a relation.
Tip 6 - Count the test cases correctly. If a question says 100 random source inputs and 3 MRs, each producing one follow-up per source input, then there are 300 follow-up tests and 400 executions in total. Read carefully whether the question asks for follow-up tests or total executions.
Tip 7 - Interpret MR violations correctly. A violation indicates a failure, so investigate the defect. Also consider that the MR itself might be wrong. A pass does not mean the output is correct.
Tip 8 - Choose the right context. The combination is most suitable when outputs are hard to verify, inputs are easy to generate, and automation is available. It is less suitable when expected results are easy to calculate, because a conventional oracle is then simpler and stronger.
Tip 9 - Watch for tolerance and determinism. With floating-point or non-deterministic systems, a good answer mentions comparing outputs within an acceptable tolerance, or controlling random seeds for reproducibility.
Tip 10 - Reproducibility matters. Random tests should record their seeds or inputs so that failures can be reproduced and retested. Options mentioning logging of generated inputs are usually correct good practice.

Sample Exam-Style Question
A Test Analyst is testing a weather simulation model whose exact outputs cannot be predicted. The team can generate thousands of random input scenarios automatically. Which approach BEST addresses the lack of expected results?
a) Execute random inputs and only check for crashes.
b) Use random inputs as source test cases and verify metamorphic relations, such as increasing humidity input never decreasing the predicted rainfall probability, on derived follow-up test cases.
c) Calculate the expected outputs manually for each random input.
d) Use boundary value analysis only.
Correct answer: b). It combines automatic random generation with MRs as an oracle. Option a) misses incorrect results, c) is impractical, and d) does not solve the oracle problem.

Summary
Combining metamorphic testing with random testing pairs the scalability of random input generation with the oracle capability of metamorphic relations. Random testing provides the source test cases, MRs derive the follow-up test cases, and relation checks expose defects without knowing exact expected outputs. For the exam, remember the terminology (MR, source and follow-up test cases), the oracle problem motivation, the partial-oracle limitation, and how to identify valid relations and suitable contexts.