Metamorphic Testing for Non-Functional Testing and AI-Based Systems (ISTQB CTAL-TA)
Overview
Metamorphic Testing (MT) is a rule-based test technique covered in the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus. It is especially valuable when there is no reliable test oracle, meaning you cannot easily determine the correct expected result for a given input. This situation is common in AI-based systems such as machine learning models, recommender systems and image classifiers. It is also common in non-functional testing such as performance, scalability and reliability, where an exact expected value is often unknown. This guide explains why MT matters, what it is, how it works, and how to answer exam questions about it.
1. Why Metamorphic Testing Is Important
The test oracle problem: Traditional testing compares actual results with expected results. For many modern systems, the expected result is unknown, too expensive to compute, or non-deterministic. Examples include:
- Search engines and ranking algorithms
- Machine learning classifiers and neural networks
- Scientific simulations and complex numerical computations
- Compilers, optimizers and translation systems
- Performance under varying loads, where exact response times cannot be predicted
Benefits of MT:
- It provides a partial oracle through relationships between multiple executions, so you do not need exact expected outputs.
- It can generate many follow-up test cases from a small set of source test cases, which makes it highly scalable and easy to automate.
- It detects subtle faults, such as inconsistency, bias, robustness problems and lack of invariance, that conventional testing may miss.
- It suits AI systems whose behaviour is learned rather than explicitly specified.
- It supports non-functional testing by checking expected relative behaviour, for example: doubling the load should not reduce throughput below a defined threshold.
2. What Metamorphic Testing Is
MT is a technique in which test cases are derived from metamorphic relations (MRs). An MR is a necessary property of the test object that describes how a change to an input should affect the output. Instead of checking a single output against a known correct value, MT checks whether the relationship between the outputs of several related executions holds.
Key terms:
- Source test case: the original input, often taken from existing tests or chosen randomly.
- Follow-up test case: a new input created by transforming the source test case according to an MR.
- Metamorphic relation (MR): the rule linking the input transformation to the expected relationship between outputs.
- Violation: an output relationship that does not match the MR. A violation indicates a likely defect, but a pass does not prove correctness.
Classic simple example: For a sine function, sin(x) = sin(pi - x). You may not know the exact value of sin(1.234), but you can check that sin(1.234) equals sin(pi - 1.234). If they differ, there is a defect.
3. How Metamorphic Testing Works (Process)
The process as reflected in ISTQB material:
1. Analyse the test object and identify MRs: Use specifications, domain knowledge and stakeholder input to find properties that must hold across related inputs.
2. Create source test cases: Use existing tests, random generation or equivalence partitions.
3. Generate follow-up test cases: Apply the input transformation defined by each MR to each source test case.
4. Execute both source and follow-up test cases.
5. Compare outputs against the MR: If the expected relationship does not hold, report a potential defect.
6. Iterate: Refine MRs, combine MRs, and automate generation to reach broad coverage.
Types of input transformations that commonly underpin MRs:
- Additive or multiplicative changes, such as adding a constant or scaling
- Permutation, such as reordering inputs
- Inclusion or exclusion, such as adding or removing elements
- Inversion, such as negation or reversal
- Noise addition and perturbation, which is typical for AI robustness testing
- Equivalent rephrasing, such as synonyms in NLP
4. Metamorphic Testing for AI-Based Systems
AI systems often lack a specification of correct output for every input, and their behaviour can be probabilistic. MT addresses this well.
Examples of MRs for AI systems:
- Image classification: Slightly rotating, brightening, cropping or adding imperceptible noise to an image of a cat should still result in the classification cat. This tests robustness and invariance.
- Natural language processing: Replacing a word with a synonym should not change the sentiment classification. Changing a person's name or gender in a loan application text should not change the decision. This tests fairness and bias.
- Recommender systems: Adding an item a user has already rated highly should not drastically change unrelated recommendations.
- Search engines: Results for a more restrictive query, such as adding an AND term, should be a subset of the results for the broader query.
- Machine learning training: Permuting the order of training data, for algorithms that are order-independent, should produce an equivalent model. Duplicating all training data should not change a k-NN classification.
- Autonomous driving: Adding rain or fog effects to camera images should not cause the steering angle to deviate beyond a tolerance.
Quality characteristics tested: functional correctness, robustness, fairness or bias, consistency, and adversarial resilience.
5. Metamorphic Testing for Non-Functional Testing
Non-functional characteristics often cannot be measured against exact expected values. MT allows relative checks between executions.
Examples:
- Performance efficiency: If the workload doubles, response time should increase by no more than a defined factor. A query on a subset of data should not take longer than the same query on the full data set.
- Scalability: Adding server resources should not decrease throughput.
- Reliability: Running the same transaction repeatedly under identical conditions should yield consistent results and no degradation, which helps detect memory leaks.
- Security: Encrypting then decrypting data should return the original. Adding unauthorized privileges to a request should never grant more access.
- Usability and compatibility: Rendering the same page on different but equivalent browsers or resolutions should preserve key layout elements.
- Resource utilization: Processing a smaller input file should not consume more memory than processing a larger one.
6. Strengths and Limitations
Strengths:
- Mitigates the test oracle problem
- Highly automatable and scalable
- Reuses existing test cases as source tests
- Effective for complex, non-deterministic and AI systems
- Reveals defects in robustness, bias and consistency
Limitations:
- Identifying good MRs requires domain expertise and creativity.
- MRs provide only a partial oracle. Passing all MRs does not prove correctness, because a system can be consistently wrong.
- Some MRs may only hold approximately, especially for AI or performance. These need tolerances, which can cause false positives or false negatives.
- Results for non-deterministic systems may require statistical evaluation over multiple runs.
- Locating the root cause of a violation can be difficult.
7. How MT Relates to Other Techniques
- It is a rule-based technique in CTAL-TA, alongside other rule-based techniques where test design follows explicit rules.
- It complements back-to-back testing, which uses an alternative implementation as a pseudo-oracle. MT uses relations within the same system instead.
- It can be combined with equivalence partitioning, boundary value analysis or random testing to create source tests.
- In AI testing, it is often used alongside adversarial testing, A/B testing and pairwise testing.
8. Exam Tips: Answering Questions on Metamorphic Testing for Non-Functional Testing and AI-Based Systems
Tip 1: Link MT to the test oracle problem. If a question mentions that expected results are unknown, hard to compute or non-deterministic, MT is very likely the correct answer.
Tip 2: Know the vocabulary precisely. Be ready to identify the source test case, the follow-up test case and the metamorphic relation in a scenario. The follow-up test is derived from the source test by a transformation.
Tip 3: An MR is a relation between outputs of multiple executions. It is not an expected value for a single test. Distractors often describe a normal expected result. Reject those.
Tip 4: Recognise valid MRs. A valid MR must be a necessary property of the system. Example: adding an irrelevant filter to a search should not increase the number of results. Watch for distractors where the relation does not logically hold. For instance, a statement that a sorted list's length doubles after sorting is invalid.
Tip 5: A violation indicates a defect; a pass does not prove correctness. Questions often test this nuance. Choose the answer saying MT can reveal defects but cannot confirm absence of defects.
Tip 6: Match MRs to quality characteristics. Synonym replacement or name changes point to fairness or bias. Image noise or rotation points to robustness. Load doubling points to performance efficiency or scalability. Repeated identical runs point to reliability or consistency.
Tip 7: Expect scenario questions on AI systems. When an AI classifier, chatbot, recommender or autonomous system is described and asked which technique is best for testing without labelled expected outputs, pick MT.
Tip 8: Remember the main challenge. If asked about the greatest difficulty in applying MT, the answer is usually identifying suitable metamorphic relations, which requires domain knowledge.
Tip 9: Remember the automation advantage. MT can generate large numbers of follow-up tests automatically from few source tests. This is often cited as a key benefit.
Tip 10: Distinguish MT from back-to-back testing. Back-to-back testing compares outputs of two different implementations of the same specification. MT compares outputs of the same implementation for related inputs.
Tip 11: Consider tolerances in non-functional and AI contexts. If an option mentions acceptable tolerances or statistical evaluation for approximate MRs, it is probably correct for performance or ML scenarios.
Tip 12: Use elimination. Remove options that require exact expected outputs, that confuse MT with exploratory or experience-based testing, or that claim MT guarantees completeness.
Sample exam-style question:
A test analyst is testing an ML model that predicts house prices. No reliable oracle exists for exact prices. Which test is the BEST example of a metamorphic test?
a) Compare the prediction for a house against its actual sale price
b) Increase the floor area of a house while keeping all other features the same, and verify the predicted price does not decrease
c) Review the training data for missing values
d) Measure the model's response time under load
Answer: b. It transforms a source input into a follow-up input and checks an expected output relation, the MR that price should not decrease with more area. Option a requires an oracle. Option c is a data review. Option d is performance measurement without a relation.
Summary
Metamorphic testing derives follow-up tests from source tests using metamorphic relations. It overcomes the test oracle problem, which makes it ideal for AI-based systems and for non-functional characteristics such as performance, scalability and reliability. In the exam, connect MT to missing oracles, identify the source and follow-up tests and the MR, match MRs to quality characteristics, and remember both its automation strength and its limitation as a partial oracle.
