Solutions to the Test Oracle Problem: Pseudo-Oracles, Property-Based Testing and Human Oracles (ISTQB CTAL-TA)
Introduction
In the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, the Test Analyst is responsible for designing tests whose results can be judged correctly. Every test needs a way to decide whether the observed outcome is right or wrong. The mechanism that makes this decision is called the test oracle. When no reliable oracle exists, or when using it is impractical, testers face the test oracle problem. This guide explains what the problem is, why it matters, how the main solutions work (pseudo-oracles, property-based testing including metamorphic relations, and human oracles), and how to answer exam questions on the topic.
1. What Is a Test Oracle?
A test oracle is a source used to determine the expected results of a test so they can be compared with the actual results of the system under test. Typical oracles include:
- Requirements and specifications
- User stories with acceptance criteria
- Existing systems, such as a legacy version or a competitor product
- Models, such as state models and business rule tables
- Domain experts and users
- Calculations made by hand or with a spreadsheet
- Regulatory standards
2. What Is the Test Oracle Problem?
The test oracle problem occurs when it is difficult, costly or impossible to determine the correct expected result for a given input. Common causes are:
- Complex calculations: scientific simulations, machine learning models, financial risk engines and weather prediction.
- Non-deterministic behaviour: concurrency, randomness and AI or ML outputs.
- Large volumes of output: big data transformations and image or video rendering.
- Missing, vague or outdated specifications.
- Subjective quality attributes: usability, look and feel, and how natural an audio or translated text is.
- New, innovative systems that nothing can be compared against.
3. Why Is This Important?
- Without an oracle, a test can be executed but cannot be evaluated. You can run it, but you cannot say whether it passed or failed.
- A weak oracle leads to false positives (reporting defects that do not exist) and false negatives (missing real defects).
- Test automation depends on automated oracles. Without them, automation only detects crashes, not wrong results.
- The Test Analyst must pick oracle strategies that fit the risk, cost and nature of the system. This is a core skill tested at Advanced Level.
- It ties into test design. Expected results are part of every test case, so oracle decisions are made during test analysis and design, not only during execution.
4. Solution 1: Pseudo-Oracles
What it is: A pseudo-oracle is an independently developed program, system or model that produces results for the same inputs. Its output is compared with the output of the system under test. It is not guaranteed to be correct, but it is independent, which is why it is called pseudo.
How it works:
1. Build or obtain an alternative implementation, for example a simplified algorithm, a prototype in another language, a spreadsheet model, a legacy system or a competitor product.
2. Feed identical inputs to both systems.
3. Compare the outputs. Discrepancies point to a potential defect in either one and must be investigated.
Variants:
- N-version programming or back-to-back testing
- Legacy system comparison during migration
- Reference implementations
Advantages:
- Works well for complex calculations.
- Enables automated comparison of large numbers of tests.
Limitations:
- Costly to develop and maintain.
- Both versions may share the same defect, especially if written from the same faulty specification or by the same people. This is the common-mode failure risk.
- A mismatch does not reveal which version is wrong.
Example: A new mortgage calculation engine is compared with an Excel model built independently by the business team.
5. Solution 2: Property-Based Testing and Metamorphic Testing
What it is: Instead of knowing the exact expected output, the tester defines properties (invariants or rules) that every correct output must satisfy. Test inputs are often generated automatically and in large numbers, and each result is checked against the properties.
How it works:
1. Identify properties. For example, a sorting function's output must have the same length as the input, contain the same elements and be in non-decreasing order.
2. Generate many inputs, often random or constrained-random, using tools such as QuickCheck, Hypothesis or jqwik.
3. Check that each output satisfies the properties. Any violation is a failure.
4. Many tools then shrink a failing input to the smallest example that still fails.
Metamorphic testing is a closely related technique based on metamorphic relations (MRs). These describe how outputs should change when inputs are changed in a known way.
- sin(x) = sin(pi - x)
- Searching for 'shoes red' should return a subset of the results for 'shoes'.
- Rotating an image should not change an image classifier's label.
- Doubling all prices should double the order total.
The relationship can be checked even when the exact correct output is unknown.
Advantages:
- No exact expected result is needed.
- Highly automatable and scales to huge input spaces.
- Good at finding edge-case defects.
- Very useful for AI and ML systems, simulations and scientific software.
Limitations:
- Properties are only partial oracles. An output may satisfy every property and still be wrong.
- Defining good properties requires domain knowledge and skill.
- Random generation may miss specific important values, so it should be combined with boundary value and equivalence partitioning tests.
6. Solution 3: Human Oracles
What it is: A person, such as a domain expert, end user, business analyst or the tester, judges whether the output is correct or acceptable.
How it works:
- Experts review outputs, for example a radiologist checking the results of medical imaging software.
- Exploratory and session-based testing, where testers use their knowledge and heuristics.
- Acceptance testing by users.
- Crowd testing or panels for subjective qualities.
- Heuristic oracles, such as consistency with history, with the product image, with comparable products, with user expectations or with the claims made about the product.
Advantages:
- Can evaluate subjective, ambiguous or context-dependent outputs.
- Brings domain knowledge and judgment.
- Needs no implementation effort.
Limitations:
- Slow and expensive, and does not scale to large output volumes.
- Subject to error, fatigue, bias and inconsistency between people.
- Hard to repeat and automate.
Ways to reduce the cost include reducing output size, sampling, producing readable outputs and minimising the number of tests that need human evaluation.
7. Other Related Approaches
- Partial or implicit oracles: crashes, exceptions, memory leaks, timeouts and assertion failures. These detect obvious failures without needing expected values.
- Statistical oracles: checking distributions or averages, which suits non-deterministic systems.
- Model-based oracles: a state model or decision table predicts the result.
- Consistency or regression oracles: comparing with a previous version's baseline, also called golden master testing.
8. Choosing a Solution: Decision Guide
- Can you build or obtain an independent implementation, or does a legacy or reference system exist? Use a pseudo-oracle.
- Is the exact output unknown, but you know rules or relations it must obey, and you need volume and automation? Use property-based or metamorphic testing.
- Is the output subjective or does it require expert judgment, and is the volume manageable? Use a human oracle.
- In practice, combine them. For example, use properties for broad automated checks and human experts to sample outputs.
9. Worked Example
Scenario: A team develops software that predicts flood risk using a complex hydrological simulation, and no exact expected outputs exist.
- Pseudo-oracle: compare with an established open-source hydrological model.
- Metamorphic relation: increasing rainfall while keeping everything else constant should never decrease the predicted water level.
- Property: water levels must never be negative, and flooded area must not exceed the total area.
- Human oracle: hydrologists review a sample of predicted flood maps for plausibility.
10. Exam Tips: Answering Questions on Solutions to the Test Oracle Problem: Pseudo-Oracles, Property-Based Testing and Human Oracles
Tip 1: Recognise the oracle problem in the scenario. Keywords include 'expected results are unknown', 'complex calculation', 'AI/ML', 'no specification', 'non-deterministic', 'subjective' and 'huge output'. These signal that an oracle solution is required.
Tip 2: Map keywords to solutions.
- 'Independent implementation', 'legacy system', 'competitor product', 'back-to-back' or 'spreadsheet model' points to pseudo-oracle.
- 'Rules the output must satisfy', 'relationship between inputs and outputs', 'invariants', 'generated inputs' or 'transformations of input' points to property-based or metamorphic testing.
- 'Expert judgment', 'look and feel', 'plausibility', 'usability' or 'domain specialist' points to human oracle.
Tip 3: Know the key weakness of each.
- Pseudo-oracle: shared or common-mode defects and development cost.
- Property-based: only a partial oracle, so a correct-looking output may still be wrong.
- Human: slow, costly, error-prone and not scalable.
Exam distractors often claim that a technique is 'complete' or 'always correct'. Reject such absolute statements.
Tip 4: Watch for cost and scale constraints. If the scenario mentions thousands or millions of outputs, or continuous integration, a human oracle is rarely the best answer. Prefer automated pseudo-oracles or property checks.
Tip 5: Independence matters. A pseudo-oracle written by the same developer from the same specification gives weak assurance. The best answer usually stresses independent development.
Tip 6: Metamorphic means relations, not exact values. If an option describes checking how output changes when input is modified, that is metamorphic testing, even if the term is not used.
Tip 7: Pick the most appropriate answer, not just a possible one. Advanced Level questions often have several plausible options. Choose the one that best fits the stated context, risk, budget and system type.
Tip 8: Do not confuse the oracle with test data or test basis. The oracle decides expected results. Generating inputs alone is not an oracle; checking properties is.
Tip 9: Combination answers are often correct when the scenario describes both large volume and subjective aspects. An example is automated property checks plus expert sampling.
Tip 10: Implicit oracles are not enough on their own. 'No crash' detects only gross failures. If asked how to verify correctness, choose a stronger oracle.
11. Sample Exam-Style Question
A Test Analyst is testing a new machine-learning image classifier. Correct labels for millions of newly captured images are not available. Which approach is MOST suitable for checking a large number of results automatically?
a) Ask domain experts to label every image
b) Apply metamorphic relations, such as checking that slightly rotating or brightening an image does not change its label
c) Rely on the absence of exceptions during classification
d) Compare results with the classifier's own previous outputs only
Answer: b.
- Option a does not scale.
- Option c is only an implicit, weak oracle.
- Option d may simply reproduce existing defects and is not independent.
- Metamorphic relations allow automated checking without knowing the exact labels.
12. Summary
The test oracle problem arises when expected results are hard to determine. The three main solutions each have a different strength:
- Pseudo-oracles provide independent comparison.
- Property-based and metamorphic testing check rules and relations at scale without exact expected values.
- Human oracles bring judgment for subjective or expert assessments.
Each has costs and limitations, and they are often combined. In the exam, identify the oracle difficulty in the scenario, match keywords to the technique, consider scale, cost and independence, and reject options that claim any oracle is complete or infallible.
