Test Case Quality Criteria: Correctness, Feasibility, Necessity, Traceability and Completeness (ISTQB CTAL-TA)
Introduction
In the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, the Test Analyst is responsible for designing and implementing test cases that are effective, efficient and maintainable. A test case is only as valuable as its quality. A poorly written test case can miss defects, waste execution effort, mislead stakeholders or become impossible to maintain.
This guide covers five quality criteria used to judge whether a test case is fit for purpose: Correctness, Feasibility, Necessity, Traceability and Completeness. These criteria appear in the syllabus areas on test design and test implementation, and in discussions of reviewing testware. They also frequently appear in scenario-based exam questions.
Why Test Case Quality Criteria Are Important
Test cases are work products. Like requirements, designs and code, they can contain defects. If a test case is defective, the test that relies on it is unreliable. The consequences include:
- False confidence: incorrect expected results may let defects pass unnoticed (false negatives).
- Wasted effort: unnecessary or duplicate test cases consume execution time and maintenance budget without adding value.
- Blocked execution: infeasible test cases cannot be run because the environment, data or tools are missing.
- Poor coverage visibility: without traceability, the team cannot show which requirements or risks have been tested, which undermines test reporting and impact analysis.
- Inconsistent execution: incomplete test cases (missing preconditions, data or expected results) lead different testers to run them differently and report different outcomes.
Applying these criteria supports the goals of the test process: building confidence in the test object, finding defects, giving stakeholders accurate information and making the best use of limited test resources. It also supports reviews of testware, which the Test Analyst often takes part in.
What the Criteria Are
1. Correctness
A test case is correct when it accurately reflects the test basis and checks what it is supposed to check. This means:
- The expected results are right according to the requirements, user stories, specifications or other test oracles.
- The input values, preconditions and steps are valid for the condition being tested.
- The test case actually exercises the intended test condition (for example, a boundary value test really sits on the boundary).
- The test case has no logical errors, ambiguity or contradictions.
Example: A requirement states that orders above 100 EUR receive free shipping. A test case expecting free shipping at exactly 100 EUR is incorrect if the requirement says "above" rather than "100 or more".
2. Feasibility
A test case is feasible when it can actually be executed in the planned test environment, with the available resources, within the constraints of the project. Feasibility considers:
- Availability of the test environment, tools, test data and interfaces (or suitable stubs, drivers and simulators).
- Required skills and access rights.
- Time and cost constraints.
- Whether the expected result can actually be observed or measured (testability).
Example: A test case that needs 10,000 concurrent real users on a production-like environment is not feasible if only a small integration environment exists and no load-generation tool is available. It may need to be redesigned, moved to another test level, or flagged as a risk.
3. Necessity
A test case is necessary when it adds value. It should cover a test condition, risk or requirement that is not already adequately covered by other test cases. Necessity is about avoiding:
- Redundant or duplicate test cases.
- Test cases that test low-value or out-of-scope features.
- Test cases whose effort is out of proportion to the risk they address.
Necessity links closely to risk-based testing. The depth and number of test cases should reflect the level of product risk. A Test Analyst should ask: "If we removed this test case, would we lose meaningful coverage or information?"
Example: Ten test cases that all enter different valid values from the same equivalence partition, with no further distinguishing factor, are largely redundant. One or two representative values are usually enough.
4. Traceability
A test case is traceable when it can be linked in both directions to the test basis (requirements, user stories, risks, acceptance criteria) via test conditions, and onward to test results and defects. Traceability enables:
- Coverage measurement and reporting (which requirements and risks are tested, passed, failed or not run).
- Impact analysis when requirements change: identifying which test cases must be updated or rerun.
- Auditability and compliance, which are especially important in safety-critical or regulated domains.
- Justification of each test case's existence, which supports the necessity criterion.
Example: Test case TC-045 is linked to test condition TCond-12, which traces to user story US-7 and risk R-3. When US-7 changes, the team can immediately see that TC-045 needs review.
5. Completeness
Completeness has two dimensions:
- Completeness of an individual test case: it contains all the information needed to execute it and evaluate the outcome. This typically includes an identifier, objective, preconditions, test data, steps (at the appropriate level of detail), expected results, postconditions and priority.
- Completeness of the set of test cases: together, the test cases cover the relevant test conditions to the agreed coverage criteria. This includes positive and negative paths, boundaries and relevant quality characteristics.
Note that the right level of detail depends on context. High-level (logical) test cases suit experienced testers and changing requirements. Low-level (concrete) test cases suit less experienced testers, regulated environments or automation. A test case is complete when it holds the information needed for its context. It does not have to contain the maximum possible detail.
Example: A test case for a password reset that omits the precondition "user account exists and is active" is incomplete, because testers may get inconsistent results.
How the Criteria Work Together
The criteria are interdependent and sometimes in tension:
- Traceability supports necessity: a test case that traces to no requirement or risk may be unnecessary, or may reveal a gap in the test basis.
- Completeness vs necessity: aiming for full coverage can create redundant tests. Risk-based prioritisation balances the two.
- Correctness vs feasibility: a test case can be logically correct but impossible to execute. Both must hold.
- Completeness supports correctness: missing expected results make it impossible to judge whether the test passed.
In practice, the Test Analyst applies these criteria during:
- Test analysis: identifying and prioritising test conditions, which supports traceability and necessity.
- Test design: applying test techniques to derive test cases, which supports correctness and completeness of coverage.
- Test implementation: preparing data, environments and procedures, which confirms feasibility and individual completeness.
- Reviews of testware: using checklists that cover these criteria to find defects in test cases before execution.
Practical Checklist for Reviewing a Test Case
- Correctness: Are the expected results consistent with the test basis? Is the technique applied properly?
- Feasibility: Can it be executed with the available environment, data, tools, skills and time?
- Necessity: Does it address a risk or condition not already covered? Is its priority justified?
- Traceability: Is it linked to test conditions, requirements and risks? Can results and defects be linked back?
- Completeness: Does it include preconditions, data, steps, expected results and postconditions at the right level of detail? Does the set cover all required conditions?
Exam Tips: Answering Questions on Test Case Quality Criteria: Correctness, Feasibility, Necessity, Traceability and Completeness
1. Learn the keyword triggers. Exam scenarios often hint at the criterion through specific wording:
- "expected result is wrong", "does not match the specification", "misinterpreted the requirement" point to Correctness.
- "cannot be executed", "environment not available", "no test data", "tool missing", "too expensive" point to Feasibility.
- "duplicate", "redundant", "already covered", "low risk with high effort", "adds no value" point to Necessity.
- "cannot determine which requirement", "impact of change unknown", "coverage cannot be reported", "audit" point to Traceability.
- "missing preconditions", "no expected result", "testers obtain different outcomes", "untested conditions", "gaps in coverage" point to Completeness.
2. Separate correctness from completeness. If information is present but wrong, the issue is correctness. If information is absent, the issue is completeness. This is a classic distractor.
3. Separate necessity from traceability. A test case with no link to any requirement is primarily a traceability issue. If the scenario emphasises that it duplicates other tests or wastes effort, the issue is necessity. Read which consequence the question highlights.
4. Remember that feasibility is about execution, not logic. A perfectly correct test case can still be infeasible. If the problem lies in resources, environment, tools, data or time, choose feasibility.
5. Context determines the right level of detail. Do not automatically choose the answer that says "add more detail". The syllabus emphasises that the level of detail depends on factors such as tester experience, regulatory requirements, automation and stability of the test basis. A high-level test case can still be complete for an experienced team.
6. Link answers to risk. When asked how to decide whether to keep, drop or prioritise test cases, answers based on product risk and coverage objectives are usually correct. Answers like "run everything" or "remove all low-priority tests" usually are not.
7. Watch for "MOST" and "BEST" in questions. Several options may be partly true. Choose the one that most directly addresses the stated problem. For example, if testers report inconsistent results, the best fix is making preconditions and expected results explicit (completeness). Adding traceability links would not fix that problem.
8. Know the benefits of traceability. Expect questions asking which activity is enabled by traceability. Common correct answers are impact analysis, coverage reporting, auditability and identifying gaps in the test basis.
9. Consider who finds the issue and when. Reviews of test cases (static testing of testware) are the typical way to detect violations of these criteria early. If a question asks how to ensure test case quality before execution, reviews with checklists are a strong candidate.
10. Eliminate absolute statements. Options containing "always", "never" or "all test cases must" are often wrong in ISTQB exams, because good test case quality depends on context.
Worked Example Question
A Test Analyst reviews a test suite and finds that several test cases for the login function use different valid usernames from the same equivalence partition. None of them checks the account lockout rule described in the requirements. Which TWO quality criteria are MOST affected?
a) Correctness and Feasibility
b) Necessity and Completeness
c) Traceability and Correctness
d) Feasibility and Traceability
Answer: b. The redundant tests from the same partition affect necessity. The missing lockout coverage affects completeness of the test set. Nothing in the scenario suggests wrong expected results, execution problems or missing links.
Summary
- Correctness: the test case is right according to the test basis.
- Feasibility: the test case can actually be executed.
- Necessity: the test case adds value and is not redundant.
- Traceability: the test case is linked to its test basis, risks, results and defects.
- Completeness: the test case contains all the information it needs, and the set of test cases covers the required conditions.
Master the keyword triggers, distinguish closely related criteria, and always relate your answer to context and risk. This approach will help you handle CTAL-TA exam questions on test case quality with confidence.
