Evaluating the Test Basis During Test Analysis (ISTQB CTAL-TA)
Introduction
In the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, test analysis is the activity where the Test Analyst works out what to test. A key task within it is evaluating the test basis. This means examining the source documents and artifacts that tests will be derived from, to check that they are good enough to test against. This guide explains what the activity is, why it matters, how it works in practice, and how to answer exam questions on it.
What Is the Test Basis?
The test basis is the body of knowledge used as the foundation for test analysis and design. Typical examples:
- Requirements specifications (business, functional and non-functional requirements)
- User stories with their acceptance criteria
- Use cases
- Architecture and design documents
- Interface specifications
- Business process models and workflow diagrams
- Risk analysis reports
- Regulatory and contractual documents
- The code itself, existing systems, or legacy behavior (where documentation is missing)
- Domain knowledge and stakeholder expertise
What Does Evaluating the Test Basis Mean?
During test analysis, the Test Analyst evaluates the test basis to judge whether it is suitable for deriving test conditions. The evaluation has two purposes:
1. Finding defects in the test basis itself. Common problems include ambiguities, omissions, inconsistencies, contradictions, inaccuracies, redundancies and untestable statements.
2. Assessing testability. This means checking that the test basis is precise enough that test conditions and expected results can be derived from it.
The syllabus emphasizes that evaluation can be done using review techniques and by applying test techniques analytically. Analytical approaches include:
- Building decision tables
- Drawing state transition models
- Identifying equivalence partitions and boundaries
- Creating use case flows
Doing this often reveals gaps such as missing combinations of conditions, undefined states or unclear boundaries.
Why Is It Important?
- Early defect detection (shift left): Defects found in requirements are far cheaper to fix than those found in code or production. This is the ISTQB principle that early testing saves time and money.
- Better test conditions: Weak test bases lead to weak tests. If requirements are ambiguous, tests may verify the wrong behavior.
- Defined expected results: Without clear specifications there is no reliable test oracle, so tests cannot determine pass or fail.
- Traceability: A well-evaluated test basis supports clear traceability between requirements, test conditions and test cases. This aids coverage measurement and impact analysis.
- Risk reduction: Gaps in the test basis often correspond to areas of product risk.
- Stakeholder alignment: Questions raised by the Test Analyst help business, development and testing reach a common understanding.
How It Works: The Process in Practice
Step 1: Identify and gather the test basis. Determine which documents and sources apply to the test level and test object, for example system requirements for system testing or interface specifications for integration testing.
Step 2: Analyze for quality characteristics. Check each item against these qualities:
- Complete: Are all situations, inputs, error conditions and non-functional aspects covered?
- Consistent: Do statements contradict each other or other documents?
- Unambiguous: Can the statement be interpreted only one way?
- Correct: Does it reflect stakeholder needs?
- Testable: Can an objective expected result be defined? A statement like 'the system shall be fast' is not testable. 'Responds within 2 seconds for 95% of requests' is testable.
- Feasible and relevant: Can it actually be implemented and does it belong in scope?
Step 3: Apply test techniques analytically. Drafting partitions, decision tables or state models from the specification exposes missing rules. For example, creating a decision table may reveal that the specification never says what happens when a customer is both a premium member and has an overdue balance.
Step 4: Use review techniques. The Test Analyst may take part in informal reviews, walkthroughs, technical reviews or inspections. Supporting tools include checklists such as requirements checklists, user story checklists (for example the INVEST criteria) and acceptance criteria checks.
Step 5: Report and communicate findings. Defects and questions are raised with authors and stakeholders, for example business analysts and product owners. In Agile contexts this often happens during backlog refinement, three amigos sessions, or acceptance-test-driven development discussions.
Step 6: Derive and prioritize test conditions. Once the basis is sufficiently clear, the Test Analyst identifies test conditions. These are prioritized using risk analysis and linked to the test basis for traceability.
Context Matters
- Sequential lifecycles (e.g. V-model): Evaluation typically happens when formal specifications become available, often through formal reviews.
- Iterative/Agile lifecycles: Evaluation is continuous and collaborative. User stories and acceptance criteria are refined just in time, and the Test Analyst contributes to clarifying them before or during the iteration.
- Weak or missing documentation: The Test Analyst may rely on experience, interviews, existing system behavior, or exploratory sessions. The lack of a formal test basis itself is a risk to be communicated.
- Safety-critical or regulated domains: Rigorous, documented evaluation and bidirectional traceability are often mandatory.
Relationship to Other Test Analysis Activities
Evaluating the test basis sits alongside these other analysis tasks:
- Identifying features and sets of features to be tested
- Defining and prioritizing test conditions
- Capturing bi-directional traceability between the test basis and test conditions
The ISTQB view is that analysis answers 'what to test' and design answers 'how to test'. Evaluating the test basis supports both by ensuring there is a sound foundation.
Worked Example
Requirement: 'Users who are over 18 can open an account. Users under 18 need parental consent.'
When evaluating this requirement, the Test Analyst notices:
- What about users who are exactly 18? This is a boundary gap.
- How is age verified? This is an omission.
- What does parental consent consist of, and how is it captured? This is ambiguous.
- What error message appears for invalid ages, such as negative values or more than 150? This is a missing error handling definition.
Applying equivalence partitioning and boundary value analysis analytically exposed these defects before any code was written. The Test Analyst raises them with the business analyst. Once the requirement is clarified, precise test conditions can be defined.
Exam Tips: Answering Questions on Evaluating the Test Basis During Test Analysis
1. Know the purpose. Expect the correct answer to mention identifying defects in the test basis (ambiguities, omissions, inconsistencies) and assessing testability. Answers focused on executing tests or measuring code coverage are distractors.
2. Recognize the defect types. Scenario questions often give a requirement and ask what is wrong with it. Learn to spot:
- Ambiguous wording, such as 'user-friendly', 'quickly' or 'appropriate'
- Missing boundaries or error handling
- Contradictions between documents
- Untestable statements with no measurable criteria
3. Remember that test techniques can evaluate the test basis. A common exam point is that applying black-box techniques such as decision tables or state transitions during analysis reveals defects in specifications. If an option says 'creating a decision table revealed missing rules in the requirements', it is likely correct.
4. Choose communication over assumption. When asked what the Test Analyst should do on finding an ambiguity, the best answer is usually to raise it with the relevant stakeholder (author, business analyst or product owner) and get clarification. Making assumptions, ignoring the issue, or rewriting the requirement alone are typically wrong.
5. Link to early testing and cost. If a question asks why this activity is valuable, look for answers about finding defects early, reducing rework costs and preventing defects from propagating into design and code.
6. Distinguish analysis from design and implementation. Evaluating the test basis and identifying test conditions belong to test analysis. Creating test cases with concrete data belongs to test design. Preparing test procedures and environments belongs to test implementation. Exam distractors often mix these phases.
7. Consider the lifecycle context. In Agile scenarios, prefer answers involving collaboration with the team. Examples include refining acceptance criteria, using INVEST, or participating in backlog refinement. In sequential scenarios, formal reviews of specifications are more likely.
8. Watch for testability keywords. The ability to define expected results objectively is central. An option stating that the requirement lacks measurable acceptance criteria, so expected results cannot be determined, often signals the correct choice.
9. Traceability matters. Answers that maintain traceability between test basis items and test conditions are generally favored. Traceability enables coverage evaluation and impact analysis when the test basis changes.
10. Read K-level cues. This topic may appear at K2 (understand/explain) or K3/K4 (apply/analyze) levels. For K3/K4 questions, mentally apply a technique to the given requirement to discover the gap. Do not simply pick the most generic-sounding answer.
11. Eliminate extreme options. Choices like 'testing cannot start until the test basis is perfect' are usually wrong. Choices like 'the Test Analyst must not question requirements' are also usually wrong. The realistic answer is to identify issues, communicate them, manage the related risk and proceed appropriately.
Summary
Evaluating the test basis is a proactive quality activity performed by the Test Analyst during test analysis. It involves:
- Reviewing requirements, user stories, designs and other sources
- Finding defects such as ambiguity, incompleteness, inconsistency and untestability
- Using both review techniques and the analytical application of test techniques
Doing so prevents costly defects, produces stronger test conditions, enables reliable expected results and supports traceability. In the exam, focus on the purpose, the defect types, the use of techniques to expose gaps, communication with stakeholders, and the correct placement of this activity within the test process.
