Scenario Coverage and Prioritizing Scenarios by Risk (ISTQB CTAL-TA Behavior-Based Test Techniques)
Overview
In the ISTQB Certified Tester Advanced Level Test Analyst (CTAL-TA) syllabus, scenario-based testing is one of the behavior-based test techniques, alongside techniques such as CRUD testing and state transition testing. It focuses on how users or other systems interact with the test object to reach a goal. Two ideas sit at the heart of the technique:
1. Scenario coverage: how you measure the thoroughness of scenario-based testing.
2. Prioritizing scenarios by risk: how you decide which scenarios to design, implement and run first, and how deeply to test them, when time is limited.
Why It Is Important
- Business focus: Scenarios mirror real user workflows, so defects found here are often the ones users would actually hit.
- Measurable thoroughness: Scenario coverage gives a clear exit criterion and a basis for progress reporting, for example 'all main and alternative scenarios covered, 80% of exception scenarios covered'.
- Limited resources: Real systems can have a very large number of scenarios. Risk-based prioritization makes sure the scenarios with the highest risk are tested first and most thoroughly.
- Alignment with risk-based testing: The Test Analyst identifies and assesses product risks. Ranking scenarios by risk applies that work directly to test design.
- Stakeholder communication: Coverage and priority give business stakeholders a language they understand, such as 'all checkout flows are tested'.
What It Is
Scenarios are sequences of interactions between actors (users or systems) and the test object. They are typically derived from use cases, user stories, business process models (for example BPMN), activity diagrams or workflow descriptions. They usually fall into three types:
- Main (basic) scenario: the typical, most common path to the goal, often called the happy path.
- Alternative scenarios: valid variations that still reach the goal by a different route, for example paying by voucher instead of by card.
- Exception (error) scenarios: paths where something goes wrong and the system must handle it, for example a payment being declined, a timeout or invalid data.
Scenario coverage is the coverage criterion for this technique. The coverage items are the scenarios themselves:
Scenario coverage = (number of scenarios exercised by tests / total number of identified scenarios) x 100%
The main and alternative scenarios are sometimes reported separately from the exception scenarios to give a more meaningful picture.
Prioritizing scenarios by risk means ranking scenarios by their risk level. Risk level is usually a combination of likelihood (the chance that the scenario contains a defect) and impact (the consequence if it fails). This ranking then drives:
- the order in which tests are designed and executed;
- how thoroughly each scenario is tested, for example adding data variations or boundary values;
- which scenarios may be omitted or deferred when time runs out.
How It Works
Step 1: Identify the scenarios. Analyze the test basis, such as a use case flow or a process model. List the main flow, each alternative flow and each exception flow. Each flow becomes a coverage item. Loops and branch points may create many paths, so agree on a practical rule, for example 'each loop executed zero, one and many times' or 'each alternative covered at least once'.
Step 2: Assess the risk of each scenario.
Likelihood factors include:
- complexity of the flow;
- new or changed functionality;
- integration points with other systems;
- inexperienced developers;
- areas with a history of defects.
Impact factors include:
- frequency of use (how many users follow this path);
- financial loss;
- safety or legal consequences;
- reputational damage;
- availability of workarounds.
Combine the two into a risk level (high, medium or low, or a numeric score such as likelihood x impact).
Step 3: Rank and allocate effort.
- High-risk scenarios: tested first, with more test cases and more data variations.
- Low-risk scenarios: tested later, lightly, or deferred.
The main scenario is often high priority because it is used most frequently, so its failure affects most users. However, an exception scenario with severe impact, such as double charging a customer or a safety-related error handler, can rank as high or higher.
Step 4: Design test cases. Typically at least one test case is designed per scenario. Additional test cases can be added for high-risk scenarios using other techniques, such as equivalence partitioning or boundary value analysis, on the data used within each step.
Step 5: Measure and report coverage. Track the coverage achieved against the target. Report it together with the residual risk of any scenarios that are not covered.
Worked Example
An online checkout use case has the following scenarios:
- S1 Main: pay by card successfully.
- S2 Alternative: pay by voucher.
- S3 Alternative: split payment between voucher and card.
- S4 Exception: card declined.
- S5 Exception: payment gateway timeout.
- S6 Exception: voucher expired.
The risk assessment gives:
- S1: high frequency and high impact, so high risk.
- S5: complex integration and possible double charging, so high risk.
- S3: new and complex feature, so medium-high risk.
- S4: common case with medium impact, so medium risk.
- S2: medium risk.
- S6: rare, with a simple workaround, so low risk.
The execution order is therefore S1, S5, S3, S4, S2, S6.
If time allows only S1, S5, S3 and S4 to be tested, then scenario coverage = 4/6 = 66.7%. The residual risk lies mainly in S2 and S6, which should be reported to stakeholders.
Relationship to Other Concepts
- Scenario-based testing complements state transition testing (system states) and CRUD testing (data lifecycle).
- It is well suited to system testing, system integration testing and acceptance testing, where end-to-end workflows matter.
- Risk-based prioritization of scenarios is a specific application of the general risk-based testing approach in the syllabus.
- 100% scenario coverage does not guarantee that all data combinations, all states or all code are tested. It only means each identified scenario was exercised at least once.
Exam Tips: Answering Questions on Scenario Coverage and Prioritizing Scenarios by Risk
1. Know the coverage items. In scenario-based testing the coverage items are the scenarios (main, alternative and exception). Do not confuse them with states, transitions, decision table rules or code statements.
2. Calculate carefully. Count all identified scenarios, including the main one, in the denominator. Watch out for scenarios that are duplicated or that one test case covers together with another. One test case usually covers one scenario, so count distinct scenarios exercised, not test cases executed.
3. Minimum number of test cases. To achieve 100% scenario coverage you need at least as many test cases as there are distinct scenarios. Read whether the question allows one test to cover several paths.
4. Risk = likelihood x impact. When a question gives frequency of use, business criticality, complexity or defect history, map each factor to likelihood or impact. The highest combined value is ranked first.
5. Do not assume the happy path always wins. The main scenario is often top priority because of frequency, but an exception scenario with severe consequences (financial, safety, legal) may be ranked higher. Read the stem for impact clues.
6. Look for keywords. Phrases like 'most frequently used', 'new', 'complex', 'integrates with a third party' or 'regulatory' signal higher risk. Phrases like 'rarely used', 'workaround available' or 'unchanged since last release' signal lower risk.
7. Limited-time questions. If asked which scenarios to test given limited time, choose the option that covers the highest-risk scenarios first. Do not pick the option that simply maximizes the count of scenarios covered.
8. Residual risk. Remember that untested scenarios represent residual risk that should be reported. An answer mentioning reporting uncovered high-risk scenarios is usually correct.
9. Eliminate distractors. Reject answers that say 100% scenario coverage means the system is defect-free or fully tested. Also reject answers that apply code coverage measures to scenario-based testing.
10. Scenario-type identification. Practice classifying flows. 'The user cancels' or 'the server is unavailable' are usually exception scenarios. 'The user chooses another valid option' is an alternative scenario.
11. Apply-level questions (K3). Expect a use case or process description from which you must derive the scenarios, compute coverage and rank them. Sketch the flows on scratch paper and label them M, A1, A2, E1 and so on before answering.
Summary
Scenario coverage measures how many identified main, alternative and exception scenarios have been exercised. Prioritizing scenarios by risk ensures that the scenarios most likely to fail and most damaging if they fail are tested first and most thoroughly. In the exam:
- identify the scenarios correctly;
- calculate coverage using distinct scenarios;
- rank scenarios by combining likelihood and impact;
- always consider and report residual risk.
