Measuring Defect Prevention: Defect Removal Efficiency and Phase Containment Effectiveness

Measuring Defect Prevention: Defect Removal Efficiency (DRE) and Phase Containment Effectiveness (PCE) – ISTQB CTAL-TA Guide

Introduction
In the ISTQB Certified Tester Advanced Level Test Analyst (CTAL-TA) syllabus, Software Defect Prevention covers how a test analyst helps the organisation avoid defects and catch them as early as possible. You cannot improve what you cannot measure. Defect Removal Efficiency (DRE) and Phase Containment Effectiveness (PCE) are two classic metrics for this. They show how well the process finds defects before they escape to later phases or to the customer.

1. Why It Is Important

• Cost of defects rises over time: A defect fixed in the requirements review that introduced it is far cheaper than the same defect found in system test or in production. Rework, retesting, regression and reputation damage all add cost.
• Evidence for process improvement: DRE and PCE show which activities work, such as reviews, static analysis and unit testing, and which let defects slip through.
• Supports shift-left: Low containment in early phases is a signal to invest in reviews, better acceptance criteria, ATDD/BDD, static analysis or developer testing.
• Feeds root cause analysis: Escaped defects are prime candidates for root cause analysis. That analysis drives preventive actions.
• Objective reporting: These metrics give stakeholders a quantitative view of quality and of test and review effectiveness. This is better than relying on raw defect counts, which can mislead.

2. What It Is
Key terms

• Phase (or activity) of introduction / injection: where the defect was created, such as requirements, design, coding or configuration.
• Phase of detection: where the defect was found, such as requirements review, design review, code review, unit test, system test, acceptance test or production.
• Phase containment: a defect is contained when it is detected and removed in the same phase in which it was introduced.
• Escaped (leaked) defect: a defect that passes out of its phase of introduction undetected and is found later.

Defect Removal Efficiency (DRE)
DRE measures the proportion of defects removed by an activity, or by the whole development process, relative to all defects that were present and could have been found.
Overall (release-level) DRE:
DRE = Defects found before release / (Defects found before release + Defects found after release) x 100%
Activity / phase-level DRE:
DRE(phase) = Defects found in the phase / (Defects found in the phase + Defects that were present in that phase but found later) x 100%
ISTQB also uses the closely related term Defect Detection Percentage (DDP). For a test level, DDP = defects found by that test level / (defects found by that level + defects found afterwards). In exam questions, treat DDP and test-level DRE as the same calculation.

Phase Containment Effectiveness (PCE)
PCE focuses on the origin of defects. It measures the proportion of defects introduced in a phase that were also caught in that same phase.
PCE(phase) = Defects introduced AND found in the phase / Total defects introduced in the phase (wherever they were eventually found) x 100%
A high PCE means defects do not leak out of the phase where they were created.

Key difference

• DRE asks: of everything that was present in this phase (including defects inherited from earlier phases), how much did we remove?
• PCE asks: of everything this phase created, how much did we catch before it left the phase?

3. How It Works – A Worked Example
To calculate these metrics, every defect needs two attributes: the phase introduced, usually determined during analysis or root cause analysis, and the phase found. Arrange the data in a defect containment matrix with rows for phase introduced and columns for phase found.

Phase introduced, then defects found per phase

• Requirements (total 40): Req review 20, Design review 5, Code/unit 3, System test 8, Production 4
• Design (total 30): Design review 15, Code/unit 4, System test 6, Production 5
• Code (total 70): Code/unit 40, System test 25, Production 5
• Column totals: Req 20, Design 20, Code 47, System test 39, Production 14. Grand total 140.

Step 1 – Overall DRE
Defects found before release = 140 - 14 = 126.
DRE = 126 / 140 = 90%.

Step 2 – PCE per phase (diagonal value divided by row total)

• Requirements PCE = 20 / 40 = 50%
• Design PCE = 15 / 30 = 50%
• Code PCE = 40 / 70 = 57.1%

Step 3 – Phase DRE (defects found in the phase divided by defects present in the phase)

• Design review:
  • Found = 5 + 15 = 20.
  • Present = requirements defects that escaped the requirements review (40 - 20 = 20), plus design defects (30), giving 50.
  • DRE = 20 / 50 = 40%.
• Code/unit:
  • Found = 3 + 4 + 40 = 47.
  • Present = requirements defects still open (40 - 20 - 5 = 15), plus design defects still open (30 - 15 = 15), plus code defects (70), giving 100.
  • DRE = 47 / 100 = 47%.
• System test:
  • Found = 8 + 6 + 25 = 39.
  • Present = 39 found + 14 later found in production = 53.
  • DRE (or DDP) = 39 / 53 = 73.6%.

Step 4 – Interpret

• Half of the requirements defects escaped their phase. Twelve of them escaped all the way to system test or production, which is costly.
• Typical preventive actions would be better requirements reviews, checklists, acceptance criteria workshops, prototyping or example mapping.
• Design review removed only 40% of the defects available to it. Its review technique, preparation time or reviewer selection should be examined.
• Root cause analysis of the 14 production defects should feed back into process improvement.

Practical considerations and limitations

• Origin data is required: Without a reliable phase-introduced value, PCE cannot be calculated. Defect reports need this field, and analysts must classify defects consistently.
• Time lag: Post-release defects keep arriving. DRE is usually measured after a fixed window, for example 3 or 6 months in production. Early figures overstate DRE.
• Unknown defects: The true total is never known. DRE is an estimate based on defects found so far.
• Severity weighting: Ten trivial escapes are not equal to one critical escape. Consider separate metrics by severity.
• Agile and iterative contexts: Phases may be replaced by activities or iterations, such as refinement, story review, developer testing, sprint testing and release. The same logic applies.
• Avoid misuse: These metrics evaluate processes, not individuals. Using them to blame people encourages hiding or misclassifying defects.
• Duplicates and rejected reports: Exclude these so the counts are not inflated.

4. How the Test Analyst Supports Phase Containment

• Participating in early reviews of requirements, user stories and acceptance criteria.
• Applying checklists, often derived from past escaped-defect patterns.
• Recording accurate defect data: phase found, suspected phase introduced, root cause category and severity.
• Contributing to root cause analysis and defect clustering.
• Recommending improvements when containment is low, and tracking whether DRE and PCE improve afterwards.

5. Exam Tips: Answering Questions on Measuring Defect Prevention: DRE and PCE

• Identify the metric asked for: Is the question about removal at a level or phase (DRE/DDP), or about defects contained where introduced (PCE)? Underline the words introduced and found.
• PCE uses the row: The numerator is defects introduced AND found in the same phase. The denominator is ALL defects introduced in that phase, including those found later and in production.
• DRE/DDP uses what was present: The numerator is defects found in the phase or level. The denominator is those defects plus every defect that was present but found afterwards.
• Overall DRE: pre-release defects divided by (pre-release + post-release defects). Memorise this form.
• Distractor traps:
  • Dividing by the grand total instead of the phase total.
  • Forgetting production defects in the denominator.
  • Including defects found in earlier phases in a later phase's denominator.
  • Swapping numerator and denominator, which gives a result over 100%. A value above 100% is impossible.
• Sanity-check with a quick estimate: If 90 of 100 defects were found before release, DRE must be about 90%.
• Rounding: Do the division accurately and choose the closest option. Options are often designed so that wrong denominators give plausible-looking numbers.
• Draw a mini matrix: In scenario questions, quickly sketch introduced versus found, even in rough form. It prevents most errors.
• Interpretation questions:
  • Low PCE in requirements or design points to improving reviews and static testing.
  • Low DRE in system testing points to improving test design techniques, coverage or risk-based prioritisation.
  • Many production escapes call for root cause analysis.
• Know the limitations: Questions may ask why a metric is unreliable. Typical reasons are missing phase-of-origin data, short post-release measurement windows, misclassification, ignoring severity, or using metrics to judge individuals.
• Link to cost: When asked why phase containment matters, the expected answer is that defects become more expensive to fix the later they are found. Early containment reduces cost and rework.
• Terminology: ISTQB may use defect detection percentage, escaped defects, defect leakage or containment. Map them correctly: leakage is the complement of containment (100% - PCE for a phase).

Quick Recap

• DRE = found / (found + found later). It measures how efficient an activity or process is at removing the defects present.
• PCE = introduced-and-found-in-phase / introduced-in-phase. It measures how well a phase catches its own defects.

Both rely on good defect data and on root cause analysis. Both drive defect prevention and shift-left improvements. In the exam, focus on the correct denominator, read the scenario carefully, and interpret results in terms of process improvement.