Defect Classification Methods: Orthogonal Defect Classification, IEEE 1044 and Defect Taxonomies

Defect Classification Methods: Orthogonal Defect Classification (ODC), IEEE 1044 and Defect Taxonomies (ISTQB CTAL-TA Software Defect Prevention)

Introduction
Within the Software Defect Prevention area of the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, classifying defects is the link between finding defects and preventing them. A defect that is only logged, fixed and closed teaches the organisation nothing. A defect that is classified with consistent attributes becomes data. That data can be analysed to find patterns, root causes and weak process areas, and to improve test design.

This guide covers three methods you are expected to recognise and apply:
1. Orthogonal Defect Classification (ODC), developed at IBM.
2. IEEE 1044, the IEEE Standard Classification for Software Anomalies.
3. Defect taxonomies, such as those of Beizer, Kaner/Falk/Nguyen and organisation-specific lists.

1. Why Defect Classification Is Important
Root cause analysis (RCA): Classified defects let teams group similar defects and ask why they keep occurring. For example, 40% of defects introduced during requirements may point to a weak requirements review practice.

Defect prevention: Once a recurring cause is known, the process can be changed so that class of defect is not introduced again. Examples include adding checklists, training, static analysis rules or better review techniques.

Phase containment and early detection: Recording where a defect was introduced and where it was detected shows how many defects escape their phase. This supports metrics such as phase containment effectiveness and Defect Detection Percentage (DDP).

Test design and risk-based testing: Taxonomies of typical defects feed defect-based test techniques and checklist-based testing. They help the Test Analyst target likely failure areas.

Process improvement and comparison: Standardised classification allows comparison across projects, releases and teams. It provides the objective evidence needed for test and process improvement models.

Better communication and reporting: Common categories for severity, type and effect remove ambiguity between testers, developers, managers and customers.

Prediction and management: Defect distributions by type and phase can show whether a product is ready for release or whether a phase is under-performing.

2. What Defect Classification Is
Defect classification is the systematic assignment of attributes (categories with defined values) to each defect or anomaly. Attributes typically describe:
- what the defect is (type, mode)
- where it is (component, artifact)
- when it was introduced and detected (phase or activity)
- how it was detected (trigger, technique)
- what impact it has (severity, effect, impact on the customer)
- what was done about it (resolution, disposition)

A good classification scheme is:
- Simple and unambiguous: people classify consistently.
- Mutually exclusive: each defect fits one value per attribute.
- Complete: every defect can be classified, often with an 'Other' value.
- Actionable: the results lead to concrete improvements.
- Tailored: adapted to the organisation, domain and lifecycle.

3. Orthogonal Defect Classification (ODC)
Origin: Developed by Ram Chillarege and colleagues at IBM Research in the early 1990s. ODC aims to turn defect data into feedback about the development process. In this sense it is a measurement method, not just a list of categories.

Meaning of 'orthogonal': The attributes are independent of each other. Each captures a different dimension of the defect, and the values within an attribute do not overlap. The classification therefore describes the defect itself (its semantics), not opinions about it. This reduces subjectivity and lets the data be analysed statistically, for example by cross-tabulating trigger against defect type.

Two moments of classification:
(a) Opener section, recorded when the defect is found (usually by the tester):
- Activity: what was being done when the defect was found. Examples: design review, code inspection, unit test, function test, system test.
- Trigger: the environment or condition that caused the defect to surface. This is ODC's distinctive attribute. Triggers depend on the activity:
  - Review/inspection triggers: design conformance, logic/flow, backward compatibility, lateral compatibility, concurrency, internal document, language dependency, side effects, rare situation.
  - Function test triggers: coverage, variation, sequencing, interaction (also simple path and complex path for white-box testing).
  - System test triggers: workload/stress, recovery/exception, start-up/restart, hardware configuration, software configuration, blocked test.
- Impact: the effect on the customer had the defect escaped. Examples: capability, usability, performance, reliability, installability, maintainability, documentation, integrity/security, serviceability, migration, standards.

(b) Closer section, recorded when the defect is fixed (usually by the developer):
- Target: what was fixed, such as design, code, build/package or information/documentation.
- Defect Type: the nature of the correction. Values:
  - Assignment/Initialisation: a value set wrongly or not set.
  - Checking: missing or incorrect validation of data or conditions.
  - Algorithm/Method: efficiency or correctness problems in a procedure.
  - Function/Class/Object: a capability is missing or wrong and needs a design change.
  - Timing/Serialisation: shared resources or real-time behaviour.
  - Interface/O-O Messages: communication between components, modules or drivers.
  - Relationship: associations among procedures, data structures and objects.
  - Build/Package/Merge and Documentation in some versions.
- Qualifier: Missing, Incorrect or Extraneous.
- Age: base, new, rewritten or re-fixed code.
- Source: developed in-house, reused from a library, outsourced or ported.

How ODC is used:
- Defect type distributions map to development phases. A high share of Function defects found late suggests weak design. A rising share of Assignment and Checking defects is normal as code matures.
- Trigger distributions show how effective the testing is. If system test finds mainly simple coverage triggers, the earlier function test was weak. If no workload or stress triggers appear, stress testing may be missing.
- Cross-analysis, such as type by phase or trigger by activity, supports in-process feedback and release decisions. You do not have to wait for post-release defects.

Strengths: objective, process-oriented, statistically analysable, supports in-process feedback and rapid RCA.
Limitations: requires training and discipline, takes effort to classify, and needs enough defects for meaningful statistics. Values may need tailoring.

4. IEEE 1044 (IEEE Standard Classification for Software Anomalies)
Origin and versions: First published in 1993, with a companion guide, IEEE 1044.1. It was revised as IEEE 1044-2009. The standard gives a uniform approach to classifying anomalies found anywhere in the software lifecycle: requirements, design, code, test and operation.

Key terminology: IEEE 1044 uses anomaly as the umbrella term for any condition that deviates from expectations. The 2009 version distinguishes:
- Defect: an imperfection or deficiency in a work product.
- Failure: an event in which a system or component does not perform a required function.
- Fault: a manifestation of an error in software.
- Error: a human action that produces an incorrect result.
- Problem: difficulty or uncertainty experienced by people.
These definitions match the ISTQB chain: error (mistake) leads to defect (fault/bug), which leads to failure.

The 1993 classification process: four sequential steps, each with three activities (Record, Classify, Identify impact):
1. Recognition: the anomaly is observed.
2. Investigation: it is analysed to find causes and related issues.
3. Action: a resolution plan is made, and preventive actions are identified.
4. Disposition: the anomaly is closed, deferred, merged and so on.
Supporting data is recorded at each step.

IEEE 1044-2009 defect attributes:
Defect ID, Description, Status, Asset, Artifact, Version detected, Version corrected, Priority, Severity, Probability, Effect, Type, Mode, Insertion activity, Detection activity, Failure reference, Change reference and Disposition.

Important attribute values to know:
- Type: Data, Interface, Logic, Description, Syntax, Standards, Other.
- Mode: Wrong, Missing, Extra. This parallels ODC's Incorrect, Missing, Extraneous.
- Effect: Functionality, Usability, Security, Performance, Serviceability, Other.
- Insertion activity: where the defect was introduced, such as Requirements, Design, Coding, Configuration, Documentation.
- Detection activity: where it was found, such as Requirements, Design, Code, Supplier testing, Unit test, Integration test, System test, Acceptance test, Audit, Customer, Production.
- Severity (technical impact) and Priority (urgency of fixing) are kept separate.

IEEE 1044-2009 failure attributes:
Failure ID, Status, Title, Description, Environment, Configuration, Severity, Analysis, Disposition, Observed by, Opened by, Assigned to, Closed by, Date observed, Date opened, Date closed, Test reference, Incident reference, Defect reference and Failure reference.

How IEEE 1044 is used:
- As a template for defect report fields in defect management tools.
- To build phase containment matrices from Insertion activity against Detection activity.
- To support RCA by Type, Mode and Insertion activity.
- To standardise reporting across organisations and suppliers. The standard expects tailoring to the organisation.

Strengths: a recognised international standard, comprehensive, lifecycle-wide, clear terminology, supports tool configuration and contractual consistency.
Limitations: it can feel heavyweight, organisations often use only a subset, and it is less focused than ODC on statistically linking defect data to process stages.

5. Defect Taxonomies
Definition: A defect taxonomy is a hierarchical, systematic list of defect categories, often with subcategories and examples. It may be generic or organisation-specific, and is usually built from historical defect data, literature and experience.

Well-known taxonomies:
- Boris Beizer (Software Testing Techniques): a numbered hierarchy with top-level classes of Requirements, Features and Functionality, Structural, Data, Implementation and Coding, Integration, System and Software Architecture, and Test Definition and Execution defects.
- Kaner, Falk and Nguyen (Testing Computer Software): over 400 common software errors in categories such as user interface errors, error handling, boundary-related errors, calculation errors, initial and later states, control flow errors, errors in handling or interpreting data, race conditions, load conditions, hardware, source and version control, documentation and testing errors.
- Whittaker's attack-oriented lists and quality characteristic based taxonomies built on ISO/IEC 25010.
- Domain-specific taxonomies, for example for web security (OWASP Top 10), mobile apps or embedded systems.

How taxonomies are used:
1. Defect-based test design: The Test Analyst picks categories from the taxonomy and designs tests to reveal those defect types. Coverage is measured as the proportion of relevant taxonomy items addressed. This is the defect-based technique in the experience-based testing part of the CTAL-TA syllabus.
2. Checklists for reviews and checklist-based testing: taxonomy items become review checklist questions.
3. Root cause analysis and prevention: classifying historical defects by taxonomy shows clusters, often visualised with a Pareto chart. Prevention actions are then targeted at the dominant categories.
4. Risk analysis: categories that occurred often in the past indicate likely risk areas.
5. Training and exploratory testing: taxonomies guide test charters and help less-experienced testers.

Building and maintaining a taxonomy:
- Define the goal, such as test design, RCA or both.
- Choose the level of detail: too coarse gives no insight, too fine gives inconsistent use.
- Start from a published taxonomy and tailor it with historical project defects.
- Validate it with stakeholders and pilot it.
- Review and update it regularly as technologies and processes change.

Strengths: practical and easy to understand, directly supports test design, reusable, and captures organisational knowledge.
Limitations: may be incomplete or out of date, categories may overlap (not orthogonal), and results depend on tester experience. Generic taxonomies may not fit a specific domain.

6. How Classification Supports Defect Prevention
1. Defects are recorded with consistent attributes, such as an IEEE 1044 based defect report.
2. Defects are classified using a chosen scheme: ODC, IEEE 1044 attributes or a taxonomy.
3. Data is analysed with distributions, Pareto charts, phase containment matrices, trend charts and cross-tabulations.
4. Root causes are identified for significant clusters, using techniques such as five whys or Ishikawa (fishbone) diagrams.
5. Preventive actions are defined. Examples: requirement review checklists, coding standards, static analysis rules, training, new test techniques, earlier test levels.
6. The effect is measured in later releases to confirm improvement. This closes the loop.

The Test Analyst's role is to:
- supply accurate, complete and consistently classified defect data
- contribute domain and business insight to RCA
- use classification results to improve test design, for example by adding tests for frequently escaping categories
- help maintain the defect taxonomies and checklists

7. Comparison at a Glance
ODC: IBM origin. Orthogonal, independent attributes split into opener (Activity, Trigger, Impact) and closer (Target, Type, Qualifier, Age, Source). Its main purpose is in-process feedback and statistical process analysis. Its key unique concept is the Trigger.

IEEE 1044: an international standard. A comprehensive attribute set for defects and failures, with standard terminology (anomaly, defect, failure, fault, error, problem). Its main purpose is consistent recording and reporting across the lifecycle. Key concepts are Insertion activity versus Detection activity, Type, Mode and Effect, plus the 1993 process of Recognition, Investigation, Action, Disposition.

Defect taxonomies: a hierarchical list of defect categories, such as Beizer or Kaner. Their main purpose is test design (defect-based techniques), checklists and RCA. The key concept is the list of likely defects used to target testing.

Exam Tips: Answering Questions on Defect Classification Methods: Orthogonal Defect Classification, IEEE 1044 and Defect Taxonomies
Tip 1: Spot the keywords.
- Trigger, opener/closer, orthogonal, IBM, in-process feedback, defect type distribution by phase point to ODC.
- Anomaly, standard attributes, insertion activity, detection activity, Mode (Wrong/Missing/Extra), Recognition-Investigation-Action-Disposition point to IEEE 1044.
- Hierarchical list, Beizer, Kaner, defect-based test technique, checklist, likely defects point to a defect taxonomy.

Tip 2: Know what 'orthogonal' means. It means independent, non-overlapping attributes and values. Distractors often say orthogonal means 'covers all test levels' or 'ranked by severity'. Both are wrong.

Tip 3: Separate the Trigger from the Activity and the Impact. The Activity is what you were doing, such as system test. The Trigger is the condition that exposed the defect, such as workload/stress or recovery/exception. The Impact is the effect on the customer. Scenario questions often test this distinction.

Tip 4: Link classification to its purpose. If a question asks how to prevent defects, the right answer usually involves analysing classified data (patterns or root causes) and changing the process. Fixing the individual defect faster is not prevention. If a question asks how to design tests from historical defects, the answer is usually a defect taxonomy with the defect-based technique.

Tip 5: Insertion versus detection gives phase containment. A question showing many defects inserted in requirements but detected in system test or production is testing whether you see a phase containment problem. The fix is to strengthen early static testing, such as requirement reviews.

Tip 6: Do not confuse severity and priority. In IEEE 1044 and ISTQB terms, severity is the degree of impact and priority is the urgency to fix. A low-severity spelling error on the home page can still be high priority.

Tip 7: Remember the Mode and Qualifier parallel. IEEE 1044 Mode (Wrong, Missing, Extra) and ODC Qualifier (Incorrect, Missing, Extraneous) express the same idea. 'Missing' defects often point to requirements or design gaps.

Tip 8: Expect a tailoring answer. ISTQB favours answers saying classification schemes should be tailored to the organisation, kept simple enough for consistent use, and reviewed over time. Avoid absolutes such as 'always use all IEEE 1044 attributes'.

Tip 9: Watch out for consistency and data-quality traps. Classification is only useful if it is applied consistently. Answers that mention training, clear definitions and avoiding overlapping categories are usually correct. Answers that let each tester invent categories are wrong.

Tip 10: Use elimination in scenario questions. First decide what the stakeholder needs: process feedback (ODC), standard reporting (IEEE 1044) or test ideas (taxonomy). Then eliminate options that serve a different purpose. Also reject options that confuse error, defect and failure.

Tip 11: Interpret simple distributions. Two common patterns:
- Many Function-type defects found late suggest design or requirements weaknesses.
- System testing revealing mostly coverage/simple-path triggers suggests earlier test levels were insufficient.
Answers suggesting targeted improvement in the corresponding earlier activity are usually right.

Tip 12: Mind the K-level. For K2 (understand) questions, pick the option that correctly explains the purpose or benefit. For K3/K4 (apply/analyse) questions, apply the scheme to the scenario. Classify the given defect or decide which preventive action the data supports.

Sample question 1
A test team notices that most defects found in system test are exposed only when the system runs under high load or after a restart. Using ODC, which attribute captures this information?
A) Defect Type
B) Trigger
C) Qualifier
D) Target
Answer: B. The Trigger records the condition that surfaced the defect, here workload/stress and start-up/restart.

Sample question 2
Defect data shows that 55% of production defects have the insertion activity 'Requirements'. What is the BEST preventive action?
Answer: Improve requirements quality and early static testing. Examples: formal requirement reviews with checklists derived from historical defects, and acceptance criteria refinement. Simply adding more system test cases is not the best answer.

Sample question 3
A Test Analyst wants to design tests targeting common boundary, error handling and race condition problems based on industry experience. What should be used?
Answer: A defect taxonomy, such as Kaner/Falk/Nguyen, applied with a defect-based test technique.

Summary
Defect classification turns individual defects into organisational learning. ODC uses orthogonal attributes, especially Trigger and Defect Type, to give in-process feedback on development and test activities. IEEE 1044 provides standard terminology and a comprehensive, tailorable attribute set for defects and failures, including insertion and detection activities. Defect taxonomies are hierarchical lists of likely defects that drive defect-based testing, checklists and root cause analysis. In the exam, match the method to its purpose, recognise each method's signature keywords, and choose answers that use classified data to prevent defects and improve testing.