Characteristics of Reliable Evaluation Criteria

Characteristics of Reliable Evaluation Criteria (CIA Part 2: Engagement Planning)

Introduction
In internal auditing, every conclusion about whether something is working well depends on a standard of comparison. That standard is called the evaluation criteria. Without criteria, an auditor has no defensible basis for calling a condition good, bad, effective or deficient. The Global Internal Audit Standards (and the earlier IPPF guidance) require internal auditors to identify the most relevant criteria during engagement planning. They must also judge whether those criteria are adequate for evaluating the activity under review. This guide explains what reliable criteria are, why they matter, how auditors develop and assess them, and how to answer CIA Part 2 exam questions on the topic.

Why It Is Important
1. Criteria are the foundation of every finding. A finding normally has four or five attributes: Condition, Criteria, Cause, Effect, and sometimes Recommendation. Criteria answer the question what should be? The condition is what is. The gap between the two is the finding. Weak criteria make the whole finding weak.

2. Credibility and defensibility. Management will challenge findings based on vague, biased or irrelevant standards. Reliable criteria make conclusions objective, repeatable and acceptable to stakeholders.

3. Consistency. Two competent auditors applying the same reliable criteria to the same facts should reach similar conclusions. This supports the quality and professionalism of the audit activity.

4. Efficient planning. Clear criteria guide the engagement objectives, scope, work program and evidence requirements. Time is not wasted collecting irrelevant information.

5. Standards compliance. The Standards require auditors to determine evaluation criteria during planning. If management's criteria are inadequate, auditors must work with management to develop appropriate criteria. Under the Global Internal Audit Standards (Standard 13.4), auditors must also identify and communicate with management if the criteria are absent or inadequate. Auditors may report such inadequacy as a finding in itself.

What It Is
Evaluation criteria are the standards, measures, expectations or benchmarks against which an activity, process, control or outcome is assessed.

Common sources of criteria:
Internal sources

• Organizational policies and procedures
• Budgets
• Key performance indicators (KPIs)
• Strategic objectives
• Board-approved risk appetite
• Contracts
• Service level agreements

External sources

• Laws and regulations
• Industry standards (e.g., ISO standards, PCI DSS)
• Recognized frameworks such as COSO Internal Control, COSO ERM, COBIT and the NIST Cybersecurity Framework
• Accounting standards such as IFRS or GAAP

Leading practices

• Benchmarks from peer organizations
• Professional association guidance
• Best practices

The characteristics of reliable (good) evaluation criteria
IIA guidance and widely accepted audit literature describe several characteristics. A popular way to remember them is that criteria should be relevant, reliable, objective (neutral), understandable, complete, comparable, measurable and acceptable.

1. Relevant
Criteria must relate directly to the engagement objectives and the activity's goals. They should support conclusions that meet stakeholder needs. Using a manufacturing safety standard to judge an accounts payable process is irrelevant.

2. Reliable
Criteria should produce consistent, reasonable conclusions when applied by different auditors in similar circumstances. Reliable criteria come from credible sources and are applied the same way each time.

3. Objective / Neutral
Criteria must be free from bias, whether from the auditor or from management. They should not be designed to produce a predetermined outcome.

4. Understandable
Criteria should be clearly stated and not open to significantly different interpretations. Both auditors and auditees must understand them.

5. Complete
Criteria should include all relevant factors needed to evaluate the subject matter. Important elements must not be omitted, so the evaluation is not misleading.

6. Measurable
Criteria should allow reasonably consistent quantitative or qualitative measurement. "Invoices are paid within 30 days" is measurable. "Invoices are paid promptly" is not.

7. Comparable
Criteria should be consistent with those used in similar engagements or by similar organizations. This allows meaningful comparison over time and across units.

8. Acceptable
Criteria should be acceptable to the stakeholders, especially management and the board. Agreeing criteria with management at the planning stage reduces disputes at reporting.

9. Authoritative (often added)
Criteria from recognized authoritative bodies carry more weight than criteria an auditor creates. Examples include laws, the board, regulators and recognized frameworks.

Memory aid: "RROUC-MCA", or think "Really Reliable Objective Understandable Criteria Make Clear Assessments".

How It Works in Practice
Step 1: Understand the activity and engagement objectives. During the preliminary survey, the auditor learns the activity's objectives, risks and controls. Engagement objectives determine which criteria will be relevant.

Step 2: Identify existing criteria. The auditor first looks for criteria management already uses, such as policies, procedures, KPIs, contracts and regulatory requirements.

Step 3: Assess the adequacy of criteria. The auditor tests the identified criteria against the characteristics above. Are they relevant, measurable, complete, objective and understandable? Are they current and approved?

Step 4: If criteria are adequate, use them. The auditor documents the criteria in the planning memo and work program.

Step 5: If criteria are inadequate or missing, develop them with management. The auditor works with management and/or the board to develop appropriate criteria. Sources include authoritative frameworks, regulations, industry benchmarks or leading practices. The absence of adequate criteria may itself be an engagement observation, because it indicates weak governance or control design.

Step 6: Communicate and agree. The auditor discusses the criteria with management, ideally at the opening meeting or in the engagement plan. This builds acceptance and avoids disputes later.

Step 7: Apply criteria during fieldwork and reporting. The auditor compares the condition with the criteria and identifies the cause and effect. The criteria are then stated in the engagement communication.

Example
An auditor is reviewing customer complaint handling. Management's policy says only that complaints "should be resolved quickly and satisfactorily." This criterion is not measurable or clearly understandable.

The auditor works with management to adopt a measurable standard. Drawing on industry benchmarks and regulatory guidance, they agree that 95% of complaints will be acknowledged within 24 hours and resolved within 10 business days. Satisfaction will be measured by a post-resolution survey.

The new criterion is relevant, measurable, understandable, comparable and acceptable to management. The original vague policy may be reported as a control design weakness.

Relationship to Other Planning Concepts

• Engagement objectives: Criteria flow from objectives. If the objective is to assess regulatory compliance, the relevant regulations are the criteria.
• Risk assessment: Criteria help define what risk exposure is acceptable, linked to risk appetite.
• Work program: Procedures are designed to gather sufficient, reliable, relevant and useful evidence against the criteria.
• Findings: Criteria form the second attribute of a finding (Condition, Criteria, Cause, Effect).
• Evidence vs. criteria: Do not confuse the two. Information (evidence) must be sufficient, reliable, relevant and useful. Criteria have their own characteristics. Both use the word "reliable," which exam writers exploit.

Exam Tips: Answering Questions on Characteristics of Reliable Evaluation Criteria
Tip 1: Know the sequence. The auditor first identifies and evaluates management's existing criteria. If they are inadequate, the auditor works with management and/or the board to develop suitable criteria. Avoid answers where the auditor unilaterally imposes criteria without consulting management, unless the question clearly calls for it. Also avoid answers where the auditor proceeds without criteria at all.

Tip 2: Look for measurability. When an answer choice turns a vague standard into a quantifiable one, it is often correct. For example, it may change "timely" into "within 5 days." Questions asking "Which criterion is most appropriate?" usually reward specific, measurable choices.

Tip 3: Relevance to engagement objectives wins. If several choices are measurable, choose the one that links most directly to the objective of the engagement.

Tip 4: Authoritative sources rank higher. Criteria from laws, regulations, board-approved policies or recognized frameworks (COSO, ISO) are generally stronger. They outrank criteria based on an individual's opinion or a single manager's preference.

Tip 5: Watch for bias. Criteria that management set to make performance look good, or that the auditor chose to support a predetermined conclusion, fail the objectivity test. Answers describing such criteria are wrong.

Tip 6: Missing criteria can be a finding. If a scenario describes an activity with no performance measures or policies, a likely correct response has two parts. The auditor develops criteria with management, and may report the lack of criteria as a control weakness.

Tip 7: Do not confuse the characteristics of criteria with those of evidence. Evidence: sufficient, reliable, relevant, useful. Criteria: relevant, reliable, objective, understandable, complete, measurable, comparable, acceptable. If a question asks about criteria, "sufficient" is usually a distractor.

Tip 8: Recognize the role of criteria in findings. Questions may describe a finding and ask which element is missing. If the write-up lacks a statement of what should be, the missing element is the criteria.

Tip 9: Timing matters. Criteria are established during engagement planning, not during fieldwork or reporting. Choices that set criteria after the testing is done are generally incorrect.

Tip 10: Use elimination. Eliminate answers that are vague ("adequate," "reasonable," "appropriate" with no measure). Eliminate answers that are irrelevant to the objective, or that come from unauthoritative or biased sources. The remaining answer is usually the best.

Tip 11: Read for keywords. Words like best, most appropriate and primary signal that more than one option may be partially correct. Pick the one meeting the most characteristics, especially relevance and measurability.

Sample Practice Question
An internal auditor is planning an engagement to evaluate the efficiency of the purchasing department. Which of the following would be the most appropriate evaluation criterion?
A. The purchasing manager's opinion of departmental performance.
B. Purchase orders should be processed efficiently.
C. 90% of purchase requisitions should be converted into purchase orders within three business days, consistent with industry benchmarks.
D. The number of purchase orders processed last year.

Answer: C. It is measurable, relevant to efficiency, comparable through industry benchmarks and objective.

• A is subjective and potentially biased.
• B is vague and not measurable.
• D is historical data, not a standard of what should be.

Summary
Reliable evaluation criteria are the yardstick of internal auditing. They should be relevant, reliable, objective, understandable, complete, measurable, comparable and acceptable, and preferably authoritative.

During planning, auditors identify management's criteria and assess their adequacy. When the criteria fall short, auditors collaborate with management to develop better ones, and may report the deficiency.

On the CIA exam, favor answers that are specific, measurable, tied to engagement objectives, drawn from authoritative sources and agreed with management during planning.