User Questionnaires and Accessibility Conformance Levels

User Questionnaires and Accessibility Conformance Levels: A Complete ISTQB CTAL-TA Guide

Introduction
In the ISTQB Advanced Level Test Analyst (CTAL-TA) syllabus, usability and accessibility belong to the quality characteristics that the Test Analyst is directly responsible for evaluating. Two topics appear often in exam questions:
1. Standardized user questionnaires, such as SUMI, WAMMI and SUS, which measure user satisfaction.
2. Accessibility conformance levels, mainly the WCAG Levels A, AA and AAA, which show how accessible a product is.
This guide covers why these topics matter, what they are, how they work in practice, and how to answer exam questions about them.

1. Why It Is Important
Satisfaction is part of usability. ISO 25010 and ISO 9241-11 define usability through three things: effectiveness, efficiency and satisfaction in a specified context of use. Effectiveness and efficiency can be observed and timed. Satisfaction is subjective, so it needs a reliable instrument, and questionnaires provide one.
Standardized questionnaires give comparable numbers. Questionnaires built in-house are often biased and cannot be compared with anything outside the project. Standardized questionnaires such as SUMI, WAMMI and SUS have been validated, so their results can be:

• benchmarked against industry databases;
• compared across releases;
• compared against competitors.

Accessibility is often a legal requirement. Many laws and standards make accessibility mandatory, usually by referring to WCAG conformance levels. Examples include:

• the US Section 508 and the Americans with Disabilities Act (ADA);
• the European Accessibility Act and EN 301 549;
• the UK Equality Act.

If a product fails to meet the required level, the organisation risks lawsuits, fines, exclusion from public tenders and damage to its reputation.
Accessibility improves the product for everyone. Accessible design helps users with permanent disabilities, temporary impairments (such as a broken arm) and situational limits (such as bright sunlight or a noisy room). It also tends to improve general usability.
The Test Analyst's role. The Test Analyst is expected to:

• help define measurable usability and accessibility requirements;
• choose suitable evaluation techniques;
• report results in a form stakeholders can act on.

2. What It Is
2.1 User Questionnaires
A user questionnaire is a structured set of statements or questions. Users answer it after using the system, usually on a Likert scale (for example, from strongly agree to strongly disagree). It is a survey technique for usability evaluation. It is typically used:

• at the end of a usability test session;
• after a beta release;
• with real users in production.

Key standardized questionnaires:

• SUMI (Software Usability Measurement Inventory)
  • A 50-item questionnaire for software products.
  • It produces a global usability score plus five subscales: Efficiency, Affect, Helpfulness, Control and Learnability.
  • Scores are standardized against a reference database. The mean is 50 and the standard deviation is 10, so a score above 50 means better than the average commercial product.
  • It can be used as an acceptance criterion, for example: the system must achieve a SUMI global score of at least 55.
• WAMMI (Website Analysis and MeasureMent Inventory)
  • A questionnaire designed for websites and web applications.
  • It is usually completed online by real visitors.
  • It measures five factors: Attractiveness, Controllability, Efficiency, Helpfulness and Learnability.
  • It gives a Global Usability Score that is benchmarked against a database of other websites.
• SUS (System Usability Scale)
  • A short, technology-independent questionnaire with 10 items on a 5-point scale.
  • It produces a single score from 0 to 100. This is not a percentage.
  • A score of about 68 is generally considered average.
  • It is quick and cheap, and it is reliable even with small samples.

Strengths of questionnaires:

• They are cheap and can reach many users, including remote users.
• They give quantitative results that can be repeated.
• They support benchmarking and tracking trends.
• They capture the subjective satisfaction of real users.

Limitations of questionnaires:

• They measure perception, not actual performance.
• They show that users are unhappy, not why. They therefore do not pinpoint specific defects.
• They need a big enough and representative sample.
• Users must actually have used the system.
• Answers can be affected by the users' mood or by their relationship with the organisation.

For these reasons, questionnaires are best combined with:

• usability test sessions, which involve observation, think-aloud and measuring task completion and time;
• heuristic evaluation and expert reviews.

2.2 Accessibility Conformance Levels
Accessibility is the degree to which people with the widest range of characteristics and capabilities can use a product. This includes people with visual, auditory, motor, cognitive and speech impairments.
The main reference standard is the W3C Web Content Accessibility Guidelines (WCAG 2.x). It is organised in three layers:

• Four principles (POUR):
  • Perceivable: information is presented in ways users can perceive, for example text alternatives for images and captions for video.
  • Operable: users can operate the interface, for example by keyboard and with enough time.
  • Understandable: content and operation are clear, for example readable text, predictable behaviour and help with input errors.
  • Robust: content works reliably with different user agents and assistive technologies, such as screen readers.
• Guidelines under each principle.
• Testable success criteria under each guideline. Each success criterion is assigned a conformance level.

The three conformance levels:

• Level A (minimum)
  • The most basic requirements. Without them, some groups of users cannot use the content at all.
  • Examples: text alternatives for non-text content, full keyboard operability, no keyboard traps, and not relying on colour alone to convey information.
• Level AA (mid-range)
  • Includes all Level A criteria plus additional ones that remove significant barriers.
  • Examples: a minimum colour contrast of 4.5:1 for normal text, text that can be resized to 200% without loss of content, captions for live audio, consistent navigation, and a visible keyboard focus.
  • AA is the level most commonly required by laws, regulations and organisational policies.
• Level AAA (highest)
  • Includes all A and AA criteria plus the strictest ones.
  • Examples: a contrast ratio of 7:1, sign language interpretation for prerecorded media, and content readable at a lower secondary education level.
  • W3C does not recommend AAA as a general policy for entire sites, because some content cannot meet every AAA criterion.

Conformance levels are cumulative. To claim AA, every A and AA criterion must be met. Meeting most AA criteria but failing a single A criterion means the product does not conform even to Level A.
Other WCAG conformance requirements:

• Conformance applies to full pages. Part of a page cannot be excluded.
• It applies to complete processes. For example, every step of a checkout must conform.
• Only accessibility-supported technologies may be relied on.
• Non-conforming technologies must not interfere with the rest of the page.

3. How It Works in Practice
Using questionnaires in a test project:

1. Define measurable satisfaction requirements. For example: SUS score of at least 75 for the new booking flow, or a SUMI Learnability subscale above 50.
2. Select the instrument. Choose SUMI for software applications, WAMMI for websites, and SUS for a quick, general-purpose measure.
3. Select representative users. Their profiles should match the intended context of use, including users with disabilities where relevant.
4. Administer the questionnaire. This is usually done immediately after users perform realistic tasks in a usability test, or online with live website visitors.
5. Analyse the results.
   • Calculate the global score and the subscale scores.
   • Compare them with the benchmark, the previous release or the target value.
   • Identify weak dimensions, such as low Helpfulness.
6. Investigate the root causes. Use observation data, interviews or heuristic evaluation to find out why the scores are low.
7. Report and retest. Re-measure after changes to show improvement. This supports formative evaluation (improving the design during development) and summative evaluation (verifying the final product).

Testing accessibility conformance:

1. Determine the target level. This comes from legal or contractual obligations or company policy, and is usually WCAG 2.1 or 2.2 Level AA.
2. Derive test conditions from each success criterion at and below the target level.
3. Run automated checks. Accessibility scanners and contrast analysers find missing alt attributes, contrast failures, missing form labels and similar issues. However, automated tools find only a portion of the issues.
4. Perform manual expert testing. This includes:
   • keyboard-only navigation;
   • screen reader testing (for example NVDA, JAWS or VoiceOver);
   • checking that alt text is meaningful;
   • checking focus order;
   • checking error identification;
   • zooming to 200% or 400%.
5. Test with users with disabilities to confirm real-world usability, not just technical compliance.
6. Report findings. Map each failure to its success criterion and level, then state whether the product conforms, often in a conformance report or an accessibility statement.

Example: A government portal must meet WCAG 2.1 AA. Testing finds three problems:

• body text has a contrast of 3.8:1, which fails an AA criterion;
• a date picker cannot be operated by keyboard, which fails an A criterion;
• there is no sign language video, which is only an AAA criterion.

The portal does not conform to AA. It does not even conform to A, because of the keyboard issue. The missing sign language video does not affect the AA claim.

4. Exam Tips: Answering Questions on User Questionnaires and Accessibility Conformance Levels

• Link questionnaires to satisfaction. If a question asks how to measure user satisfaction or subjective opinion, the answer is usually a standardized questionnaire such as SUMI, WAMMI or SUS. If it asks about effectiveness or efficiency, think of usability test sessions that measure task completion, errors and time.
• Recognize the instruments by their context. SUMI is for software and has 50 items and five subscales. WAMMI is for websites and is completed online by visitors. SUS is a quick 10-item scale producing a score from 0 to 100. If the scenario mentions a website and live visitors, WAMMI is the likely answer.
• Know the key benefit: benchmarking. The main reason to choose a standardized questionnaire over a self-made survey is that it is validated and its results can be compared with a reference database or previous releases.
• Know the key limitation: no root cause. Questionnaires tell you that a problem exists, not where or why. Answer options claiming that questionnaires identify specific defects in the user interface are usually wrong.
• Remember the timing. Questionnaires are completed after users have used the system, often at the end of a usability session or in production. They are not a static technique for reviewing specifications.
• Remember that WCAG levels are cumulative. AA includes A, and AAA includes AA and A. A single failure at Level A means no conformance at any level. Watch for distractors that treat the levels as independent.
• Remember AA as the usual target. When a scenario mentions legal or regulatory compliance or public sector websites, Level AA is typically the expected answer. AAA is not normally required for whole sites.
• Order the levels correctly. A is the minimum, AA is the mid-range and AAA is the highest. Do not confuse these with letter grades where A is best.
• Use the POUR principles. If asked which principle a defect violates, map it as follows:
  • missing alt text or captions: Perceivable;
  • keyboard traps or time limits: Operable;
  • confusing error messages: Understandable;
  • broken markup that screen readers cannot interpret: Robust.
• Remember automation is not enough. Correct answers usually state that automated accessibility tools must be complemented by manual testing and testing with users with disabilities.
• Read scenario details carefully. Look for keywords such as legal requirement, benchmark, website visitors, subjective, minimum level and contrast. They usually point straight to the right option.
• Eliminate absolute statements. Options containing always, guarantees or only are usually wrong. Examples are 'questionnaires guarantee a usable product' and 'AAA must always be achieved'.
• Think like a Test Analyst. The best answer usually combines techniques: questionnaires for satisfaction, observation for root causes, and standards-based checks plus real users for accessibility. Choose answers that show balanced, risk-based evaluation.

Summary

• Standardized user questionnaires (SUMI, WAMMI and SUS) provide validated, quantitative measures of user satisfaction that can be benchmarked. They do not explain the causes of problems.
• WCAG accessibility conformance levels (A, AA and AAA) are cumulative tiers of testable success criteria built on the POUR principles.
• Level AA is the common legal and contractual target.
• For the exam, link satisfaction to questionnaires, website surveys to WAMMI, compliance to WCAG AA, and remember that one failed Level A criterion breaks conformance at every level.