Learn Experience-Based Testing (CTAL-TA) with Interactive Flashcards

Master key concepts in Experience-Based Testing through our interactive flashcard system. Click on each card to reveal detailed explanations and enhance your understanding.

Test Charter Missions: Target, Resources and Information

In the ISTQB Advanced Level Test Analyst syllabus, exploratory testing is a key experience-based technique, and it is often managed through session-based test management. Each time-boxed session is guided by a test charter, which states the mission of the session. A widely used charter template, popularized by Elisabeth Hendrickson, frames the mission in three parts: 'Explore (target) with (resources) to discover (information).' This structure keeps the session focused while leaving the tester free to design and run tests on the fly.

1. Target: This is what will be explored. It may be a feature, user story, module, interface, workflow, quality characteristic, or risk area, such as 'the checkout payment flow' or 'the login screen's error handling.' A well-chosen target is narrow enough to cover within one session (typically 60 to 120 minutes) yet broad enough to allow real exploration. Targets are usually derived from risk analysis, test basis gaps, or areas where scripted tests are weak.

2. Resources: These are what the tester will use during exploration. Examples include specific test data, tools such as proxies, log viewers or browser developer tools, configurations, devices, personas, heuristics, checklists, defect taxonomies, or attack techniques. Naming resources helps the test analyst apply the experience-based techniques covered in the syllabus, such as error guessing and checklist-based testing, in a deliberate way.

3. Information: This is what the session aims to reveal. It might be defects, security weaknesses, usability problems, performance issues, unclear requirements, or simply knowledge about how the system behaves. Stating the desired information gives the session a clear purpose and supports meaningful debriefing.

Well-written charters improve test management and traceability. They help the Test Analyst plan coverage, allocate sessions, estimate effort, and report results through session sheets and debriefs. They also balance freedom with accountability: testers apply their skill and creativity, while stakeholders can see what was explored, how it was explored, and what was learned.

Factors That Influence Test Charter Design

In the ISTQB Advanced Level Test Analyst syllabus, a test charter guides an exploratory testing session. It states the mission, the target area, and sometimes the resources and techniques to use. Its design is not arbitrary. Several factors shape what a charter contains and how detailed it is. First, product and project risks matter most. High-risk areas, critical business functions, and features with a history of defects need more focused and more numerous charters. Second, the test objectives and the information stakeholders need decide the purpose of each charter. A charter might aim to learn a new feature, confirm a fix, probe security, or assess usability, and that aim shapes its scope. Third, the test basis and the information available influence how specific a charter can be. Where requirements are incomplete or changing, charters tend to be broader and more investigative. Where documentation is detailed, charters can target specific conditions. Fourth, the knowledge, experience, and skill of the testers affect the level of detail. Experienced testers can work from short, high-level charters. Less experienced testers benefit from more guidance, such as suggested techniques, heuristics, test ideas, or checklists. Fifth, the time available affects scope. Charters in session-based test management are usually time-boxed, often 60 to 120 minutes, so the mission must be achievable within one session. Sixth, product maturity and the development lifecycle play a role. Early or agile iterations may call for exploratory, learning-oriented charters, while later phases favor charters that target regression risks or integration issues. Seventh, practical constraints shape what can be explored, including test environments, tools, test data, and access to systems. Finally, results from previous sessions provide feedback. Defect clusters, coverage gaps, and questions found during debriefs lead to new or refined charters. Considering these factors helps the Test Analyst create charters that are focused, achievable, risk-driven, and valuable, while still leaving room for the creativity and learning that make exploratory testing effective.

Additional Information in a Test Charter

In the ISTQB Advanced Level Test Analyst syllabus, a test charter is the guiding document for a session of exploratory testing. Its core elements state the purpose of the session: the test objective (the mission), the test conditions or areas to explore, and the time box. Additional Information is the supporting detail that turns a short mission statement into a practical guide. It gives the tester enough context to explore efficiently without turning the charter into a rigid scripted test.

Typical additional information includes the following:

1. Priority: how important the charter is compared with others, usually based on product risk.
2. References: requirements, user stories, risk items, defect reports, or specifications that give background.
3. Actors or user roles: the personas or user types whose viewpoint the tester should take.
4. Setup and preconditions: the test environment, configuration, tools, and system state needed before starting.
5. Test data: the data sets or accounts to use, or the kinds of data worth exploring.
6. Activities and ideas: suggested actions, heuristics, attacks, or variations to try, such as boundary values or interrupted workflows.
7. Oracle notes: how to judge whether a result is correct when no expected results are written down.
8. Constraints: areas that are out of scope or assumptions the tester should respect.

This information serves several purposes. It reduces setup time and wasted effort. It helps less experienced testers work effectively. It gives traceability back to risks and requirements, which supports coverage measurement and reporting. It also makes sessions easier to repeat, compare, and review in session-based test management.

The Test Analyst must balance detail against freedom. Too little information leaves the session unfocused and hard to evaluate. Too much detail removes the creativity and adaptability that give experience-based testing its value. After the session, findings such as defects, issues, coverage achieved, and new test ideas are recorded and discussed in a debriefing, which may refine future charters.

Level of Detail in a Test Charter and Tester Freedom

In the ISTQB Advanced Level Test Analyst syllabus, a test charter guides exploratory testing, usually within session-based test management. The charter states the mission of a time-boxed session. Typical contents include the test objective, the scope or areas to explore, the risks or test conditions to address, constraints, suggested techniques or heuristics, test data, environment, and the session duration. The Test Analyst must decide how much detail to put into the charter, and that choice directly controls how much freedom the tester has. The two are inversely related. A highly detailed charter names specific features, conditions, data sets, and approaches. This improves focus, coverage traceability, repeatability, and consistency across testers, and it helps less experienced testers or those new to the domain. It also supports stakeholders who need evidence of what was tested, such as in regulated or high-risk contexts. However, too much detail turns the charter into a quasi-scripted test case. This reduces the tester's ability to follow unexpected leads, use intuition, and find defects that predefined tests miss, which is the main value of experience-based testing. A lightly detailed charter, for example 'Explore the checkout process to discover usability and error-handling problems', gives the tester wide latitude to investigate, adapt, and apply skill and creativity. The risks are uneven coverage, harder progress measurement, and results that depend heavily on the individual tester's competence. The appropriate level depends on several factors: tester skill and domain knowledge, product risk, project maturity, how well the area is already covered by other tests, regulatory needs, and how results will be reported. Balance is achieved by keeping charters focused but not prescriptive, documenting what was actually done in session sheets, and refining future charters through debriefs with the test manager. This keeps the testing accountable without losing the adaptive, learning-driven nature of exploratory testing.

Session Sheets and Test Logs in Session-Based Testing

In the ISTQB Advanced Level Test Analyst syllabus, session-based testing is a structured way to manage exploratory testing, an experience-based technique. Testing is organized into time-boxed sessions, typically 60 to 120 minutes, each guided by a test charter. The charter states the test objectives, the area to explore and possibly the techniques or risks to address. Because exploratory testing designs, executes and learns at the same time, documentation is essential for accountability, traceability and reporting. Session sheets and test logs provide it.

A session sheet is the main record of a session and is often based on a standard template. It usually contains: the charter and its identifier; the tester's name; the date, start time and duration; the test environment, configuration and data used; and a breakdown of how time was spent. Time is commonly split into test design and execution, bug investigation and reporting, and session setup. The sheet may also show how much time went to the charter versus unplanned opportunities. Finally, it records defects found, issues or questions raised, and any follow-up charters suggested.

The test log, often kept within the session sheet as test notes, is a running record of what the tester actually did. It captures actions taken, inputs used, observations, ideas tested, coverage achieved and anomalies seen. These notes make the session reproducible. They let defects be reported with enough detail to reproduce them and provide evidence of what was and was not covered.

After each session, a debriefing is held between the tester and the test manager or lead, often using the PROOF approach: Past, Results, Obstacles, Outlook and Feelings. The session sheet and log support this review. Aggregated session data gives measurable metrics, such as sessions completed, coverage by area, defects found and time distribution. This helps the Test Analyst and test management assess progress, risk coverage and product quality, and plan further charters, making exploratory testing manageable and auditable.

Read-Do and Do-Confirm Checklists

In the ISTQB Advanced Level Test Analyst syllabus, checklist-based testing is an experience-based technique. The tester uses a list of conditions, questions or reminders to guide testing. The checklist captures knowledge from experience, defect history, standards and quality characteristics. The syllabus describes two styles of checklist, an idea popularised by Atul Gawande in The Checklist Manifesto: Read-Do and Do-Confirm.

Read-Do checklists: The tester reads an item, performs the action it describes, and only then moves to the next item, much like following a recipe. The order matters and each step is completed as it is read. This style suits less experienced testers, unfamiliar applications and rarely performed tasks. It also fits complex or high-risk activities where skipping or reordering a step could cause serious problems. Examples include configuring a test environment or executing a regulatory compliance check. Read-Do checklists give strong guidance and consistency. However, they can slow experienced testers down and may discourage creative exploration.

Do-Confirm checklists: The tester first carries out the work from memory, skill and judgement. At defined pause points, the tester stops and reviews the checklist to confirm that nothing important was missed. The checklist acts as a safety net rather than a script. This style suits experienced testers who know the domain well. It preserves their flexibility and efficiency while still guarding against lapses of attention. Examples include reviewing a test basis for usability issues, or confirming that key quality characteristics were covered after an exploratory session.

Good practice for both types: Keep checklists short and focused. Write items as high-level conditions or questions rather than detailed steps. Make items clear, verifiable and relevant to current risks. Review and update checklists regularly, because they can lose effectiveness over time as developers learn to avoid the listed issues. The Test Analyst should choose the type that matches the tester's experience, the task's criticality and the level of variability desired in testing.

Collecting Information for Checklist Items

In the ISTQB Advanced Level Test Analyst syllabus, checklist-based testing is an experience-based technique in which the Test Analyst uses a list of items, conditions or questions to guide test design and execution. The value of a checklist depends on the quality of the information used to build it, so collecting information for checklist items is a key activity.

Checklist items can come from many sources. The most important is the experience of testers, developers and domain experts: what has gone wrong before, which features users rely on most, and how and why software tends to fail. Historical defect data is another major input. Defect reports, defect taxonomies, root cause analyses and production incident logs show recurring failure patterns that can become items such as 'Are boundary values validated?' or 'Is the session handled correctly after a timeout?'. Other useful sources include industry standards, regulatory requirements, organizational guidelines, quality characteristics such as usability, security, performance and maintainability, product risk analyses, user feedback, help desk tickets, lessons learned from retrospectives, and published heuristics or generic checklists.

The Test Analyst should check that the collected items are relevant to the test object and its context, remove duplicates, and group related items, for example by feature or quality characteristic. Items are often phrased as questions so testers can check them clearly. Checklists are usually kept at a high level so they can be reused and so testers keep some freedom in how they test. This gives variety in test execution, but it can also lower repeatability.

Information gathering is not a one-time task. Checklists should be reviewed and updated regularly. New defect findings, changes in the product, and new technologies or risks should lead to new items. Items that no longer find defects should be revised or removed. Regular maintenance keeps the checklist effective, helps avoid the pesticide paradox, and keeps the list a manageable length. Over time, this turns individual tester experience into shared, reusable knowledge for the organization.

Writing Checklist Items as Yes or No Questions

In the ISTQB Advanced Level Test Analyst syllabus, checklist-based testing is an experience-based technique. The tester uses a list of items, conditions or quality characteristics to guide test design and execution. A widely recommended practice is to write each checklist item as a closed question that can be answered Yes or No. This turns vague reminders into clear, verifiable checks.

Why use Yes/No questions? First, they reduce ambiguity. An item such as 'Error handling' gives no guidance, but 'Does the system display a meaningful message when a mandatory field is left empty?' tells the tester exactly what to verify. Second, a binary answer makes each item a clear decision point. This supports consistent results across testers, simple reporting, and checklist coverage measurement, which is the proportion of items checked and passed. Third, a Yes/No answer is quick to record during sessions or reviews.

Good practice when writing these items:
1. Make each question atomic. Check one thing only, so the item can be evaluated separately and directly. Avoid compound questions joined with 'and' or 'or'.
2. Use a consistent polarity. Usually 'Yes' should mean the desired outcome, for example 'Are all input fields labelled?' Then any 'No' clearly signals a potential defect.
3. Keep items specific and observable. Avoid subjective wording such as 'Is the screen user-friendly?' unless clear criteria are defined.
4. Avoid questions that are better handled by automated tools, that duplicate entry or exit criteria, or that are too general to act on.
5. Keep the checklist reasonably short and focused on high-risk areas, common defect types and lessons learned.

Checklists are living artefacts. They should be reviewed and updated regularly based on defect analysis, because items that no longer find defects lose value over time. Well-formed Yes/No questions also help less experienced testers apply expert knowledge consistently. They leave room for exploration around each item while still providing traceable, repeatable results.

Structuring and Prioritizing a Checklist

In the ISTQB Advanced Level Test Analyst syllabus, checklist-based testing is an experience-based technique in which the tester uses a list of items, conditions, or questions to guide test design and execution. A checklist is only useful if it is well structured and sensibly prioritized.

Structuring a checklist means organizing its items logically so testers can apply them consistently and efficiently. Items are commonly grouped by quality characteristic (functional suitability, usability, security, performance), by functional area or module, by test level, or by type of work product being examined. A hierarchical layout lets testers move from general concerns to specific details. Each item should be phrased clearly, often as a question that can be answered with yes, no, or not applicable. Items should be self-contained, unambiguous, and not overlap. Sources for items include the tester's experience, historical defect data, defect taxonomies, standards, user expectations, and knowledge of what matters to the business. Checklists should stay reasonably short, because long lists become tedious and are applied carelessly. Keeping items at a high level also introduces useful variability, since different testers interpret them differently and may find different defects.

Prioritizing a checklist means ordering items so the most important ones are checked first. Priority is usually based on product risk, combining the likelihood of failure with its impact on users and the business. Other factors include how often a defect type has occurred before, the criticality of the feature, regulatory requirements, and stakeholder priorities. Placing high-priority items at the top ensures the most valuable checks are completed even if time runs short.

Checklists are living artifacts. Test Analysts should review and update them regularly, adding items for newly discovered defect types and removing obsolete ones. Items that no longer find defects may indicate that developers have learned from past mistakes, so the list should evolve with the product and the project to remain effective.

Maintaining and Sharing Checklists

In the ISTQB Advanced Level Test Analyst syllabus, checklist-based testing is an experience-based technique. The tester uses a list of items, conditions, or questions to guide test design and execution. A checklist keeps its value only if it is actively maintained and shared, because it captures what testers and the organization have learned over time.

Maintaining checklists: Checklists are built from experience, defect history, standards, quality characteristics, user expectations, and knowledge of why software fails. The product, technology, and risks change, so the checklist must change too. The Test Analyst should review it regularly. Items that reflect newly found defect types, failures in production, or emerging risks should be added. Obsolete, redundant, or rarely useful items should be removed. Defect analysis and retrospectives are key inputs: if a defect escaped testing, the checklist may need an item to catch similar problems later. Checklists should stay reasonably short and focused. Overly long lists are hard to use and encourage superficial checking. Items are usually phrased at a high level so testers can vary how they test them. This variation supports coverage and reduces the pesticide paradox, although it lowers repeatability. Version control helps track changes and keeps everyone using the current version.

Sharing checklists: Sharing a checklist turns individual expertise into organizational knowledge. Less experienced testers benefit from the insights of senior colleagues, and testing becomes more consistent across teams and projects. Checklists can be kept in a central repository or test management tool, reviewed by peers, and tailored for specific applications, platforms, or quality characteristics such as usability, security, or accessibility. Feedback from everyone who uses them should flow back into updates.

Benefits and risks: Well-maintained, shared checklists give structure, help measure coverage by tracking which items have been checked, and support onboarding. However, a stale checklist creates false confidence. It also risks mechanical, unthinking execution. Testers should treat checklists as guides that support, not replace, critical thinking and exploratory investigation.

Benefits of Crowd Testing

In the ISTQB Certified Tester Advanced Level Test Analyst syllabus, crowd testing appears alongside experience-based techniques such as exploratory testing and checklist-based testing. In crowd testing, testing is outsourced to a large, distributed group of independent testers, often reached through an online platform. These testers usually work on their own devices and in their own environments. They rely mainly on their personal experience, intuition and domain knowledge rather than detailed scripted test cases. The main benefits for a Test Analyst are the following. First, crowd testing provides diversity and realism. Testers bring a wide range of devices, operating systems, browsers, network conditions, locations, languages and cultural backgrounds. This lets the system be tested under real-world conditions that are hard and expensive to reproduce in a lab. It is especially valuable for mobile, web, localization and compatibility testing. Second, it offers a broad mix of experience and perspective. Many testers with different skill levels and domain knowledge explore the product in different ways. This increases the chance of finding defects that an in-house team might miss because of familiarity or bias. It also gives useful insight into usability and user experience from the point of view of actual end users. Third, it is scalable and flexible. A large number of testers can be engaged quickly for short periods, such as before a release. Testing can also run around the clock across time zones. This shortens feedback cycles and supports agile and continuous delivery. Fourth, it can be cost-effective. Organizations avoid buying and maintaining large device labs. Payment models, such as paying per valid defect found, link cost to results. Finally, crowd testing complements, rather than replaces, internal testing. The Test Analyst must still manage the risks it brings: confidentiality and security of the test object, inconsistent quality of defect reports, duplicate reports, limited coverage control and the need for clear test charters. When these risks are managed well, crowd testing strengthens experience-based testing by adding breadth, realism and independence.

Limitations of Crowd Testing

In the ISTQB Advanced Level Test Analyst syllabus, crowd testing is an experience-based approach in which a large, distributed group of testers, often external freelancers, tests a product on their own devices, platforms and networks. It offers broad coverage of real-world environments and many different perspectives, but it has several important limitations.

First, tester skill and experience vary widely. Experience-based techniques depend heavily on the tester's knowledge, so results from an unvetted crowd can be uneven. Some participants may lack domain expertise, understanding of business rules or knowledge of testing techniques. As a result, they may find many shallow, obvious defects and miss subtle or complex ones.

Second, defect reporting quality is often inconsistent. Reports may be incomplete, poorly written or hard to reproduce. Several testers may also report the same defect. Triaging, de-duplicating and verifying these reports adds overhead for the internal team.

Third, coverage is hard to control and measure. Testers tend to explore popular or easy features, leaving complex workflows, edge cases and back-end functionality undertested. Without clear charters, coverage gaps are hard to identify, and traceability to requirements or risks is weak.

Fourth, confidentiality, security and legal concerns arise. Exposing unreleased software, intellectual property or sensitive data to external people creates a risk of leaks. Non-disclosure agreements, data privacy rules and regulatory requirements can restrict or prevent crowd testing, especially in safety-critical or regulated domains.

Fifth, payment models can create the wrong incentives. Pay-per-defect schemes may encourage testers to report many trivial issues rather than investigate deeply.

Sixth, crowd testers have limited access to internal tools, test environments, documentation and stakeholders. This limits their ability to perform integration testing, back-end testing or tests that need special configurations.

Finally, crowd testing is hard to coordinate, repeat and fit into structured processes such as regression testing or continuous integration. It should therefore complement, not replace, systematic testing performed by skilled internal Test Analysts.

More Experience-Based Testing questions
349 questions (total)
Practice questions
One session at a time, always new questions