Managing Embedded AI Features in Enterprise Software

1. Why This Topic Matters

Most organisations did not go shopping for AI; AI came to them. Productivity suites, CRM platforms, HR systems, collaboration tools, ticketing systems, security consoles and even accounting packages now ship with generative assistants, predictive scoring, automated summarisation, transcription and recommendation engines. These capabilities are often switched on by a vendor update, a licence tier change or a default feature flag, frequently without a formal procurement decision, a risk assessment or any awareness from the security or governance function.

This creates several problems that the ISACA AAISM (Advanced in AI Security Management) syllabus expects you to recognise:

- Shadow AI and inventory gaps: An AI inventory that only lists systems the organisation built or deliberately bought is incomplete. Embedded features are the largest and fastest-growing blind spot.
- Uncontrolled data flows: Prompts, documents, emails, meeting recordings and customer records may be transmitted to third-party model endpoints, sometimes in other jurisdictions, and sometimes retained or used for model improvement.
- Regulatory exposure: Under the EU AI Act the organisation is a deployer of these features and inherits obligations (human oversight, logging, transparency to affected persons, impact assessments for high-risk uses). Privacy law (GDPR, CCPA and others) applies regardless of whether the vendor or the customer chose to enable the feature.
- Accountability confusion: Business owners assume the vendor is responsible; the vendor's terms assume the customer is responsible for how outputs are used. Without governance, nobody owns the risk.
- Security risk: Prompt injection via emails or documents, data exfiltration through assistant features, over-privileged AI agents acting on behalf of users, and unreliable outputs feeding decisions.

In exam terms, this topic sits squarely in the AI Governance and Program Management domain but draws heavily on third-party risk management, data governance and change management. Questions test whether you can extend an existing governance program to cover AI the organisation did not build and cannot fully control.

2. What Embedded AI Features Are

Embedded AI features are AI or machine learning capabilities delivered inside enterprise software that the organisation already uses, rather than as a standalone AI system. Characteristics include:

- Delivered by the software vendor, often through SaaS updates, with little or no customer involvement in design.
- Frequently enabled by default or enabled at tenant level by an administrator with a single toggle.
- Built on third-party foundation models (for example, a vendor using a hyperscaler's large language model), creating a fourth-party dependency.
- Governed by vendor terms of service, data processing agreements and AI-specific addenda that may change over time.
- Accessing the same data the application already holds, which means the AI feature inherits the application's entire data access footprint.

Typical examples: generative assistants in office and email suites, meeting transcription and summarisation, AI-driven lead scoring in CRM, resume screening and skills inference in HR platforms, chatbot features in service desks, code completion in development tools, anomaly detection in finance systems, and AI agents that can take actions across connected applications.

The governance challenge is that the organisation occupies the deployer role in the AI supply chain. It cannot change the model, retrain it or audit its training data, yet it remains accountable for the decisions, data exposure and harms that result from use within its environment. This is a shared responsibility model: the vendor is responsible for model development, security of the service and disclosed limitations; the customer is responsible for configuration, access, data input, output use, user behaviour and oversight.

3. How Managing Embedded AI Works: The Governance Lifecycle

Step 1 - Establish policy and ownership
Before any technical action, the AI governance framework (policy, principles, risk appetite) must explicitly state that embedded and vendor-supplied AI is in scope. Assign accountability: typically an AI governance committee or steering group sets direction, application or business owners are accountable for features in their systems, and the security, privacy, legal and procurement functions provide second-line oversight. A RACI matrix clarifying who may approve enabling a feature is a frequent exam answer.

Step 2 - Discover and inventory
You cannot govern what you do not know exists. Discovery methods include: vendor release notes and roadmap reviews, administrator console audits of feature flags, SaaS security posture management (SSPM) and cloud access security broker (CASB) telemetry, software asset management records, network and DNS monitoring for AI API endpoints, licence and procurement reviews, and user surveys. The output feeds the AI inventory / AI register, recording the feature, the hosting application, the underlying model and provider, data accessed, purpose, risk classification, owner and status (enabled, disabled, under review).

Step 3 - Risk assess and classify
Apply the organisation's AI risk assessment methodology to each feature, proportionate to impact. Key questions: What data is sent to the model? Where is it processed and stored? Is customer data used for training? Does the feature make or materially influence decisions about people (employment, credit, access)? Can it take autonomous actions? What is the regulatory classification (for example, EU AI Act high-risk categories)? Risk-rank features so that effort is concentrated where harm is greatest.

Step 4 - Vendor and contractual due diligence
Embedded AI must be folded into third-party risk management. Review AI-specific terms: data use and retention, training opt-outs, sub-processor and fourth-party disclosure, model documentation (model cards, system cards), security certifications, incident notification, data residency, indemnities and the right to be informed of material changes. Negotiate addenda where terms are unacceptable. Maintain evidence for audit and regulators.

Step 5 - Configure and control
Preferred control hierarchy: disable features that are not approved; where approved, apply least privilege, restrict to pilot groups, enforce tenant-level settings that prevent data training and external sharing, apply data loss prevention and sensitivity labelling to limit what the assistant can access, enforce logging and retention of prompts and outputs where required, and require human review for consequential outputs. Default posture favoured by ISACA guidance: disabled or restricted until assessed and approved, then enabled in a controlled manner, rather than blanket blocking or blanket acceptance.

Step 6 - Change management and monitoring
Vendors change models, add features and alter terms frequently. Integrate vendor AI release monitoring into the change management process so new capabilities trigger reassessment rather than silently activating. Monitor usage logs, DLP alerts, access patterns, user feedback and output quality. Report metrics (coverage of inventory, percentage of features assessed, open exceptions, incidents) to the governance committee.

Step 7 - Awareness and acceptable use
Update the acceptable use policy to cover embedded assistants: what data may be entered, how outputs must be verified, disclosure requirements when AI-generated content is shared, and how to report concerns. Train administrators specifically, because a single toggle can change the organisation's risk profile.

Step 8 - Review, audit and decommission
Periodically re-evaluate features against risk appetite and regulation, verify vendor attestations, and retire or disable features that no longer deliver value or exceed acceptable risk. Ensure exit arrangements cover deletion of data held by the AI service.

4. Key Concepts the Exam Expects You to Connect

- Deployer versus provider obligations: Using a vendor's AI does not transfer accountability for outcomes.
- Shared responsibility model: Mirrors cloud security; the customer owns configuration, data, identity and use.
- AI inventory as a foundational control: Governance, risk assessment and compliance all depend on it.
- Third-party and fourth-party risk: The model provider behind the software vendor may be the real data recipient.
- Risk-based and proportionate: Not every feature needs a full impact assessment; high-impact features do.
- Three lines model: Business owns the risk, security/privacy/legal oversee, audit assures.
- Change management linkage: Vendor-pushed features are changes to the control environment.
- Enable securely rather than prohibit: ISACA favours balancing business value with risk; outright bans drive shadow AI.

5. Exam Tips: Answering Questions on Managing Embedded AI Features in Enterprise Software

Tip 1 - Look for the governance-first answer. When a question asks what should be done FIRST, the answer is usually to establish policy and ownership or to inventory and assess, not to buy a tool or disable everything. ISACA consistently places policy, accountability and risk assessment before technical controls.

Tip 2 - Inventory and discovery are the foundational control. If options include updating the AI inventory, performing discovery of enabled features, or reviewing admin consoles, that is frequently the BEST or FIRST step when the scenario reveals the organisation does not know what AI is running.

Tip 3 - Remember the organisation is the deployer. Any option implying that the vendor bears full responsibility for outcomes, compliance or data protection is a distractor. The organisation remains accountable for how the feature is used and what data it exposes.

Tip 4 - Prefer contractual plus configuration controls over trust. When a vendor says it does not train on customer data, the BEST answer is to obtain contractual commitments and verify the tenant configuration, not to accept the assurance or rely on a marketing statement.

Tip 5 - Default-deny until assessed, but not permanent prohibition. If choices include blanket ban, blanket enable, or disable pending risk assessment with controlled enablement, the last is almost always correct. Permanent bans are rarely the ISACA answer because they ignore business value and encourage shadow AI.

Tip 6 - Tie vendor updates to change management. Scenarios describing an AI feature that appeared after a software update point to a gap in change and vendor release management. The BEST answer usually involves integrating vendor AI roadmap monitoring into change control and reassessment, not reprimanding users.

Tip 7 - Escalate decisions to the right level. Enabling a high-risk feature (for example, AI screening of job applicants) should go to the AI governance committee or senior management with legal and privacy input, not be decided by an IT administrator. Watch for options that place the decision with the wrong role.

Tip 8 - Distinguish between risk assessment and risk response. Questions about what the outcome of due diligence should be expect risk treatment options (accept, mitigate via configuration, transfer via contract, avoid by disabling), aligned with risk appetite.

Tip 9 - Watch for privacy and data residency cues. If the scenario mentions personal data, regulated data or cross-border processing, the correct answer will involve the privacy function, data processing agreements, residency controls or impact assessments, not purely a security tool.

Tip 10 - Eliminate extremes and absolutes. Options containing words like always, never, all, or immediately disable everything are usually wrong. Look for the balanced, risk-based, documented, accountable choice.

6. Worked Example Questions

Example 1. An organisation discovers that its email platform vendor has enabled a generative summarisation assistant for all users following a routine update. Which of the following should the AI security manager do FIRST?
A. Disable the feature permanently across the tenant.
B. Add the feature to the AI inventory and initiate a risk assessment.
C. Issue a communication telling users not to use the assistant.
D. Request the vendor's model card.

Best answer: B. Inventorying and assessing is the governance foundation; A is an extreme response made before understanding risk; C is a stopgap without governance; D is part of the assessment but not the first overarching action.

Example 2. A SaaS HR platform offers an AI feature that ranks candidates. Legal confirms this is a high-risk use under applicable regulation. Which control is MOST important before enabling it?
A. Enabling detailed audit logging.
B. Establishing human review and override of AI rankings with documented accountability.
C. Confirming the vendor holds a security certification.
D. Training recruiters on how to interpret scores.

Best answer: B. For high-risk decisions affecting individuals, meaningful human oversight with clear accountability is the core deployer obligation; the other options support it but are secondary.

Example 3. A vendor states verbally that customer prompts are never used to train its models. What is the BEST response?
A. Accept the statement and enable the feature.
B. Obtain the commitment in the contract or data processing addendum and verify tenant settings.
C. Block all traffic to the vendor.
D. Ask users to avoid entering sensitive data.

Best answer: B. Contractual commitment plus technical verification converts an assurance into an enforceable, auditable control.

7. Summary

Managing embedded AI features is about extending mature governance disciplines (inventory, risk assessment, third-party risk management, change control, policy and awareness) to AI that arrives inside software the organisation already uses. The organisation is the deployer, accountability cannot be outsourced, and the correct posture is risk-based and proportionate: discover, assess, contract, configure, monitor and review. On the exam, favour answers that begin with governance and visibility, assign accountability to the right role, use contracts and configuration rather than trust, and enable AI securely instead of banning it outright.