Data Flow and Processing Requirements

Overview
In the PMI Cognitive Project Management for AI (PMI-CPMAI) methodology, Data Flow and Processing Requirements sits within the early phase of Identifying Business Needs and Solutions (closely tied to CPMAI Phase I, Business Understanding, and leading into Phase II, Data Understanding). It is the activity of working out how data will move from its sources, through transformation and processing, into an AI model, and then out to the people and systems that act on the results. It covers both training and production (inference).

CPMAI treats AI projects as data projects first. A business need cannot become a workable AI solution until the team knows what data will flow, at what speed, in what volume and format, and under what constraints.

Why It Is Important
1. AI projects fail mostly because of data. CPMAI stresses that most AI project failures come from data problems: data that is missing, siloed, low quality, late or inaccessible. They rarely come from algorithm problems. Mapping data flow early exposes these risks before money is spent.

2. It decides whether the project is feasible. A use case such as real-time fraud detection needs streaming data with very low latency. If the organization only has nightly batch extracts, the solution is not feasible as framed. In that case the project must be rescoped.

3. It drives infrastructure, cost and timeline. Processing requirements shape decisions about:
- cloud vs. on-premise vs. edge,
- storage (data lakes, warehouses),
- compute (GPUs, distributed processing),
- integration effort.
These decisions directly affect budget and schedule.

4. It supports Go/No-Go decisions. CPMAI includes AI Go/No-Go checks in early phases. Data availability, accessibility and processing feasibility are key inputs.

5. It enables governance, privacy and trustworthy AI. Knowing where data travels shows where sensitive data (PII, PHI, financial data) is exposed. This supports compliance with regulations such as GDPR and HIPAA, and with internal policies.

6. It prevents training-serving mismatch. Production data flows must match the data used in training. Otherwise model performance degrades.

What It Is
Data flow and processing requirements describe the following elements.

Data sources: internal systems (ERP, CRM, transactional databases), external sources (third-party data, public datasets, APIs), sensors/IoT, documents, images, audio and logs.

Data types and structure: structured (tables), semi-structured (JSON, XML) and unstructured (text, images, video, audio).

The Vs of data:
- Volume: how much data.
- Velocity: how fast it arrives and must be processed.
- Variety: how many formats.
- Veracity: how trustworthy and accurate it is.
- Value: its usefulness for the business objective.

Ingestion method:
- batch (scheduled, periodic),
- real-time/streaming (continuous),
- micro-batch,
- event-driven.

Processing and transformation: cleansing, deduplication, normalization, enrichment, feature engineering, labeling/annotation, aggregation, anonymization, and format conversion. Pipelines may follow ETL (Extract-Transform-Load) or ELT (Extract-Load-Transform) patterns.

Storage: data lakes, data warehouses, feature stores, and vector databases, with their retention needs.

Model consumption:
- Training flow: historical data, often large volume, processed in batch.
- Inference flow: new data arriving in production, which may require low latency.

Output and delivery: where predictions go, such as dashboards, applications, APIs, downstream systems, or human reviewers in human-in-the-loop designs.

Feedback loops: how outcomes and new labeled data return to the system for monitoring and retraining.

Non-functional requirements: latency, throughput, scalability, availability, security, privacy, lineage, auditability and cost.

How It Works (Step by Step)
Step 1 - Start with the business need. Clarify the problem, the decision being supported and the success criteria. For example: "Reduce customer churn by 10% by flagging at-risk customers weekly." The word weekly already signals that batch processing may be enough.

Step 2 - Identify the AI pattern. CPMAI's seven patterns are:
- recognition,
- conversational/human interaction,
- predictive analytics and decision support,
- patterns and anomalies,
- hyperpersonalization,
- autonomous systems,
- goal-driven systems.
Each pattern implies typical data needs. Recognition needs labeled images or audio. Anomaly detection often needs streaming data. Conversational systems need text corpora.

Step 3 - Inventory data sources. For each source, determine:
- who owns it,
- whether it is accessible,
- its quality,
- whether there are legal or contractual limits on its use.

Step 4 - Map the end-to-end flow. Create a data flow diagram running from source, to ingestion, to processing/preparation, to storage, to model training and inference, to output, and back through the feedback loop. Show handoffs, systems and responsible teams.

Step 5 - Define processing requirements. Specify:
- frequency (real-time vs. batch),
- latency targets (e.g., under 200 ms per prediction),
- volume and growth expectations,
- transformation and labeling needs,
- compute requirements.

Step 6 - Address governance and security. Identify sensitive data, encryption, access control, anonymization, consent, data residency, lineage tracking and audit requirements.

Step 7 - Assess gaps and feasibility. Compare what the solution needs against what exists. Gaps might include:
- missing labels,
- siloed systems,
- lack of streaming infrastructure,
- insufficient historical data.
Then decide how to fill each gap: data acquisition, augmentation, synthetic data, labeling, new integrations, or rescoping the project.

Step 8 - Feed the Go/No-Go decision and plan iteratively. CPMAI is iterative and agile. Start with a minimal viable data flow for a small-scope pilot, then expand in later iterations.

Example
A retailer wants product recommendations on its website.
- Sources: clickstream (streaming), purchase history (batch from the ERP), product catalog (semi-structured).
- Requirements: real-time inference under 100 ms; nightly batch retraining; anonymized user IDs; a feature store to keep training and serving consistent; a feedback loop that captures clicks on recommendations.
- Gap found: clickstream data is not currently captured in a usable form. Instrumentation must be added before modeling can begin.

Key Terms to Know
- Data pipeline
- ETL/ELT
- Batch vs. streaming
- Latency
- Throughput
- Data lineage
- Data provenance
- Feature store
- Data drift
- Training-serving skew
- Data silos
- Data labeling
- Human-in-the-loop
- Edge processing
- Data governance
- Data stewardship
- Minimum viable data

Exam Tips: Answering Questions on Data Flow and Processing Requirements
1. Think data first. When a scenario asks what the project manager should do next, and data sources, access or quality have not been confirmed, the best answer is usually to assess or validate the data. Choosing algorithms or building models comes later. CPMAI places Data Understanding before Data Preparation and Modeling.

2. Match processing to the business need. Words like instantly, live, immediately or at the point of transaction point to real-time/streaming requirements. Words like monthly reports, weekly lists or overnight point to batch. Do not pick an expensive real-time design when batch meets the requirement.

3. Watch for feasibility red flags. Watch for:
- data that does not exist,
- data with no labels,
- data locked in silos,
- data whose use is legally restricted.
The correct answer often involves addressing the data gap, rescoping, or making a No-Go decision rather than proceeding.

4. Remember both flows. Questions may test whether you understand that training data flows and production/inference data flows differ. Both must be planned. Answers that consider only training are often incomplete.

5. Prioritize governance and privacy. If a scenario involves personal or regulated data, prefer answers that include anonymization, access control, consent, compliance review and lineage. Avoid answers that skip them to save time.

6. Choose stakeholder engagement. Data owners, IT/data engineering, security and compliance teams should be involved in defining data flows. Answers that involve the right stakeholders early are usually stronger than answers where the PM decides alone.

7. Favor iterative, small-scope approaches. CPMAI promotes starting small and iterating. An answer suggesting a pilot with a limited, well-understood data flow often beats a "build the full enterprise pipeline first" answer.

8. Don't forget feedback and monitoring. AI systems degrade through data drift. Answers that include feedback loops, monitoring and retraining pipelines reflect CPMAI's lifecycle thinking.

9. Eliminate technology-first distractors. Options that jump to selecting a specific tool, vendor, cloud platform or algorithm before requirements are understood are usually wrong in the business-needs phase.

10. Read for the phase. Identify where the scenario sits in the CPMAI lifecycle:
- Business Understanding,
- Data Understanding,
- Data Preparation,
- Model Development,
- Model Evaluation,
- Model Operationalization.
The best answer is the one appropriate to the current phase.

Quick Practice Example
Question: A bank wants AI to flag fraudulent card transactions before they are approved. The team has three years of historical transaction data in a nightly-updated warehouse. What should the project manager address first?

Best answer: Confirm that a real-time data ingestion and inference flow can deliver transaction data to the model within the approval latency window. The historical batch data supports training. However, the business need requires real-time processing in production, so this requirement determines feasibility.

Why the others are wrong: Answers such as "select a deep learning algorithm", "start training on the warehouse data immediately" or "increase the warehouse refresh to hourly" ignore the core processing requirement or address it inadequately.

Summary
Data flow and processing requirements turn a business need into a concrete, feasible data blueprint for an AI solution. They define sources, movement, transformation, storage, consumption, output and feedback, together with performance, security and governance constraints. On the exam, the strongest answers:
- are data-first and match the processing approach to the business need,
- respect privacy and governance,
- involve stakeholders,
- plan for both training and production,
- follow CPMAI's iterative, phase-appropriate approach.