Data Collection Processes for Ongoing Feeds

Overview
In the PMI Cognitive Project Management for AI (CPMAI) methodology, Phase II: Data Understanding asks the team to identify which data the AI solution needs, where it comes from, and how it will keep arriving. Data Collection Processes for Ongoing Feeds covers the repeatable mechanisms that keep supplying fresh, reliable data to an AI system after the first training dataset has been assembled. A one-time data pull may be enough for a prototype. A production AI system needs a steady, governed, monitored flow of data to make inferences, retrain models, and stay accurate as the real world changes.

Why It Is Important
1. AI is data-centric, not code-centric. CPMAI stresses that AI projects fail more often because of data problems than algorithm problems. A model is only as good as the data it keeps receiving.
2. Model drift and data drift. Real-world patterns change: customer behavior, sensor conditions, fraud tactics, language use. Without ongoing feeds the model goes stale. Accuracy degrades, sometimes silently.
3. Operationalization depends on it. In Phase VI (Model Operationalization), the model needs production data in the same format, quality, and timeliness it was trained on. If the feed is not designed early, deployment stalls.
4. Avoiding the proof-of-concept trap. Many AI pilots succeed on a curated historical dataset but cannot move to production because no sustainable data pipeline exists. CPMAI warns against this.
5. Cost, compliance, and risk. Ongoing feeds bring recurring costs (licensing, storage, labeling), privacy obligations (consent, retention, PII handling), and security risks. These must be planned, budgeted, and governed.
6. Go/No-Go decisions. CPMAI is iterative and includes go/no-go checkpoints. If a sustainable data source cannot be secured, the project may not be feasible. Identifying this early saves money.

What It Is
An ongoing data feed is a continuous or recurring flow of data from one or more sources into the AI system's data environment. The data collection process defines:
- Sources: internal systems (ERP, CRM, transaction logs), external providers (third-party APIs, purchased datasets), IoT sensors, user interactions, web data, and human-generated labels.
- Frequency and latency: real-time streaming, near-real-time micro-batches, or scheduled batches (hourly, daily, weekly).
- Format and schema: structured, semi-structured (JSON, XML), or unstructured (text, images, audio, video).
- Volume and velocity: how much data arrives and how quickly. These are the Big Data Vs: Volume, Velocity, Variety, Veracity, Value.
- Ownership and access rights: who owns the data, who approves access, and the licensing or contractual terms.
- Quality controls: validation rules, completeness checks, deduplication, anomaly detection.
- Labeling processes: for supervised learning, how new data gets ground-truth labels (human-in-the-loop, crowdsourcing, automated labeling, feedback loops).
- Governance: privacy, consent, retention, lineage, auditability, and regulatory compliance (e.g., GDPR, HIPAA).

How It Works
Step 1: Identify data requirements from the business problem. In Phase I (Business Understanding) the team defines the AI pattern (e.g., recognition, predictive analytics, conversational, patterns and anomalies, hyperpersonalization, goal-driven systems, autonomous systems). Each pattern implies different data needs. A fraud anomaly detector needs streaming transaction data. A demand forecaster may need daily batch sales data plus external weather or economic feeds.

Step 2: Inventory and assess candidate sources. Determine whether data exists, where it lives, whether it is accessible, and whether it can be supplied continuously, not just once. Assess quality, representativeness, bias, and stability of each source.

Step 3: Define collection mechanisms.
- Batch ingestion: scheduled ETL/ELT jobs extracting from databases or files.
- Streaming ingestion: event streams, message queues, or IoT gateways for low-latency needs.
- API integration: pulling from internal or third-party services, with rate limits and SLAs.
- Change data capture (CDC): capturing only new or modified records.
- Human-generated data: annotation workflows, surveys, user feedback.
- Feedback loops: capturing model predictions and actual outcomes so they become future training data.

Step 4: Establish data quality and validation gates. Automated checks for schema conformance, missing values, out-of-range values, duplicates, and distribution shifts. Bad data is quarantined rather than flowing silently into models.

Step 5: Plan labeling for ongoing data. Decide how new data will be labeled, by whom, at what cost, and with what quality assurance (inter-annotator agreement, gold-standard samples). Labeling is often the most underestimated ongoing cost.

Step 6: Address governance and compliance. Confirm consent and legal basis for continuous collection. Define retention and deletion policies, anonymization or pseudonymization, access controls, and lineage tracking. Ensure third-party contracts allow ongoing use for AI training.

Step 7: Assign ownership and SLAs. Name data owners and data stewards. Agree on service levels for availability, timeliness, and quality with source system owners and vendors.

Step 8: Monitor and iterate. Monitor feed health (latency, failures, volume anomalies) and data drift. Feed findings back into CPMAI's iterative cycle. Drift may trigger retraining (Phase IV Model Development again) or re-examination of data needs (Phase II).

Key Concepts to Know
- Training data vs. inference data: both need ongoing feeds, and they must stay consistent to avoid training-serving skew.
- Data drift: change in input distributions.
- Concept drift: change in the relationship between inputs and the target.
- Data lineage: tracing where data came from and how it was transformed.
- Data versioning: snapshots of datasets so model versions can be reproduced.
- Human-in-the-loop: people validating or labeling data continuously.
- Data quantity sufficiency: enough new data must arrive to support retraining.
- Representativeness and bias: ongoing feeds can introduce new bias if populations shift or sources change.

Common Pitfalls
- Assuming historical data alone is enough for a production AI system.
- Ignoring the cost and effort of ongoing labeling.
- Relying on a third-party source without contractual rights for continuous use.
- Having no monitoring, so the feed breaks or degrades silently.
- Mismatched formats between training data and live data.
- Overlooking privacy consent for new or expanded uses.
- No clear data owner accountable for feed quality.

Example Scenario
A retailer builds a product recommendation engine (hyperpersonalization pattern). Initial training uses two years of purchase history. For production, the team designs ongoing feeds:
- Clickstream events streamed in near real time.
- Daily batch updates of inventory and pricing.
- Weekly ingestion of returns data as a negative-feedback signal.
- A feedback loop recording which recommendations were clicked or purchased.

They add validation gates and drift monitoring, assign a data steward, confirm customer consent under the privacy policy, and schedule monthly retraining. Without these feeds, recommendations would become irrelevant as seasons and products change.

Exam Tips: Answering Questions on Data Collection Processes for Ongoing Feeds
1. Think beyond the initial dataset. If a scenario describes a model that worked in a pilot but degrades in production, the answer usually involves establishing ongoing data feeds, monitoring, and retraining. It is rarely just picking a better algorithm.
2. Data first, algorithm second. CPMAI favors answers that address data availability, quality, and sustainability before model tuning. Distractors often suggest jumping to model changes.
3. Know where it fits. Identifying ongoing data needs belongs to Phase II: Data Understanding. Building pipelines belongs to Phase III: Data Preparation. Running them in production belongs to Phase VI: Operationalization. Questions may test which phase an activity belongs to.
4. Look for feasibility red flags. If no sustainable data source exists, the best answer is often to raise it at a go/no-go decision, revisit scope, or find alternative sources. Proceeding and hoping for the best is the wrong choice.
5. Remember labeling. For supervised learning scenarios, the correct answer frequently includes planning how new data will be labeled continuously.
6. Governance is never optional. Answers that include consent, privacy, data rights, and compliance for ongoing collection usually beat answers that ignore them, especially with third-party or personal data.
7. Monitoring and drift. Pick answers that include automated quality checks and drift detection over manual, ad hoc reviews.
8. Match frequency to the use case. Real-time fraud detection needs streaming. Monthly forecasting may only need batch. Choose the option that fits the business requirement, not the most sophisticated technology.
9. Ownership and stakeholders. When a question asks for the best first step, look for engaging data owners and stakeholders to confirm access, SLAs, and responsibilities.
10. Watch for training-serving consistency. If the live data differs in format or distribution from the training data, the right answer aligns the pipelines and validates the inputs.
11. Iterative mindset. CPMAI is iterative and agile. The best answers often loop back to earlier phases when data issues appear, rather than pushing forward.
12. Eliminate extreme answers. Options like 'collect all possible data' or 'stop the project immediately' are usually wrong. Prefer balanced, risk-aware, business-aligned choices.

Quick Recap
Ongoing data feeds keep AI systems accurate and relevant after deployment. Plan them early by:
- identifying sources, frequency, format, and ownership;
- building validation and labeling processes;
- securing governance and compliance;
- monitoring for failures and drift.

In exam questions, favor answers that are data-centric, sustainable, governed, monitored, and iterative.