Data Freshness and Relevance for Training

Overview
In the PMI Cognitive Project Management for AI (CPMAI) methodology, Phase II: Data Understanding asks the project team to identify what data is needed, where it comes from, and whether it is fit for purpose. Two of the most important fitness tests are data freshness and data relevance.

A model learns patterns from historical data. If that data is outdated, or does not represent the real-world problem the model will face, the model fails no matter how good the algorithm is. CPMAI stresses that AI projects fail far more often because of data problems than because of algorithm problems. Freshness and relevance are central to avoiding that failure.

Why It Is Important
1. Models reflect the data they are trained on. A model trained on stale patterns makes stale predictions. Consumer behavior, fraud tactics, prices, language use and regulations all change over time.
2. Drift is inevitable. Data drift is a change in the distribution of the inputs. Concept drift is a change in the relationship between the inputs and the target. Both erode accuracy after deployment. Planning for freshness up front reduces the impact.
3. Business value depends on real-world fit. A highly accurate model that is accurate on the wrong population, time period or context delivers no business value. It can also cause harm.
4. Risk, bias and compliance. Irrelevant or outdated data can carry historical biases or describe conditions that no longer apply, such as old policies or legacy demographics. This creates ethical and regulatory exposure.
5. Cost and effort. Collecting, cleaning and labeling data is expensive. Spending that effort on irrelevant or expired data wastes budget and schedule.
6. Go/no-go decisions. CPMAI is iterative. If fresh, relevant data cannot be obtained, the team should loop back to Business Understanding, rescope, or decide not to proceed.

What It Is
Data Freshness (also called timeliness, recency or currency) is how current the data is compared with the conditions the model will operate in. Key questions:
- When was the data collected or last updated?
- How quickly does the underlying phenomenon change?
- What is the time lag between a real-world event and its availability in the dataset (data latency)?
- How often must new data arrive to keep the model valid (refresh cadence)?

Data Relevance is how well the data relates to and represents the specific problem, target variable, population and operating context defined in Business Understanding. Key questions:
- Does the data contain features that plausibly influence the outcome we want to predict?
- Does it represent the same population, geography, channel, product line and conditions as production?
- Is the label or target defined the same way the business defines success?
- Will these same features be available at prediction time? This avoids data leakage and training-serving skew.

Freshness and relevance overlap. Data can be relevant in content but stale in time, such as pre-pandemic shopping data used to predict post-pandemic demand. Data can also be fresh but irrelevant, such as yesterday's data from a different market segment.

How It Works in Practice
Step 1: Anchor to the business problem. Start from the AI pattern and objective identified in Phase I, for example recognition, predictive analytics, conversational, anomaly detection or hyperpersonalization. Define the target, the decision it supports and the operating environment.

Step 2: Inventory candidate data sources. List internal and external sources. For each, record:
- collection dates and update frequency
- owner
- access rights
- format
- population covered

Step 3: Assess relevance.
- Map each source to the problem: does it carry signal for the target?
- Check representativeness against production. Look for coverage gaps such as missing regions, customer types or edge cases.
- Confirm label definitions and quality.
- Remove features that will not exist at inference time or that leak the answer.
- Use subject matter experts to validate that features make domain sense.

Step 4: Assess freshness.
- Determine how volatile the domain is. A fraud or news-recommendation domain is highly volatile. Medical imaging anatomy is relatively stable.
- Choose an appropriate time window for training. More data is not always better if older records reflect obsolete patterns.
- Identify regime changes such as policy changes, product launches, economic shocks or system migrations. Decide whether pre-change data should be excluded, down-weighted or kept.
- Measure data latency and confirm that the pipeline can deliver fresh data at the needed cadence.

Step 5: Plan for ongoing freshness. Define the following, which connect to Phase VI, Operationalization:
- retraining triggers, either scheduled (time-based) or performance or drift-based
- monitoring metrics
- data pipeline SLAs
CPMAI treats models as living systems that need continuous data updates.

Step 6: Document and decide. Capture findings in the data requirements or data understanding documentation. If gaps exist, options include:
- acquiring new data
- data augmentation or synthetic data, used carefully
- narrowing the scope
- iterating back to Phase I
- a no-go decision

Common Techniques and Concepts
- Time-based train/test splits. Validate on the most recent period to simulate future performance, instead of using random splits that hide temporal effects.
- Sliding or rolling windows. Train on the most recent N months.
- Sample weighting. Give recent records more weight.
- Drift detection. Use statistical tests and monitoring of input distributions and prediction performance.
- Data lineage and metadata. Use timestamps and versioning so that freshness can be verified.
- Feature availability check. Confirm each feature is available in production at the needed latency.

Example Scenarios
- A retailer trains a demand forecaster on five years of data, but a major shift to online ordering happened 18 months ago. Older data is relevant in content but may misrepresent current patterns. The best action is to investigate the regime change and consider weighting or limiting the window.
- A bank builds a fraud model on data from one region and plans to deploy it globally. The data is fresh but not representative. This is a relevance problem.
- A chatbot is trained on product documentation from two product versions ago. The data is stale and partly irrelevant, so answers will be wrong. The fix is to refresh the knowledge source and establish an update process.
- A model includes a feature that is populated only after a claim is approved. That feature is unavailable at prediction time. This is leakage, a relevance and availability failure.

Exam Tips: Answering Questions on Data Freshness and Relevance for Training
1. Think data first. CPMAI emphasizes that data issues are the top cause of AI project failure. When a scenario describes declining accuracy, poor real-world results or a mismatch between test and production performance, suspect stale or unrepresentative data before blaming the algorithm.
2. Link back to Business Understanding. Relevance is always judged against the business objective and the problem definition. The best answer often mentions confirming the data aligns with the defined problem, target and operating context.
3. Prefer investigation and validation over immediate action. Answers like assess, validate with SMEs, analyze data for representativeness or review collection dates are usually better than jumping straight to retraining with a new algorithm or buying more compute.
4. More data is not automatically better. Watch for distractors claiming that adding all historical data will improve the model. If the domain has changed, older data may hurt performance.
5. Recognize drift vocabulary. Data drift means the input distribution changed. Concept drift means the input-to-output relationship changed. The correct remedies typically involve monitoring, retraining with recent data and establishing a refresh cadence.
6. Freshness is ongoing, not one-time. Answers that include continuous monitoring, retraining triggers and pipelines in operationalization reflect the iterative CPMAI mindset.
7. Watch for availability at inference time. If a feature will not exist or will be delayed when predictions are made, it is not truly relevant for training. This is a leakage clue.
8. Representativeness equals relevance. Phrases such as different region, different customer segment, collected under old policy or missing edge cases point to relevance gaps.
9. Know the fallback options. If fresh, relevant data cannot be obtained, the CPMAI answer is to iterate. This means revisiting scope, acquiring data, or making a go/no-go decision. Proceeding anyway and fixing it later is not the answer.
10. Eliminate algorithm-centric distractors. Options that focus on hyperparameter tuning, switching to deep learning or adding model complexity rarely fix a freshness or relevance problem.
11. Consider ethics and compliance. Outdated data may embed past biases or violate current consent and regulation. Answers addressing responsible AI considerations are often strong.
12. Use phase cues. If the question is set in Data Understanding, the answer typically involves assessing and documenting data. If it is set in Operationalization, it involves monitoring and refreshing.

Quick Recap
- Freshness: Is the data current enough for how fast the domain changes?
- Relevance: Does the data match the problem, population, target and production conditions?
- Process: Anchor to business goals, inventory sources, assess relevance and freshness, plan ongoing refresh, then document and decide.
- Exam mindset: Data first, validate before acting, iterate when gaps exist, and avoid algorithm-only fixes.