Data Schema and Structure for Modeling Compatibility

Overview
In the PMI Cognitive Project Management for AI (CPMAI) methodology, the Data Understanding phase (Phase II) is where the team works out what data the AI solution needs. One key part of that work is checking whether the data's schema and structure fit the modeling approach chosen in Business Understanding (Phase I).

A data schema is the blueprint of a dataset. It names the fields (features), their data types, formats, relationships, keys, allowed values and constraints. Data structure is how the data is organized:
- Structured: tables and rows.
- Semi-structured: JSON, XML, logs.
- Unstructured: text, images, audio, video.

Modeling compatibility asks one question: can this data, as it is shaped today or after reasonable preparation, be used by the algorithm or AI pattern the project intends to use?

Why It Is Important
1. Data is the main cause of AI project failure. CPMAI stresses that most AI projects fail because of data problems, not algorithm problems. A mismatch between schema and model needs is a common, avoidable failure point.
2. It prevents wasted effort. If you only find out during modeling that the data lacks labels, has the wrong time granularity, or cannot be joined across sources, the team must loop back. That costs time and money.
3. It supports the go/no-go decision. CPMAI is iterative and includes checkpoints. Knowing early whether the data is structurally usable helps decide whether to proceed, re-scope or stop.
4. It drives data preparation planning. Schema gaps tell Phase III (Data Preparation) what to do, such as transformation, normalization, encoding, feature engineering, labeling or merging.
5. It keeps the AI pattern and the data aligned. Each of the seven AI patterns (recognition, conversational, predictive analytics, patterns and anomalies, hyperpersonalization, autonomous systems, goal-driven systems) has typical data needs. For example:
- Recognition needs labeled images or audio.
- Predictive analytics needs historical records with a target variable.

What It Covers
1. Data types and formats: numeric, categorical, ordinal, text, datetime, binary, image and so on. Many algorithms need numeric input, so categorical fields must be encoded.
2. Feature availability: are the attributes needed to predict the outcome actually captured?
3. Target or label presence: supervised learning needs labeled outcomes. If labels are missing, consider unsupervised methods or a labeling effort.
4. Granularity and grain: is each record at the right level (transaction, customer, day, sensor reading)? A grain mismatch causes aggregation or leakage problems.
5. Keys and relationships: can datasets be joined reliably through primary and foreign keys or entity resolution?
6. Temporal structure: time-series models need consistent timestamps, regular intervals and ordering.
7. Dimensionality and volume: the number of features compared with the number of samples, and whether there is enough data for the chosen technique.
8. Consistency and constraints: uniform units, encodings, naming and value ranges across sources.
9. Metadata and data dictionaries: documentation that explains each field's meaning, source and lineage.
10. Format for the target platform: whether the data can be ingested by the tools or pipelines you plan to use, such as tensors for deep learning, flat tables for gradient boosting, or token sequences for NLP.

How It Works in Practice
Step 1: Start from the business problem and AI pattern. Identify the model type you expect to use (classification, regression, clustering, NLP and so on) and its input requirements.
Step 2: Inventory the data sources. List internal and external sources, their owners, access rights and formats.
Step 3: Profile and document the schemas. Review data dictionaries, run profiling (types, null rates, distributions, cardinality) and map entity relationships.
Step 4: Run a gap analysis. Compare what the model needs with what the data provides. Typical gaps:
- missing features
- missing labels
- wrong grain
- incompatible formats
- unjoinable sources
Step 5: Decide on remediation. Options include:
- transform (encode, normalize, reshape or pivot)
- enrich (acquire external data)
- label (human-in-the-loop annotation)
- re-scope (change the AI pattern or problem framing)
Step 6: Feed the findings into Data Preparation and the go/no-go decision. Document the requirements and record risks.

Example: A retailer wants to predict customer churn. The CRM data is at the transaction grain with no churn flag. The team must:
- aggregate the data to the customer grain,
- define and derive a churn label from inactivity rules,
- join support tickets using a customer ID that is formatted differently across systems.
Finding all this in Phase II keeps the plan realistic.

Common Pitfalls
- Assuming that having lots of data means having the right data.
- Ignoring schema differences when merging sources.
- Overlooking data leakage, where fields would not be available at prediction time.
- Treating unstructured data as model-ready without a feature extraction plan.
- Skipping documentation, which hurts reproducibility and governance.

Exam Tips: Answering Questions on Data Schema and Structure for Modeling Compatibility
1. Know the phase. Schema and structure assessment belongs to Data Understanding (Phase II). The actual transformation work (cleaning, encoding, reshaping) belongs to Data Preparation (Phase III). If a question asks what to do first, assess and document before you transform.
2. Data comes before the algorithm. When a scenario shows the data does not fit the planned model, the best answer is usually to analyze the gap, then adjust the data plan or re-scope. Jumping to a more complex algorithm is rarely right.
3. Watch for label keywords. If a scenario says there are no historical outcomes, or the data is unlabeled, think about labeling effort, unsupervised approaches, or re-framing the problem.
4. Grain and joins are favorite traps. Answers that mention aligning granularity, entity resolution or consistent keys are often correct when data comes from several systems.
5. Iteration is valid. CPMAI is iterative. Going back to Business Understanding when the data cannot support the goal is a correct, mature response, not a failure.
6. Prefer documentation and stakeholder involvement. Correct answers often include consulting data owners or SMEs, building a data dictionary, and recording data requirements.
7. Eliminate extreme choices. Reject options like 'proceed to modeling and fix issues later' or 'cancel the project immediately' unless the scenario clearly supports them.
8. Match the AI pattern to the data structure. Recognition needs unstructured, labeled data. Predictive analytics needs structured historical data with a target. Conversational needs text or speech corpora. Anomaly detection may work without labels.
9. Consider prediction-time availability. Any feature used must be available when the model runs in production. Answers that flag leakage or unavailable fields are strong.
10. Read for the 'best' or 'next' step. Several answers may be reasonable. Choose the one that fits CPMAI's order: understand the business, understand the data, prepare, model, evaluate, operationalize.

Quick Recap
Schema and structure compatibility means confirming early that your data's fields, types, grain, labels, relationships and formats can support the chosen AI pattern and model. Do it in Data Understanding, document the gaps, plan remediation for Data Preparation, and do not hesitate to iterate back if the data cannot support the business objective.