Feature Engineering and Feature Selection

Overview
Feature engineering and feature selection sit at the heart of the PMI-CPMAI domain Manage AI Model Development and Evaluation. In the Cognitive Project Management for AI (CPMAI) methodology, they belong mainly to Phase III (Data Preparation). Their effects are tested in Phase IV (Model Development) and Phase V (Model Evaluation).

For a project manager the goal is not to write the code. The goal is to understand why these activities matter, how they affect cost, schedule, risk and model quality, and how to govern them in an iterative AI project.

1. Why Feature Engineering and Feature Selection Are Important

Model quality depends on input quality. The saying "garbage in, garbage out" applies directly. Well-designed features often improve performance more than switching to a more complex algorithm.

Data work dominates effort. Data preparation, including feature work, is commonly cited as about 60 to 80 percent of AI project effort. PMs who underestimate it miss schedules and budgets.

Irrelevant or redundant features cause harm. Too many of them lead to:

• overfitting
• longer training times
• higher compute cost
• reduced explainability
• the curse of dimensionality

Governance and trust. Features determine what the model "sees." Using protected attributes (race, gender, age), or proxies for them such as zip code standing in for ethnicity, can introduce bias. This creates legal, ethical and reputational risk. Trustworthy AI requires deliberate feature decisions.

Explainability. Fewer, meaningful features make models easier to interpret, audit and justify to stakeholders and regulators.

Operational feasibility. A feature that is available in training but not at prediction time cannot be used in production. Feature choices must match real-world data pipelines.

Business alignment. Feature design draws on domain knowledge. Subject matter experts (SMEs) translate business understanding (CPMAI Phase I) into measurable signals.

2. What They Are

Feature. An individual measurable property or input variable used by a model. Examples include customer age, transaction amount and pixel intensity. In tabular data, features are the columns and the target (label) is what you predict.

Feature Engineering. The process of creating, transforming or extracting features from raw data so that patterns become more learnable. It answers: How do we represent the data so the model can learn the right patterns?

Feature Selection. The process of choosing the most relevant subset of available features and discarding irrelevant, redundant or harmful ones. It answers: Which inputs should the model actually use?

Feature Extraction (often grouped with engineering). Builds new, lower-dimensional features from the original set, for example with Principal Component Analysis (PCA) or embeddings. Note the difference:

• Selection keeps original features, which preserves interpretability.
• Extraction creates new combined features, which often reduces interpretability.

3. How Feature Engineering Works

Common techniques you should recognize:

• Handling missing values: imputation (mean, median, mode, model-based), adding indicator flags, or removing records.
• Encoding categorical variables: one-hot encoding, label/ordinal encoding, target encoding.
• Scaling and normalization: min-max scaling or standardization (z-score). These are important for distance-based and gradient-based algorithms.
• Binning/discretization: converting continuous values into ranges, such as age groups.
• Mathematical transformations: log transforms for skewed data, polynomial features, ratios such as debt-to-income.
• Date/time decomposition: extracting day of week, month, holiday flags, or time since last event.
• Aggregations: counts, sums or averages over windows, such as purchases in the last 30 days.
• Interaction features: combining two variables to capture joint effects.
• Text features: tokenization, bag-of-words, TF-IDF, word embeddings.
• Image/audio features: edges, color histograms, spectrograms. Deep learning often learns these automatically.
• Outlier treatment: capping, removal or transformation.

Deep learning note: Deep neural networks perform automatic feature learning (representation learning). This reduces manual engineering for unstructured data such as images, audio and text. Manual feature engineering remains very important for structured/tabular data and classical machine learning.

4. How Feature Selection Works

There are three main families of methods:

• Filter methods: rank features using statistical measures, independent of any model. Examples include correlation, chi-square, mutual information, variance threshold and ANOVA. They are fast and scalable but ignore feature interactions.
• Wrapper methods: train models on different subsets and evaluate performance. Examples include forward selection, backward elimination and Recursive Feature Elimination (RFE). They are more accurate but computationally expensive.
• Embedded methods: selection happens during model training. Examples include LASSO (L1 regularization) and tree-based feature importance (Random Forest, gradient boosting). They balance accuracy and cost.

Dimensionality reduction techniques such as PCA, t-SNE and autoencoders reduce feature counts by transformation rather than selection.

Other selection criteria a PM should insist on:

• Remove features that are unavailable at inference time.
• Remove features that cause data leakage.
• Remove or review features that are sensitive or legally restricted.
• Review proxies for protected attributes.
• Prefer features that are stable over time to reduce drift.
• Consider the cost of acquiring each feature.

5. Critical Concepts Frequently Tested

Data leakage. Information from outside the training context, or from the future, leaks into features. The result is unrealistically high evaluation scores that collapse in production. Examples:

• using "account closed date" to predict churn
• scaling with statistics computed on the full dataset before the train/test split

Prevention: split data first, then fit transformations only on training data. Validate the temporal logic of each feature.

Overfitting vs. underfitting. Too many features can cause overfitting. Too few, or poorly engineered, features cause underfitting. Feature selection and regularization reduce overfitting.

Curse of dimensionality. As the number of features grows, data becomes sparse and models need exponentially more data.

Training-serving skew. Transformations must be applied identically in training and production. Feature stores and reusable pipelines help achieve this.

Bias and fairness. Removing a protected attribute does not guarantee fairness, because proxies can remain. Fairness testing is still required.

Iteration. CPMAI is iterative. If Phase V evaluation shows weak performance, the team often loops back to Phase III to rework features, and sometimes back to Phases I/II to revisit the problem or data understanding.

Documentation and reproducibility. Feature definitions, transformations and lineage should be documented, for example in a data dictionary or feature catalog, and version-controlled for auditability.

6. The Project Manager's Role

• Plan sufficient time and budget for data preparation and feature iterations.
• Facilitate collaboration between data scientists, data engineers and domain SMEs. SMEs suggest meaningful features.
• Ensure data governance: privacy (GDPR, HIPAA), consent, and restrictions on sensitive attributes.
• Confirm that features are available, timely and affordable in production.
• Require leakage checks and correct train/validation/test separation.
• Track feature decisions in documentation for transparency and audit.
• Ensure features support explainability needs when stakeholders or regulators demand interpretable models.
• Support the use of feature stores and MLOps pipelines for consistency and reuse.
• Use go/no-go checkpoints. If features cannot achieve the required performance, re-scope rather than force deployment.

7. Worked Example

A bank wants to predict loan default. The data contains:

• income
• debt
• employment history
• zip code
• gender
• a field showing whether the loan was later sent to collections

The team handles it as follows:

• Engineering: creates debt-to-income ratio, months employed and number of late payments in the last 12 months.
• Leakage: removes "sent to collections," since it is a future outcome.
• Sensitive attributes: removes gender, which is legally restricted.
• Proxy review: evaluates zip code as a possible proxy for protected characteristics.
• Selection: uses feature importance to drop low-value columns, which improves interpretability for regulators.

Exam Tips: Answering Questions on Feature Engineering and Feature Selection

Tip 1: Know the definitions cold.

• Engineering = creating or transforming features.
• Selection = choosing a subset of existing features.
• Extraction = deriving new compressed features, such as PCA.

Distractors often swap these terms.

Tip 2: Map to the correct CPMAI phase. Feature work belongs primarily to Phase III (Data Preparation). If the question asks what to do when evaluation results are poor, revisiting data preparation and features is often correct, because CPMAI is iterative.

Tip 3: Suspiciously high accuracy means leakage. If a scenario describes excellent test results but poor production performance, or near-perfect accuracy early on, think data leakage or training-serving skew. Do not choose "deploy immediately."

Tip 4: Prefer data and feature improvements over algorithm complexity. When performance is weak, the best answer is often better features, more relevant data, or SME involvement. Jumping to a more complex model is usually not the best answer.

Tip 5: Watch for ethics and compliance. Answers that use protected or sensitive attributes without justification are usually wrong. Remember that proxies can still create bias. The correct answer usually includes bias testing and governance review.

Tip 6: Production availability matters. A feature that exists only in historical data, or arrives too late at inference time, should not be used. Pick the answer that validates operational feasibility.

Tip 7: Fewer features, more explainability. If stakeholders or regulators require interpretability, favor feature selection over extraction methods such as PCA, which obscure meaning.

Tip 8: Recognize method families.

• Correlation, chi-square, mutual information = filter.
• RFE, forward/backward selection = wrapper.
• LASSO, tree importance = embedded.

Tip 9: Choose collaborative answers. PMI exams reward engaging SMEs and data scientists and documenting decisions. Avoid answers where the PM unilaterally makes technical decisions or skips stakeholder input.

Tip 10: Apply transformations correctly. Fit scalers, encoders and imputers on training data only, then apply them to validation and test sets. Answers that preprocess the entire dataset before splitting are a trap.

Tip 11: Overfitting clues. High training accuracy with low validation accuracy points to overfitting. Remedies include:

• feature selection
• regularization
• more data
• simpler models
• cross-validation

Tip 12: Deep learning nuance. For images, audio and text, deep learning reduces manual feature engineering through representation learning. For tabular business data, manual feature engineering is still highly valuable.

Tip 13: Budget and schedule realism. If asked why an AI project is behind schedule, underestimated data preparation and feature work is a classic root cause. Good answers plan for iterations.

Tip 14: Eliminate extremes. Be wary of options with "always," "never," or "use all available features." The best answer usually balances performance, cost, explainability, ethics and feasibility.

Quick Recap
Feature engineering makes data learnable. Feature selection keeps only what helps. Both drive accuracy, efficiency, explainability and fairness. They are iterative, governed activities in CPMAI Phase III, validated in Phase V. In exam scenarios, look for:

• leakage
• bias and proxies
• production availability
• overfitting
• SME collaboration
• the iterative loop back to data preparation