Factor Analysis and Discriminant Analysis

Factor Analysis and Discriminant Analysis: A Complete Guide for Six Sigma Black Belt Analyze Phase

Introduction

Factor Analysis and Discriminant Analysis are two powerful statistical techniques used in the Six Sigma Black Belt Analyze Phase to understand complex data structures and relationships between variables. These methods help practitioners reduce dimensionality, identify underlying patterns, and classify observations into distinct groups.

Why Is This Important?

Understanding Factor Analysis and Discriminant Analysis is critical for several reasons:

• Data Simplification: Factor Analysis reduces hundreds of variables into a manageable set of underlying factors, making data more interpretable and easier to work with.
• Pattern Recognition: These techniques help identify hidden relationships and patterns in data that aren't immediately obvious.
• Classification and Prediction: Discriminant Analysis enables organizations to classify future observations into predefined groups accurately.
• Quality Improvement: By understanding which variables drive quality outcomes, teams can focus improvement efforts on the most impactful areas.
• Cost Reduction: Identifying redundant variables allows organizations to streamline data collection and monitoring processes.
• Risk Management: These techniques help identify and predict customer defaults, product defects, and process failures.

What Is Factor Analysis?

Definition

Factor Analysis is a multivariate statistical technique that identifies underlying latent variables (called factors) that explain the correlation structure among a larger set of observed variables. It assumes that observed variables are linear combinations of unobserved factors plus error.

Key Concepts

• Latent Variables (Factors): Unobservable underlying dimensions that influence observed variables.
• Loadings: Correlations between observed variables and factors, indicating the strength of relationships.
• Communality: The proportion of variance in a variable explained by the retained factors.
• Eigenvalue: The amount of variance explained by each factor.
• Variance Explained: The cumulative percentage of total variance captured by the extracted factors.

Types of Factor Analysis

1. Exploratory Factor Analysis (EFA):
Used when researchers have no prior knowledge about the factor structure. The goal is to discover underlying patterns in the data.

2. Confirmatory Factor Analysis (CFA):
Used when researchers have a theoretical hypothesis about the factor structure and want to test it against the data.

When to Use Factor Analysis

• You have many correlated variables and want to reduce them to fewer uncorrelated factors.
• You want to identify underlying constructs or themes in survey data.
• You need to improve model efficiency by reducing dimensionality.
• You want to validate the structure of a measurement instrument.

What Is Discriminant Analysis?

Definition

Discriminant Analysis is a multivariate statistical technique that determines which variables best discriminate between predefined groups and develops a classification rule to predict group membership for new observations. It's often called Linear Discriminant Analysis (LDA) when assumptions are met.

Key Concepts

• Discriminant Function: A linear combination of variables that maximizes the separation between groups.
• Discriminant Score: The numerical value computed for each observation using the discriminant function.
• Classification Rule: A decision rule that assigns observations to groups based on their discriminant scores.
• Wilks' Lambda: A test statistic that measures whether group means differ significantly.
• Classification Accuracy: The percentage of observations correctly classified into their true groups.
• Prior Probability: The assumed probability of group membership before analyzing the data.

Assumptions of Discriminant Analysis

• Variables are normally distributed within each group.
• Variance-covariance matrices are equal across groups (homogeneity of variance).
• Variables are measured on at least an interval scale.
• Groups are mutually exclusive and exhaustive.
• No multicollinearity among predictor variables.
• Adequate sample size (at least 20 observations per variable).

When to Use Discriminant Analysis

• You have a categorical outcome variable with predefined groups (2+ groups).
• You want to identify which variables best separate the groups.
• You need to classify new observations into one of the predefined groups.
• You want to understand group differences (e.g., defective vs. non-defective products).

How Factor Analysis Works

Step-by-Step Process

Step 1: Check Appropriateness
Verify that your data is suitable for Factor Analysis:

• Ensure variables are correlated (check correlation matrix).
• Use Kaiser-Meyer-Olkin (KMO) test: values > 0.6 suggest Factor Analysis is appropriate.
• Use Bartlett's Test of Sphericity: p-value < 0.05 indicates variables are correlated.

Step 2: Calculate Correlation Matrix
Compute the correlation matrix of all observed variables to identify relationships.

Step 3: Extract Factors
Choose an extraction method:

• Principal Component Analysis (PCA): Most common; explains maximum variance.
• Common Factor Analysis: Uses communalities; assumes factors cause observed variables.
• Maximum Likelihood: Uses statistical inference for factor estimation.

Step 4: Determine Number of Factors
Use these criteria:

• Kaiser Criterion: Retain factors with eigenvalues > 1.0.
• Scree Plot: Look for the "elbow" where variance explained drops sharply.
• Cumulative Variance: Retain factors explaining 70-80% of total variance.
• Practical Significance: Factors should be interpretable and useful.

Step 5: Rotate Factors
Apply rotation to improve interpretability:

• Orthogonal Rotation (Varimax, Quartimax): Creates uncorrelated factors; easier to interpret.
• Oblique Rotation (Promax): Allows correlated factors; more realistic for natural phenomena.

Step 6: Interpret Factor Loadings
Examine which variables load highly on each factor. Variables with loadings > 0.4-0.5 are typically considered significant.

Step 7: Calculate Factor Scores
Compute scores for each observation, representing their position on each factor.

Factor Analysis Example

Suppose a manufacturing company measures 15 quality characteristics in a process. Factor Analysis reveals that these 15 variables can be reduced to 4 underlying factors: Surface Quality, Dimensional Accuracy, Structural Integrity, and Chemical Composition. This allows the team to monitor fewer factors instead of 15 individual variables.

How Discriminant Analysis Works

Step-by-Step Process

Step 1: Check Assumptions
Verify that data meets discriminant analysis requirements:

• Check for normality in each group (Shapiro-Wilk test).
• Test homogeneity of variance (Levene's test): p > 0.05 is preferred.
• Check for multicollinearity using correlation matrix or VIF.

Step 2: Prepare Data
Divide data into:

• Training Set: 60-70% of data; used to develop the classification rule.
• Validation Set: 30-40% of data; used to test classification accuracy.

Step 3: Develop Discriminant Function
The discriminant function is a linear combination: D = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ
Where:

• D = discriminant score
• b = discriminant coefficients
• X = predictor variables

Step 4: Test Statistical Significance
Use Wilks' Lambda to determine if group means differ significantly. A p-value < 0.05 indicates the discriminant function is statistically significant.

Step 5: Establish Classification Rule
Calculate cutoff scores or decision boundaries. For two groups, the decision rule is typically:

• If D > cutoff score: Classify as Group 1
• If D ≤ cutoff score: Classify as Group 2

Step 6: Classify Observations
Apply the discriminant function to all observations and assign them to groups.

Step 7: Evaluate Classification Accuracy
Create a confusion matrix and calculate:

• Overall Accuracy: (TP + TN) / Total
• Sensitivity: TP / (TP + FN) - ability to correctly identify Group 1
• Specificity: TN / (TN + FP) - ability to correctly identify Group 2
• Misclassification Rate: 1 - Overall Accuracy

Step 8: Validate on Test Set
Apply the model to the validation set to ensure generalizability.

Discriminant Analysis Example

A credit card company wants to classify customers as either Low Risk or High Risk for default based on variables like credit score, income, age, and debt ratio. Discriminant Analysis develops a classification rule that predicts default risk for new customers with 85% accuracy.

Comparing Factor Analysis and Discriminant Analysis

Similarities:

• Both are multivariate techniques handling multiple variables simultaneously.
• Both assume linear relationships among variables.
• Both require meeting specific statistical assumptions.
• Both are used in the Analyze Phase of DMAIC.

Differences:

• Purpose: Factor Analysis reduces dimensionality; Discriminant Analysis classifies observations.
• Dependent Variable: Factor Analysis has no dependent variable; Discriminant Analysis requires a categorical dependent variable.
• Output: Factor Analysis produces factors and loadings; Discriminant Analysis produces classification rules and accuracy metrics.
• Use Case: Factor Analysis is exploratory; Discriminant Analysis is confirmatory.

Practical Applications in Six Sigma

Factor Analysis Applications

• Customer Satisfaction Surveys: Reduce 50+ survey items to 5-7 underlying satisfaction dimensions.
• Process Variables: Consolidate hundreds of machine parameters into key operational factors.
• Supply Chain: Identify core supplier performance factors from multiple quality metrics.
• Product Development: Group customer needs into primary design factors.

Discriminant Analysis Applications

• Defect Classification: Classify products as defective or non-defective based on measured characteristics.
• Process Control: Identify in-control vs. out-of-control process conditions.
• Supplier Selection: Classify suppliers as qualified or non-qualified.
• Customer Segmentation: Classify customers into high-value or low-value groups.
• Predictive Maintenance: Classify equipment as requiring maintenance or operating normally.

Exam Tips: Answering Questions on Factor Analysis and Discriminant Analysis

General Strategies

1. Understand the Research Question
Carefully read the question to determine which technique is appropriate:

• If the question asks about reducing variables, identifying patterns, or underlying factors, think Factor Analysis.
• If the question asks about classifying observations, predicting group membership, or comparing groups, think Discriminant Analysis.

2. Know the Key Terms
Be prepared to define and use these terms correctly:

• Factor Analysis: Loadings, communality, eigenvalue, KMO, Bartlett's test, rotation, scree plot
• Discriminant Analysis: Discriminant function, discriminant score, Wilks' Lambda, classification accuracy, confusion matrix, sensitivity, specificity

3. Recognize Assumptions
Be ready to discuss or identify violated assumptions:

• Factor Analysis: Variables should be correlated; KMO > 0.6; Bartlett's p < 0.05
• Discriminant Analysis: Normality, homogeneity of variance, no multicollinearity, predefined groups

Specific Exam Question Types

Type 1: Choosing the Appropriate Technique
Example Question: "A Black Belt wants to reduce 30 correlated quality metrics into fewer underlying factors for easier monitoring. Which technique should be used?"

Answer Strategy: Identify the goal (dimensionality reduction → Factor Analysis). Mention KMO and Bartlett's tests to verify appropriateness. Discuss extraction, rotation, and interpretation of factor loadings.

Type 2: Interpreting Statistical Output
Example Question: "A scree plot shows the first 3 factors with eigenvalues of 5.2, 2.8, and 1.5. The fourth factor has an eigenvalue of 0.9. How many factors should be retained?"

Answer Strategy: Use the Kaiser criterion (eigenvalue > 1.0): Retain 3 factors. Mention that the scree plot shows an elbow after the third factor. State that 3 factors likely explain 70-80% of variance, which is acceptable.

Type 3: Classification Accuracy
Example Question: "A discriminant analysis model correctly classified 85 out of 100 test observations. Of 50 defective products, it correctly identified 45. Calculate sensitivity and overall accuracy."

Answer Strategy:
Overall Accuracy = 85/100 = 85%
Sensitivity = 45/50 = 90% (true positive rate for defective products)
Explain that 90% sensitivity means the model correctly identifies defective products 90% of the time.

Type 4: Practical Application Scenarios
Example Question: "A manufacturing plant has 200 process variables. Describe how Factor Analysis can improve process monitoring efficiency."

Answer Strategy: Explain that Factor Analysis would:
1. Identify correlations among the 200 variables
2. Extract underlying factors (e.g., 8-10 factors)
3. Create factor scores for each observation
4. Allow monitoring of fewer factors instead of 200 variables
5. Reduce data collection costs and improve interpretability
Mention the benefit of focusing improvement efforts on key factors.

Type 5: Problem Identification
Example Question: "A team applied Factor Analysis but KMO = 0.35 and Bartlett's test p-value = 0.45. What does this suggest?"

Answer Strategy: Explain that:
1. KMO < 0.5 indicates factor analysis is not appropriate for this data
2. Bartlett's test p > 0.05 suggests variables are not significantly correlated
3. The team should reconsider the variable selection or data preparation
4. Variables may be too independent to extract meaningful factors

Common Pitfalls to Avoid

Pitfall 1: Confusing the Techniques
Remember: Factor Analysis = dimension reduction; Discriminant Analysis = classification

Pitfall 2: Ignoring Assumptions
Always check and discuss assumptions. Violated assumptions can invalidate results.

Pitfall 3: Over-Interpreting Results
Don't claim practical significance without sufficient variance explained or statistical significance.

Pitfall 4: Neglecting Validation
Always test models on independent validation sets. Training accuracy alone is insufficient.

Pitfall 5: Ignoring Factor Interpretability
Factors should be meaningful and actionable. A factor explaining high variance but not interpretable is not useful.

Key Formulas to Remember

Factor Analysis:

• Observed Variable = Factor Loadings × Factor Scores + Error
• Communality = Sum of squared loadings for a variable
• Variance Explained = Sum of eigenvalues / Total sum of squares

Discriminant Analysis:

• Discriminant Function: D = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ
• Overall Accuracy = (TP + TN) / (TP + TN + FP + FN)
• Sensitivity = TP / (TP + FN)
• Specificity = TN / (TN + FP)

Study Tips for Exam Success

• Create Concept Maps: Draw connections between variables, factors, and classifications.
• Practice Interpretation: Work through sample output tables and scree plots.
• Understand Applications: Link techniques to real manufacturing and business scenarios.
• Review Case Studies: Study examples from DMAIC projects in your industry.
• Master Terminology: Be fluent with statistical terms; examiners test vocabulary knowledge.
• Test Yourself: Complete practice questions under timed conditions.
• Know When to Use: Be able to quickly identify which technique suits a given scenario.

Conclusion

Factor Analysis and Discriminant Analysis are essential tools in the Six Sigma Black Belt Analyze Phase toolbox. Factor Analysis simplifies complex data by identifying underlying patterns, while Discriminant Analysis classifies observations into meaningful groups. Understanding when to use each technique, how they work, and how to interpret results is crucial for exam success and practical effectiveness. By mastering these techniques and following the exam tips provided, you'll be well-prepared to apply them successfully in improvement projects and answer related exam questions with confidence.