Questions from Week 12?

• Decision trees and ensemble methods?
• Random forests and feature importance?
• Model evaluation, comparison?
• Hyperparameter tuning?
• Anything confusing in the quiz or class lab?
• Time to ask!

Supervised vs. Unsupervised Learning

Everything we’ve done so far has been supervised learning:

Think-Pair-Share

Let’s think about when you might need unsupervised learning in business contexts.

Then we’ll take a few responses…

Two Main Types of Unsupervised Learning

graph TB
    A[Unsupervised Learning<br/>Discover patterns without labels] --> B[Clustering<br/>Group similar observations]
    A --> C[Dimension Reduction<br/>Reduce number of features]

    B --> B1[K-Means: Customer segmentation]
    B --> B2[Hierarchical: Organize documents]
    B --> B3[DBSCAN: Anomaly detection]

    C --> C1[PCA: Compress features]
    C --> C2[t-SNE: Visualize high-D data]
    C --> C3[Autoencoders: Deep learning compression]

    style A fill:#43464B,stroke:#fff,stroke-width:2px,color:#fff
    style B fill:#4ECDC4,stroke:#fff,stroke-width:2px
    style C fill:#FF6B6B,stroke:#fff,stroke-width:2px

What is Clustering?

Clustering automatically groups similar observations together without being told what the groups should be.

graph TD
    A[1000 Customers<br/>Many features] --> B{Clustering<br/>Algorithm}
    B --> C1[Segment 1:<br/>Budget Shoppers]
    B --> C2[Segment 2:<br/>Premium Buyers]
    B --> C3[Segment 3:<br/>Occasional Visitors]

    style A fill:#f0f0f0
    style B fill:#43464B,color:#fff
    style C1 fill:#FF6B6B
    style C2 fill:#4ECDC4
    style C3 fill:#45B7D1

Think-Pair-Share: Identify Clustering Problems

Which of these business problems would benefit from clustering?

Then we’ll take a few responses…

Hands-On Demo: K-Means in Action

Let’s see K-Means clustering with a simple customer dataset:

Demo: Fitting K-Means

# Standardize features (important for K-Means!)
scaler = StandardScaler()
customers_scaled = scaler.fit_transform(customers)

# Fit K-Means with K=3 clusters
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
customers['cluster'] = kmeans.fit_predict(customers_scaled)

The Big Question: How Many Clusters (K)?

Problem: K-Means requires you to specify the number of clusters upfront. But how do you know what K should be?

Demo: Finding the Right K

from sklearn.metrics import silhouette_score

# Try K from 2 to 8
K_range = range(2, 9)
inertias = []
silhouettes = []

for k in K_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    km.fit(customers_scaled)
    inertias.append(km.inertia_)
    silhouettes.append(silhouette_score(customers_scaled, km.labels_))

# Plot both metrics
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 6))

# Elbow plot
ax1.plot(K_range, inertias, 'bo-', linewidth=2, markersize=8)
ax1.set_xlabel('Number of Clusters (K)', fontweight='bold')
ax1.set_ylabel('Inertia (Within-cluster SS)', fontweight='bold')
ax1.set_title('Elbow Method', fontweight='bold')
ax1.grid(True, alpha=0.3)
ax1.axvline(x=3, color='red', linestyle='--', alpha=0.5, label='Elbow at K=3')
ax1.legend()

# Silhouette plot
ax2.plot(K_range, silhouettes, 'ro-', linewidth=2, markersize=8)
ax2.set_xlabel('Number of Clusters (K)', fontweight='bold')
ax2.set_ylabel('Silhouette Score', fontweight='bold')
ax2.set_title('Silhouette Score by K', fontweight='bold')
ax2.grid(True, alpha=0.3)
ax2.axvline(x=3, color='red', linestyle='--', alpha=0.5, label='Peak at K=3')
ax2.legend()

plt.tight_layout()
plt.show()

Interactive Activity: Interpret Elbow Plots

Then we’ll take a few responses…

Reality Check: Real Data is Messy

The examples so far have been clean and simple…

But real business data rarely has such clear clusters!

Let’s look at a real-world example: grocery store customer segmentation with:

• 801 actual households
• Transaction history over 1 year
• 14 features combining:
  • Behavioral: spending, visit frequency, discount usage, recency
  • Demographic: age, income, household size, marital status, homeownership

What Do You Notice?

Example: Choosing K for This Dataset

Let’s walk through the decision process:

The Problem: Too Many Features

Modern datasets are getting wider and wider—hundreds or thousands of features:

Principal Component Analysis (PCA): Visual Overview

How PCA transforms many correlated features into fewer uncorrelated components:

graph TD
    A[50 Original Features<br/>All correlated] --> B[PCA Transformation]
    B --> PC1[PC1: 45% variance<br/>Overall Size]
    B --> PC2[PC2: 20% variance<br/>Location Quality]
    B --> PC3[PC3: 10% variance<br/>Age/Condition]
    B --> PC4[PC4: 8% variance<br/>Luxury Features]
    B --> PC5[PC5: 5% variance<br/>...]
    B --> Rest[PC6-PC50: 12% variance<br/>Mostly noise]

    style PC1 fill:#FF6B6B
    style PC2 fill:#4ECDC4
    style PC3 fill:#45B7D1
    style Rest fill:#f0f0f0

PCA: The Intuition

PCA finds new coordinate axes (principal components) that capture maximum variance:

The idea:

1. PC1 points in the direction where data varies the MOST
2. PC2 points in the next-most variation direction (perpendicular to PC1)
3. PC3 points in the next direction (perpendicular to both)
4. And so on…

The Apartment-Finding Analogy

Imagine tracking 10 apartment features: rent, sq ft, distance from downtown, bedrooms, building age, floor number, etc.

You might notice correlations:

• Larger apartments → more bedrooms, more bathrooms
• Newer buildings → farther from downtown, higher rent
• Higher floors → better views, cost more

PCA would identify underlying factors:

Result: Instead of tracking 10 correlated features, you have 3 uncorrelated components capturing 90% of information!

Step 1: The Original Data

Let’s apply PCA to a medical dataset with 569 patients and 30 features measuring tumor characteristics:

Step 3: How Many Components Should We Keep?

# Create scree plot
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 6))

# Individual variance explained
variance_explained = pca_full.explained_variance_ratio_
ax1.plot(range(1, len(variance_explained) + 1),
         variance_explained, 'bo-', linewidth=2, markersize=8)
ax1.set_xlabel('Principal Component', fontweight='bold', fontsize=11)
ax1.set_ylabel('Variance Explained', fontweight='bold', fontsize=11)
ax1.set_title('Scree Plot: Variance per Component', fontweight='bold', fontsize=12)
ax1.grid(True, alpha=0.3)
ax1.axvline(x=7, color='red', linestyle='--', alpha=0.7, label='Elbow around PC7')
ax1.legend()

# Cumulative variance explained
cumsum = np.cumsum(variance_explained)
ax2.plot(range(1, len(cumsum) + 1), cumsum, 'ro-', linewidth=2, markersize=8)
ax2.axhline(y=0.95, color='green', linestyle='--', linewidth=2, label='95% threshold')
ax2.axhline(y=0.90, color='orange', linestyle='--', linewidth=2, label='90% threshold')
ax2.set_xlabel('Number of Components', fontweight='bold', fontsize=11)
ax2.set_ylabel('Cumulative Variance', fontweight='bold', fontsize=11)
ax2.set_title('Cumulative Variance Explained', fontweight='bold', fontsize=12)
ax2.legend()
ax2.grid(True, alpha=0.3)

# Find components needed for 95%
n_95 = np.argmax(cumsum >= 0.95) + 1
ax2.axvline(x=n_95, color='green', linestyle=':', alpha=0.7)
ax2.text(n_95 + 1, 0.5, f'{n_95} components\nfor 95% variance',
         fontsize=10, bbox=dict(boxstyle='round', facecolor='lightgreen', alpha=0.7))

plt.tight_layout()
plt.show()

Interactive Activity: Reading Scree Plots

Then we’ll take a few responses…

Demo: Visualizing High-Dimensional Data with PCA

One powerful use of PCA: reducing to 2D for visualization!

Reduced from 30 features to 2 components
Retained 63.2% of variance

PCA as Feature Engineering for Machine Learning

Another powerful use case: Use PCA-transformed features as input to ML models

Why use PCA before modeling?

• Reduces overfitting - Fewer features = simpler model = better generalization
• Speeds up training - Fewer features = faster computation
• Handles multicollinearity - PCs are uncorrelated by design
• Removes noise - Minor components often represent noise

Let’s compare model performance with and without PCA using our breast cancer data:

Using Clustering/PCA in ML Pipelines

Unsupervised methods often feed into supervised models:

graph LR
    A[Raw Data<br/>100 features] --> B[StandardScaler<br/>Standardize]
    B --> C{PCA<br/>Reduce to 20 components}
    C --> D[K-Means<br/>Create 5 clusters]
    D --> E[Add cluster labels<br/>as new feature]
    E --> F[Supervised Model<br/>Random Forest<br/>Predict churn]

    style A fill:#f0f0f0
    style B fill:#FFE5B4
    style C fill:#FF6B6B
    style D fill:#4ECDC4
    style E fill:#45B7D1
    style F fill:#95E1D3

Example workflow:

1. Reduce 100 features to 20 components with PCA (noise reduction)
2. Cluster customers into 5 segments using K-Means on those 20 components
3. Add cluster ID as a categorical feature
4. Train a Random Forest to predict customer churn using PCA components + cluster ID

Benefit: Unsupervised learning creates better features for supervised learning!

Key Takeaways

• Unsupervised learning discovers patterns without labels – Use it when you want to explore data structure rather than predict outcomes

• Clustering groups similar observations – K-Means is the workhorse for customer segmentation, but choosing K requires judgment (elbow method, silhouette scores, business constraints)

• PCA reduces dimensionality by creating uncorrelated components – Compress 100 features to 10 components while retaining 90%+ of information. Use scree plots to choose how many components to keep.

• ALWAYS standardize features first – Both K-Means and PCA are highly sensitive to feature scale. Forget this and your results will be dominated by large-scale features!

• Unsupervised → Supervised pipeline – Often you’ll use clustering/PCA as preprocessing steps to create better features for supervised learning models

Connection to Thursday’s Lab

Looking Ahead

Btwn now & Thursday: Complete reading Chapters 28-30

Thursday Lab: Hands-on unsupervised learning exercises

Moving Foward: Course wrap-up & Final Project

Any Final Questions?

• About clustering and K-Means?
• About PCA and dimension reduction?
• About when to use which method?
• About Thursday’s lab?
• About the final project?