BANA 4080: Introduction to Data Mining with Python

Week 12: The Professional ML Workflow

Welcome to Week 12

Quick overview of today’s plan:
- Why the simple train/test split isn’t enough
- How cross-validation solves the “peeking problem”
- Finding optimal model settings systematically
- Turning raw data into powerful features

Important

This week is all about trying to optimize model performance!

Discussion: Homework & Questions

Questions from Week 11?

Random Forests and ensemble methods?
Feature importance interpretation?
When to use trees vs. other models?
Anything confusing in the quiz or class lab?
Time to ask!

Activity

Converse with your neighbor and identify…

1 new thing you learned last week that was clear and well explained
1 thing we covered last week that is still confusing

The Problem: We’ve Been Peeking

The Test Set Contamination Issue

Remember the golden rule from Module 8?

“Don’t touch the test set until you’ve selected your final model”

But then we did this:

Tried different max_depth values → evaluated on test set
Compared models → chose based on test set performance
Tuned hyperparameters → peeked at test set each time
Added/removed features → checked test set results

The consequence

Test scores become optimistically biased and untrustworthy.

This slide addresses the “uncomfortable truth” from Chapter 28. Students have been unknowingly contaminating their test sets throughout Chapters 25-27.

Important framing: This isn’t their fault! The simple train/test split was the right pedagogical foundation. Now they’re ready for the professional approach.

Key analogy from the textbook: It’s like studying with a practice test, adjusting your studying based on the practice test, taking it again and again, and then having that same test be your final exam. Your score would be artificially inflated.

Real-world impact: - Report test accuracy of 91% to stakeholders - Deploy to production - Actual performance: 85% - That 6-point drop erodes trust

Transition: “So how do we make decisions about models and tune hyperparameters WITHOUT contaminating our test set? That’s where cross-validation comes in.”

Solution: Cross-Validation

Wrong Way vs. Right Way

❌ What we’ve been doing:

flowchart TD
    A[Full Dataset] --> B[Train 80%]
    A --> C[Test 20%]
    B --> D[Train Model 1]
    D --> E[Evaluate on Test]
    C --> E
    E --> F{Good?}
    F -->|No| G[Try Model 2]
    G --> H[Evaluate on Test]
    C --> H
    H --> I{Good?}
    I -->|No| J[Try Model 3]
    J --> K[Evaluate on Test]
    C --> K
    K -->|Yes| L[Report Score]

    style C fill:#ff6b6b
    style E fill:#ff6b6b
    style H fill:#ff6b6b
    style K fill:#ff6b6b

Problem: Multiple peeks contaminate test set!

✓ What we should do:

flowchart TD
    A[Full Dataset] --> B[Train 80%]
    A --> C[Test 20%<br/>LOCKED]
    B --> D[5-Fold CV<br/>on Training]
    D --> E[Try Models]
    D --> F[Tune Params]
    D --> G[Engineer Features]
    E & F & G --> H[Select Best]
    H --> I[Retrain on<br/>Full Training]
    I --> J[Test ONCE]
    C --> J

    style C fill:#51cf66
    style D fill:#51cf66
    style J fill:#ffd43b

Solution: Use CV, keep test pristine!

This slide makes the contrast crystal clear.

Left side (Wrong Way): - Shows the peeking problem visually - Train model → evaluate on test → not good? → try again - Each arrow to “Evaluate on Test” is another peek - The test set (red) is being used repeatedly - This is what we’ve been doing in Chapters 25-27

Right side (Right Way): - Test set is locked away (green - safe) - All experimentation happens with CV on training set - Try different models, tune parameters, engineer features - ALL using cross-validation - Only after ALL decisions are made → test once

Teaching points:

Count the peeks:
- Wrong way: 3 peeks shown (could be many more!)
- Right way: 0 peeks until the very end
Where decisions are made:
- Wrong way: Decisions based on test set
- Right way: Decisions based on CV scores
Test set purpose:
- Wrong way: Used as validation set (contaminated)
- Right way: Used only for final honest evaluation

Key message: “The difference between these approaches is the difference between amateur and professional data science.”

Common student reaction: “Wait, we’ve been doing it wrong this whole time?” Answer: “Not ‘wrong’ - it was the right pedagogical sequence. Simple first, then proper. Now you know both!”

Transition: “Let’s zoom in on the right side - how does cross-validation actually work in practice?”

How K-Fold Cross-Validation Works

The idea:

Split training data into K equal parts (folds)
Rotate which fold is used for validation
Average results across all K iterations

Benefits:

Test set stays completely untouched
Every training point gets validated once
More reliable performance estimates
Can make unlimited decisions without peeking

This is the key slide - take time here.

Walk through the diagram: 1. First, split data into train (80%) and test (20%) 2. Lock the test set away - don’t touch it! 3. Take the training set and split into K folds (5 shown here) 4. In each iteration, use 1 fold for validation, rest for training 5. After 5 iterations, every point has been validated exactly once 6. Average the 5 validation scores

Important clarifications: - K=5 is common, but K=10 is also popular - Each data point is in exactly ONE fold - Each point gets validated ONCE (when its fold is the validation set) - The model is retrained K times (5 times here)

Why this solves the peeking problem: - We’re only using training data for all decisions - Test set remains pristine - We get K different validation scores, so we see variance/stability

Common student question: “Isn’t this expensive computationally?” Answer: “Yes - if K=5, you train 5 models instead of 1. But it’s worth it for honest performance estimates. And computers are fast!”

Transition: “Now let’s see how this works in code with sklearn.”

Cross-Validation in Scikit-Learn

from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

# Step 1: Split data (test set locked away)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 2: Use CV to evaluate on TRAINING SET ONLY
dt = DecisionTreeClassifier(max_depth=5)
cv_scores = cross_val_score(dt, X_train, y_train, cv=5, scoring='accuracy')

print(f"CV Scores: {cv_scores}")
print(f"Mean CV Accuracy: {cv_scores.mean():.3f} (+/- {cv_scores.std():.3f})")

Output:

CV Scores: [0.847, 0.853, 0.862, 0.841, 0.858]
Mean CV Accuracy: 0.852 (+/- 0.008)

Notice: We never touched X_test or y_test!

Walk through the code step by step:

Line 1-3: Import what we need
Line 5-6: First, split into train/test. Test set is now locked away.
Line 8-10: Create model and run 5-fold CV on TRAINING data only
Line 12-13: Print results

Key points to emphasize:

The cv_scores array: - Contains 5 numbers (one per fold) - Shows the validation accuracy for each iteration - Notice they’re similar but not identical (0.841 to 0.862) - This variance tells us about model stability

The mean and standard deviation: - Mean (0.852) is our best estimate of true performance - Std (0.008) tells us how consistent the model is - Low std = stable, reliable model - High std = might be sensitive to which data it sees

What we achieved: - Made a decision (max_depth=5) without peeking at test set - Got 5 different validation checks instead of 1 - Still have our pristine test set for final evaluation

Common student question: “When do we use the test set?” Answer: “Only ONCE at the very end after ALL decisions are made. We’ll see the full workflow in a moment.”

Transition: “Now that we understand CV, let’s see how to use it for hyperparameter tuning.”

Cross-Validation: Key Takeaway

The Golden Rule of Cross-Validation

Split your data FIRST, then use cross-validation on the training set for ALL modeling decisions.

✅ Compare different models
✅ Tune hyperparameters
✅ Select features
✅ Evaluate preprocessing choices

Your test set should be touched ONCE at the very end.

Why this matters:

Test set contamination is subtle and easy to do accidentally
Every peek at the test set makes your performance estimates less trustworthy
Cross-validation gives you unlimited practice exams while keeping your final exam pristine
This is the difference between amateur and professional data science

This slide crystallizes the main lesson from the CV section.

The callout box contains the core message: - Split first (train/test split is step 1) - Use CV on training set for all decisions - Test set = final evaluation only

Emphasize the examples: - “Want to compare Random Forest vs. Gradient Boosting? Use CV!” - “Want to find best max_depth? Use CV!” - “Want to see if feature scaling helps? Use CV!” - “Want to try different imputation strategies? Use CV!”

All without touching the test set!

Teaching moment: “The test set is like your final exam. You only get to take it once. Everything else - all your studying, all your practice tests, all your preparation - that happens with cross-validation on the training set.”

Common student mistake to reinforce: ❌ “I’ll try a few models on the test set and pick the best one” ✓ “I’ll try models using CV on training set, pick the best, THEN test once”

Real-world context: “In production, you don’t get multiple chances. You deploy a model and see how it performs. That’s why your test set evaluation needs to be honest - it’s your only preview of production performance.”

Transition: “Now that we know how to evaluate honestly with CV, let’s use it to find the best hyperparameters for our models.”

Hyperparameter Tuning

The Bias-Variance Tradeoff

Every model makes two types of errors:

High Bias (Underfitting)

Model too simple
Misses patterns in data
Consistent errors
Poor on training AND validation

High Variance (Overfitting)

Model too complex
Memorizes noise
Sensitive to specific training data
Great on training, poor on validation

The sweet spot: Balance bias and variance for best generalization.

Hyperparameters control this tradeoff: max_depth, n_estimators, K, etc.

This is a fundamental ML concept - make it concrete.

Bias-variance framing: Use the student analogy from Chapter 29:

High Bias Student: - Only memorized basic formulas - Doesn’t understand concepts - Gets problems consistently wrong - Too simplistic approach

High Variance Student: - Memorized every example problem - Including specific numbers and edge cases - Great on familiar problems - Struggles with anything slightly different - Memorized patterns, not principles

In ML terms:

High Bias (Underfitting): - Linear regression trying to fit non-linear data - K=50 in KNN (averages 50 neighbors - very smooth, misses detail) - Decision tree with max_depth=1 (just one split) - Symptoms: Low training score, low validation score

High Variance (Overfitting): - K=1 in KNN (just uses nearest point - memorizes training data) - Decision tree with max_depth=None (grows until perfect fit) - Polynomial regression with degree=20 - Symptoms: High training score, much lower validation score

The goal: Find the hyperparameter value that minimizes the GAP between training and validation performance while maximizing validation performance.

Visual aid: Draw a U-shaped curve on the board: - X-axis: Model complexity (simple → complex) - Y-axis: Error - Two curves: Training error (decreases) and Validation error (U-shaped) - Sweet spot: Bottom of validation curve

Transition: “Let me show you what this looks like visually with a real example.”

Visualizing Bias-Variance with KNN

Caution

More advanced models (i.e. Decision Trees, Random Forests, Gradient Boosted Machines) are more flexible to align with data patterns; however, it is up to us to tune these models to balance the bias-variance tradeoff!

This visualization makes the bias-variance tradeoff concrete.

Walk through each panel:

Left: High Bias (K=75) - Model averages 75 neighbors - very smooth - Completely misses the curved pattern - Too simple - underfits the data - Red line is nearly straight - Consistent errors (always wrong in the same way)

Middle: Balanced (K=5) - Model averages 5 neighbors - just right - Captures the general curve without memorizing noise - Green line follows the pattern nicely - This is what we want!

Right: High Variance (K=1) - Model only uses 1 neighbor - exact match - Blue line zigzags through every training point - Memorizes noise instead of learning the pattern - Will perform poorly on new data

Key teaching points:

Hyperparameters control complexity:
- Large K = simpler model (high bias)
- Small K = complex model (high variance)
- Need to find the sweet spot
Training vs. validation performance:
- K=1: Perfect on training (100% accuracy), poor on validation
- K=75: Poor on training, poor on validation
- K=5: Good on both - balanced
Generalization is the goal:
- We don’t want to memorize training data
- We want to learn patterns that work on new data
- The balanced model (K=5) will generalize best

Student engagement: Ask: “Which model would you deploy to production?” Expected answer: “The balanced one (K=5)” Follow-up: “How do we find the right K value systematically? GridSearchCV!”

Transition: “So we can see the problem. But how have we been finding these values so far?”

The Manual Approach (What We’ve Been Doing)

In previous chapters, we manually tried different values:

# Try max_depth = 5
dt1 = DecisionTreeClassifier(max_depth=5)
score1 = cross_val_score(dt1, X_train, y_train)
print(f"max_depth=5: {score1.mean()}")

# Try max_depth = 10
dt2 = DecisionTreeClassifier(max_depth=10)
score2 = cross_val_score(dt2, X_train, y_train)
print(f"max_depth=10: {score2.mean()}")

# Try max_depth = 15
dt3 = DecisionTreeClassifier(max_depth=15)
score3 = cross_val_score(dt3, X_train, y_train)
print(f"max_depth=15: {score3.mean()}")

# ... keep trying values manually

Problems with this approach:

⏱️ Time-consuming - Repetitive code for each value
🐛 Error-prone - Easy to make copy-paste mistakes
📊 Limited exploration - Only try a few values
🔄 Not systematic - What about combinations of parameters?
📝 Hard to track - Which combination was best?

A Better Way Exists!

We need an automated, systematic approach to search through hyperparameter combinations efficiently.

This slide bridges previous chapters to GridSearchCV.

Left side - What students have been doing:

Walk through the code: - “Remember doing this in Chapters 25-27?” - “You tried max_depth=5, then 10, then 15…” - “Lots of copying and pasting, changing one number each time” - “Tedious and error-prone!”

Point out the pattern: - Same code structure repeated 3 times - Only difference is the max_depth value - What if you want to try 10 values? 20 values? - What if you want to tune multiple parameters simultaneously?

Right side - Problems:

Go through each problem:

Time-consuming:
- “You spend more time writing repetitive code than thinking about the problem”
- “What if you need to try 50 different combinations?”
Error-prone:
- “Easy to copy-paste and forget to change the variable name”
- “Or forget to change the max_depth value”
- “I’ve done this myself - very frustrating!”
Limited exploration:
- “Realistically, you only try 3-5 values manually”
- “But the optimal value might be max_depth=12, not 5, 10, or 15!”
- “You’re only sampling a small part of the space”
Not systematic:
- “What if you want to tune BOTH max_depth AND n_estimators?”
- “That’s 3 × 3 = 9 combinations to try manually”
- “Add min_samples_split? Now it’s 3 × 3 × 3 = 27 combinations!”
- “This gets out of hand quickly”
Hard to track:
- “Which combination gave the best score?”
- “Did max_depth=10 with n_estimators=100 do better than max_depth=15 with n_estimators=200?”
- “You need to keep notes or save results somehow”

The callout box: - “There’s a better way - let the computer do the tedious work!” - “That’s what GridSearchCV does”

Student engagement: Ask: “How many of you have done something like this and thought ‘there must be a better way’?” Show of hands usually gets good response.

Transition: “Let me show you GridSearchCV, which automates all of this and makes hyperparameter tuning systematic, reproducible, and efficient.”

What is Grid Search?

Grid Search = Systematically trying ALL combinations of hyperparameters

🔍 The Grid Search Process

For each combination: Train model → Evaluate with cross-validation → Record score

After trying all combinations: Pick the one with the best CV score!

This slide introduces the grid search concept visually.

Key teaching points:

The visual grid: - Each blue box represents ONE complete hyperparameter configuration - In this example: 4 max_depth values × 3 n_estimators values = 12 total combinations - Grid search will try EVERY single one of these

The systematic approach: - Unlike our manual approach where we tried a few values - Grid search is exhaustive - it tries ALL combinations - Nothing gets missed

The name “Grid Search”: - It literally comes from this grid visualization - We’re searching through a grid of parameter combinations - In higher dimensions (3+ parameters), it’s harder to visualize but same concept

The process: 1. Define the grid (which parameters and which values to try) 2. For each cell in the grid: - Train a model with those parameter values - Evaluate it using cross-validation - Record the CV score 3. After trying everything, select the combination with the best score

Example walkthrough: “Look at this grid. We have max_depth on one axis and n_estimators on the other. Each blue box is a unique combination. For example, the top-left box is max_depth=5 and n_estimators=100. Grid search will train and evaluate a model for EACH of these 12 boxes.”

Why this matters: - Removes human bias (we might not try certain combinations) - Systematic and reproducible - Can find surprising optimal combinations - But can be computationally expensive (more parameters = more combinations)

The ‘curse of dimensionality’: - 2 parameters with 4 values each = 4 × 4 = 16 combinations - 3 parameters with 4 values each = 4 × 4 × 4 = 64 combinations - 4 parameters with 4 values each = 4 × 4 × 4 × 4 = 256 combinations - This is why RandomizedSearchCV exists (for very large grids)

Engagement question: “If we added a third parameter, min_samples_split, with 3 possible values, how many total combinations would we have? … That’s right, 12 × 3 = 36 combinations!”

Transition: “Now that we understand WHAT grid search does conceptually, let’s see HOW to do it in scikit-learn with GridSearchCV.”

GridSearchCV: Automated Hyperparameter Search

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# Define parameter grid to search
param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [5, 10, 15, 20],
    'min_samples_split': [2, 5, 10]
}

# Create GridSearchCV (uses cross-validation internally!)
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1  # Use all CPU cores
)

# Fit on training data (will try 3 × 4 × 3 = 36 configs, each with 5-fold CV = 180 models!)
grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")

This is where the magic happens - systematic hyperparameter search.

Walk through the code:

Lines 4-9: Define search space - We’re trying 3 values for n_estimators - 4 values for max_depth - 3 values for min_samples_split - Total combinations: 3 × 4 × 3 = 36 different configurations

Lines 12-18: Create GridSearchCV - Wraps a RandomForestClassifier - Will try all 36 configurations - Each with 5-fold CV - Uses accuracy as the metric - n_jobs=-1 means “use all CPU cores” (runs in parallel)

Line 21: The computational reality - 36 configurations × 5 folds = 180 models trained! - Sounds expensive, but computers are fast - Usually takes seconds to minutes, not hours - Worth it for finding optimal settings

Lines 23-24: Results - best_params_: The winning hyperparameter combination - best_score_: The CV accuracy of the best configuration

Key insight: GridSearchCV uses CV internally, so we NEVER touch the test set during this entire search!

Common student questions:

“Do I have to try all combinations?”
- GridSearchCV: Yes (exhaustive search)
- RandomizedSearchCV: No (random sampling, faster for large grids)
- Bayesian optimization: Smart search based on previous results
“How do I know which parameters to tune?”
- Read documentation for your model
- Most important usually: complexity parameters (max_depth, n_estimators, etc.)
- Less important: random_state, n_jobs (these don’t affect performance)
“What ranges should I try?”
- Start broad: [10, 100, 1000]
- Then narrow in: [80, 90, 100, 110, 120]
- Trade-off between thoroughness and computation time

Transition: “So we can now find optimal hyperparameters. But we still need to prepare our data properly first. That’s feature engineering.”

Think-Pair-Share: Spotting Overfitting

Scenario: You’ve tuned a gradient boosting model and see these results:

Configuration A:
  Training Accuracy: 92.3%
  CV Accuracy: 88.7%

Configuration B:
  Training Accuracy: 99.8%
  CV Accuracy: 85.1%

Discuss:

Which configuration is overfitting more?
Which would you deploy to production?
What does the gap between training and CV tell you?

After timer, facilitate discussion:

Question 1: Which is overfitting more? - Configuration B is clearly overfitting - 99.8% training but only 85.1% CV = massive gap (14.7 points!) - It memorized the training data - Configuration A has smaller gap (3.6 points)

Question 2: Which to deploy? - Configuration A for sure - Higher CV score (88.7% vs 85.1%) - Smaller gap suggests better generalization - CV score is what matters for production performance

Question 3: What does the gap tell you? - Small gap (Config A): Model generalizes well - Large gap (Config B): Model memorizes training data, won’t generalize - The gap is a direct measure of overfitting severity

Teaching moment: Students often think “higher training score = better model”

Key insight: High training score is easy (just memorize!). The hard part is generalizing to new data. That’s why we care about CV score and the train/CV gap.

Real-world parallel: - Config B = student who memorized practice test (99.8%) but bombs actual exam (85.1%) - Config A = student who understood concepts (92.3%) and does well on actual exam (88.7%)

Transition: “Now let’s talk about preparing data properly - feature engineering.”

Feature Engineering

What is Feature Engineering?

The process of creating, transforming, and selecting features to help ML models learn better.

Raw data:

YearBuilt: 1995
YearRemodel: 2015

Engineered feature:

RelativeAge = 2025 - max(YearBuilt, YearRemodel)
            = 2025 - 2015 = 10 years

An older home (1995) with recent remodel (2015) has a younger relative age than a newer home (2010) without remodeling!

Raw features:

GrLivArea: 2000 sq ft
OverallQual: 8/10

Engineered interaction:

Size_x_Quality: 16,000

Captures combined effect!

Why it matters

Good features often matter more than fancy algorithms!

Feature engineering is where data science becomes an art.

The central message: > “Garbage in, garbage out. Feature engineering is how we turn raw data into high-quality input that models need.”

Key concepts to emphasize:

1. Raw data often doesn’t directly represent what we want to learn

Example: YearBuilt - Raw: 1995 - What matters: How OLD is the house? (Age = current year - year built) - Models learn better from age than year built

2. Domain knowledge is critical

Real estate examples: - Total bathrooms = FullBath + 0.5×HalfBath + BsmtFullBath + 0.5×BsmtHalfBath - Square feet per bathroom = GrLivArea / Total bathrooms - Was renovated? = (YearRemodAdd > YearBuilt)

These features come from understanding what makes houses valuable, not from algorithms.

3. The 80/20 rule in data science: Data scientists spend: - 80% of time on data prep and feature engineering - 20% on model selection and tuning

This isn’t inefficiency - it’s because good features make a BIGGER difference than fancy algorithms!

Show impact with example: - Simple linear regression + good features > Neural network + raw features - Feature engineering is the high-leverage activity

Types of feature engineering (preview): 1. Encoding categorical variables 2. Scaling numerical features 3. Creating new features (like examples shown) 4. Handling missing data 5. Building pipelines to do this systematically

Transition: “Let’s see the most common feature engineering techniques in action.”

Four Essential Techniques

1. Encoding Categorical Variables

Dummy encoding: One column per category
Label encoding: Single numerical column
Ordinal: Preserve ordering

2. Scaling Numerical Features

StandardScaler: Mean=0, Std=1
MinMaxScaler: Range [0, 1]
Crucial for distance-based algorithms

3. Creating New Features

Polynomial: Square, cube terms
Interactions: Feature × Feature
Domain-specific: Age, ratios, etc.

4. Handling Missing Data

Imputation: Fill with median/mode
Missingness indicators
Know when to drop vs. impute

Brief overview of the four main categories from Chapter 30.

1. Encoding Categorical Variables

The problem: ML algorithms need numbers, not text

Solutions: - Dummy/One-hot: Create binary columns (0/1) for each category - Example: BldgType → BldgType_SingleFamily, BldgType_Condo, etc. - Use for: Nominal variables (no order) - Watch out: Drop one column for linear regression (dummy variable trap)

Label encoding: Assign integers (1, 2, 3, …)
- Example: Neighborhoods → 1, 2, 3, …, 28
- Use for: High-cardinality + tree models
- Never use: With linear models for nominal variables!
Ordinal encoding: Custom mapping preserving order
- Example: Quality (Poor=1, Fair=2, Good=3, Excellent=4)
- Use for: Variables with natural ordering

2. Scaling Numerical Features

The problem: Features on different scales (sq ft: 1000-4000, bedrooms: 1-6)

When it matters: - k-NN, SVM: YES (distance-based) - Linear regression with regularization: YES - Decision trees, Random Forests: NO (threshold-based)

Methods: - StandardScaler: Transforms to mean=0, std=1 (most common) - MinMaxScaler: Transforms to range [0, 1]

3. Creating New Features

The power move in feature engineering:

Polynomial features: Capture non-linear relationships (x², x³)
Interactions: Feature A × Feature B (size × quality)
Domain-specific: Requires expertise in the problem domain
- Best source: Talk to domain experts!
- They know what really matters

4. Handling Missing Data

The reality: Real-world data always has gaps

Strategies: - Imputation: Fill with median (numerical) or mode (categorical) - Missingness indicators: Create binary flag “was_missing” - Important when missingness is informative! - Example: High earners often don’t report income - Drop data: Only when < 5% missing and it’s random

Key decision: Drop vs. Impute? - < 5% missing + random: Drop - > 5% missing or informative: Impute + indicator

Transition: “Let’s see some of these techniques in action with the Ames data.”

Hands-On Demo: Encoding & Scaling

Original Features:

import pandas as pd
from sklearn.preprocessing import LabelEncoder, StandardScaler

# Load Ames data
ames = pd.read_csv('../data/ames_clean.csv')

# Look at original features
features_to_eng = ['Neighborhood', 'GrLivArea', 'YearBuilt', 'TotalBsmtSF']
print(ames[features_to_eng])

     Neighborhood  GrLivArea  YearBuilt  TotalBsmtSF
0         CollgCr       1710       2003          856
1         Veenker       1262       1976         1262
2         CollgCr       1786       2001          920
3         Crawfor       1717       1915          756
4         NoRidge       2198       2000         1145
...           ...        ...        ...          ...
1455      Gilbert       1647       1999          953
1456       NWAmes       2073       1978         1542
1457      Crawfor       2340       1941         1152
1458        NAmes       1078       1950         1078
1459      Edwards       1256       1965         1256

[1460 rows x 4 columns]

Engineered Features:

# 1. Encode Neighborhood: 28 categories → integers
le = LabelEncoder()
ames['Neighborhood_Encoded'] = le.fit_transform(ames['Neighborhood'])

# 2. Scale numerical features: mean=0, std=1
scaler = StandardScaler()
features_to_scale = ['Neighborhood_Encoded', 'GrLivArea', 'YearBuilt', 'TotalBsmtSF']
ames[features_to_scale] = scaler.fit_transform(ames[features_to_scale])

# Look at engineered features
print(ames[features_to_scale])

      Neighborhood_Encoded  GrLivArea  YearBuilt  TotalBsmtSF
0                -1.206215   0.370333   1.050994    -0.459303
1                 1.954302  -0.482512   0.156734     0.466465
2                -1.206215   0.515013   0.984752    -0.313369
3                -1.039872   0.383659  -1.863632    -0.687324
4                 0.457215   1.299326   0.951632     0.199680
...                    ...        ...        ...          ...
1455             -0.707186   0.250402   0.918511    -0.238122
1456              0.290872   1.061367   0.222975     1.104925
1457             -1.039872   1.569647  -1.002492     0.215641
1458             -0.041814  -0.832788  -0.704406     0.046905
1459             -0.873529  -0.493934  -0.207594     0.452784

[1460 rows x 4 columns]

This is a live coding demo - type it out!

Setup: - Make sure ames_clean.csv is in ../data/ - Import statements first

Walk through step by step:

Step 1: Label Encoding (Lines 8-10)

Before running: - “We have 28 unique neighborhoods - that’s too many for dummy encoding” - “Label encoding creates one column with integers 0-27” - “This works well for tree-based models”

Run the code:

le = LabelEncoder()
ames['Neighborhood_Encoded'] = le.fit_transform(ames['Neighborhood'])
print(f"Encoded 28 neighborhoods into: {ames['Neighborhood_Encoded'].unique()}")

Point out: - Went from text labels to integers - Each neighborhood gets a unique number - Compressed 28 potential dummy columns into 1 column

Warning to students: “Never use label encoding for nominal variables with LINEAR models! The model will think Neighborhood 27 is ‘bigger than’ Neighborhood 1, which is meaningless.”

Step 2: Scaling (Lines 12-18)

Before running: - “These three features are on very different scales” - “GrLivArea ranges from 800-4000” - “YearBuilt ranges from 1900-2010” - “If we use these in KNN, GrLivArea dominates!”

Run the code:

scaler = StandardScaler()
features_to_scale = ['GrLivArea', 'YearBuilt', 'TotalBsmtSF']
ames[features_to_scale] = scaler.fit_transform(ames[features_to_scale])
print("\nAfter StandardScaler:")
print(ames[features_to_scale].describe())

Point out the output: - Mean is now ≈ 0 for each feature (might be 1e-15, basically 0) - Std is now ≈ 1.0 for each feature - All features on same scale now!

Key teaching points:

When to scale:
- “About to use KNN, SVM, or regularized regression? Scale!”
- “Using Random Forest or decision trees? Don’t bother scaling”
Fit vs Transform:
- “We fit on training data, transform both train and test”
- “This prevents data leakage - we’ll see more on that next”
StandardScaler vs MinMaxScaler:
- “StandardScaler is default choice (mean=0, std=1)”
- “MinMaxScaler when you need [0, 1] range (like neural nets)”

Common student mistakes to address:

❌ Scaling before train/test split ✓ Split first, then fit scaler on train only

❌ Using label encoding for nominal variables with linear models ✓ Use dummy encoding instead

❌ Scaling when using tree-based models ✓ Trees don’t care about scale - save the computation!

Transition: “These techniques are powerful, but there’s a dangerous pitfall: data leakage. Pipelines solve this.”

Preventing Data Leakage with Pipelines

The problem: If you fit a scaler on the entire dataset (train + test), test set information “leaks” into training.

The solution: Pipelines ensure transformations fit on training data only.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

# Create pipeline: scaling → model
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(n_estimators=100))
])

# Fit on training data (scaler learns from X_train only!)
pipeline.fit(X_train, y_train)

# Predict on test data (scaler uses training stats to transform X_test)
predictions = pipeline.predict(X_test)

Important

Pipelines = reproducible, leak-free workflows!

This is the most important slide for production ML.

The Data Leakage Problem:

Wrong approach (causes leakage):

# ❌ WRONG: Fitting scaler on ALL data
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(X)  # Includes both train and test!

# Now split
X_train, X_test, y_train, y_test = train_test_split(X_all_scaled, y)

Why this is wrong: - Scaler learned mean/std from ENTIRE dataset - Including test set! - Test set information influenced the preprocessing - This is data leakage

Impact: - Overly optimistic performance estimates - Model performs worse in production - Stakeholders lose trust

The Right Approach: Pipelines

# ✓ CORRECT: Pipeline ensures proper fit/transform
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier())
])

# First split, THEN fit pipeline
X_train, X_test, y_train, y_test = train_test_split(X, y)
pipeline.fit(X_train, y_train)  # Scaler fits on X_train only

What happens internally:

When you call pipeline.fit(X_train, y_train): 1. Scaler fits on X_train (learns mean, std) 2. Scaler transforms X_train using those stats 3. Model trains on the scaled X_train

When you call pipeline.predict(X_test): 1. Scaler transforms X_test using training mean/std (NOT test mean/std!) 2. Model predicts on the scaled X_test

Key Benefits of Pipelines:

Prevents data leakage - Transformations fit on train only
Less code - One .fit() call instead of multiple steps
Easier deployment - Entire workflow in one object
Prevents mistakes - Can’t forget to scale test data

Real-World Workflow:

# Step 1: Split data
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Step 2: Create pipeline
pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier())
])

# Step 3: Use GridSearchCV with pipeline
param_grid = {'model__n_estimators': [100, 200, 300]}
grid_search = GridSearchCV(pipeline, param_grid, cv=5)

# Step 4: Fit on training data
grid_search.fit(X_train, y_train)

# Step 5: Evaluate on test set ONCE
test_score = grid_search.score(X_test, y_test)

Notice: - Imputation, scaling, and modeling all in one pipeline - GridSearchCV works with pipelines - Test set only touched ONCE at the end

Common student questions:

“Do I always need pipelines?”
- If you’re doing ANY preprocessing: YES
- Even if just scaling: YES
- Only exception: No preprocessing at all
“Can I put multiple preprocessing steps in a pipeline?”
- Absolutely! That’s the power of pipelines
- Imputation → Scaling → PCA → Model
- All as one reproducible unit
“How do I tune hyperparameters in a pipeline?”
- Use double underscore: 'model__n_estimators'
- Format: 'step_name__parameter_name'

Transition: “Now let’s tie everything together - CV + hyperparameter tuning + feature engineering.”

Putting It All Together

The Complete Professional Workflow

flowchart TD
    A[Full Dataset] --> B[Stage 1:<br/>Train/Test Split]
    B --> C[Training Set 80%]
    B --> D[Test Set 20%<br/>🔒 LOCKED]

    C --> E[Stage 2:<br/>Cross-Validation]
    E --> F[Try different models<br/>Tune hyperparameters<br/>Engineer features]
    F --> G[Stage 3:<br/>Select Best Approach]

    G --> H[Stage 4:<br/>Retrain on Full<br/>Training Set]

    H --> I[Stage 5:<br/>Evaluate on Test Set<br/>ONCE]
    D --> I
    I --> J[Report Final<br/>Performance]

    style D fill:#ff6b6b
    style E fill:#51cf66
    style I fill:#ffd43b
    style J fill:#51cf66

This is the culmination - the proper professional workflow.

Walk through each stage deliberately:

Stage 1: Train/Test Split - “Before doing ANYTHING else, split data” - “Lock test set away - don’t even look at it” - “Everything from now on uses training set only” - Typical: 80/20 or 70/30 split

Stage 2: Cross-Validation for All Decisions - “This is where you spend most of your time” - “Try different models, compare them with CV” - “Tune hyperparameters with GridSearchCV (which uses CV)” - “Engineer features, check impact with CV” - “You can make UNLIMITED decisions here” - “Test set stays pristine”

Examples of decisions in Stage 2: - Should I use Random Forest or Gradient Boosting? - What max_depth works best? - Should I scale my features? - Should I add polynomial features? - How should I handle missing data?

ALL answered using cross-validation on training set!

Stage 3: Select Best Approach - “Based on CV scores, pick your winner” - “Maybe it’s Random Forest with n_estimators=200, max_depth=15” - “Maybe you decided to scale features and add interactions” - “This is your final model configuration”

Stage 4: Retrain on Full Training Set - “Up until now, in CV, we trained on 80% of training set (4/5 folds)” - “Now we want to use ALL training data” - “Retrain your best model on the entire training set” - “This gives it maximum data to learn from”

Stage 5: Final Evaluation on Test Set - “Now and ONLY now, evaluate on test set” - “This is done ONCE” - “This is your honest, unbiased performance estimate” - “This is what you report to stakeholders”

Key principle: Test set is like your final exam - you only get to take it once. All practice and preparation (Stages 2-4) use the training set.

Common student misconceptions:

❌ “Can I try a few final models on the test set and pick the best?” ✓ “No! That defeats the purpose. Pick best model using CV, then test ONCE.”

❌ “My CV score was 85% but test score was 82%. Did I do something wrong?” ✓ “Not necessarily. Some variation is expected. If gap is large (>5%), investigate.”

❌ “Can I go back and tune more if test score is disappointing?” ✓ “Technically yes, but then you need a NEW test set. Don’t iteratively peek.”

Real-world context: “In industry, your ‘test set’ is production. You deploy once and see how it performs. You don’t get to keep redeploying and testing. That’s why this workflow is critical.”

Example timeline: - Stage 1: 5 minutes (split data) - Stage 2: Hours to days (try things, tune, engineer) - Stage 3: 5 minutes (pick winner) - Stage 4: Minutes (retrain) - Stage 5: Seconds (evaluate)

Transition: “Let’s see what you’ll practice with this in Thursday’s lab.”

Looking Ahead

Key Takeaways

Cross-validation prevents test set contamination and gives reliable performance estimates
Hyperparameter tuning finds the sweet spot in the bias-variance tradeoff
Feature engineering turns raw data into powerful inputs (often matters more than algorithms!)
Pipelines ensure reproducible, leak-free workflows
The 5-stage workflow is how professionals build trustworthy models

Reinforce the big picture:

1. Cross-Validation - “This is how you make unlimited decisions without peeking at test set” - “5-fold or 10-fold is standard” - “Built into GridSearchCV” - “Gives you mean AND variance of performance”

2. Hyperparameter Tuning - “Every model has knobs to turn” - “GridSearchCV systematically searches combinations” - “Watch for overfitting (big train/validation gap)” - “Balance between training and validation performance”

3. Feature Engineering - “Often the highest-leverage activity in ML” - “Domain knowledge is key” - “Four main categories: encoding, scaling, creating, handling missing” - “Talk to experts in the problem domain!”

4. Pipelines - “Non-negotiable for production ML” - “Prevents data leakage” - “Makes deployment easier” - “One object, entire workflow”

5. The Complete Workflow - “5 stages: Split, CV, Select, Retrain, Test” - “Test set touched ONCE” - “Everything else uses CV on training set” - “This is what separates beginners from professionals”

Connecting to previous weeks: - “You’ve learned great models: linear regression, trees, forests” - “This week: How to tune them properly and prepare data right” - “Next week: More advanced topics building on this foundation”

The meta-lesson: “Data science isn’t just about knowing algorithms. It’s about having a disciplined workflow that produces trustworthy results. These three techniques form the foundation of professional ML practice.”

Motivate Thursday lab: “Reading about this is one thing. Doing it is another. Thursday you’ll practice the full workflow with real data. You’ll see common mistakes and how to avoid them. Come ready to code!”

Connection to Thursday’s Lab

This Week’s Lab Preview

In Thursday’s lab, you’ll get hands-on practice with:

Implementing the 5-stage workflow from scratch
Using GridSearchCV to tune a Random Forest
Building feature engineering pipelines
Comparing performance with and without proper CV
Seeing what happens when you peek at the test set (spoiler: bad things!)

Come prepared to apply today’s concepts!

Set expectations for Thursday:

What students will do:

Start with the “wrong” way:
- Split data
- Try different hyperparameters, checking test set each time
- See how test score becomes unreliable
- Experience the peeking problem firsthand
Then do it the right way:
- Split data
- Use cross-validation for all decisions
- Tune with GridSearchCV
- Evaluate on test set ONCE
- Compare: Which test score is trustworthy?
Feature engineering practice:
- Encode categorical variables
- Scale numerical features
- Create new features (interactions, polynomials)
- Handle missing data
- Build pipelines to prevent leakage
Full workflow integration:
- Combine CV + GridSearchCV + Pipelines
- Complete end-to-end ML project
- Report final, honest performance

Lab structure: - Part 1: Cross-validation basics - Part 2: GridSearchCV hyperparameter tuning - Part 3: Feature engineering and pipelines - Part 4: Complete workflow on a new dataset

What to bring: - Laptop with Python environment - Course materials (slides, chapter notes) - Questions from today’s lecture - Willingness to make mistakes and learn!

Time allocation: - Expect to spend 2-3 hours - Work in pairs (encouraged) - TAs available for questions

Common lab challenges to preview:

Forgetting to split before CV:
- Students try to CV on full dataset
- Remind: Split first, CV on training set
Pipeline syntax:
- Double underscore for nested parameters
- model__n_estimators not n_estimators
Interpreting CV scores:
- Understanding mean and std
- What’s “good enough” variance?
GridSearchCV taking forever:
- Remind about computational cost
- Start with coarse grid, then refine
- Use n_jobs=-1 for parallelization

Transition: “Any questions before we wrap up?”

Questions & Next Steps

Looking Ahead

Btwn now & Thursday: Complete reading Chapters 28-30

Thursday Lab: Hands-on practice with CV, hyperparameter tuning, and feature engineering

This weekend: Week 12 quiz & homework (based on lab)

Next Week: Move into the world of unsupervised ML

Next next week: Advanced topics & Final project prep!

Homework reminders:

Readings: - Chapter 28: Cross-Validation (focus on the 5-stage workflow) - Chapter 29: Hyperparameter Tuning (understand bias-variance tradeoff) - Chapter 30: Feature Engineering (the four main techniques)

Quiz: - Covers all three chapters - Due before next Tuesday - Tests conceptual understanding - 10 questions, multiple choice and scenarios

What to review before Thursday: 1. How k-fold cross-validation splits data 2. The difference between overfitting and underfitting 3. When to scale features (which algorithms care) 4. Why pipelines prevent data leakage

Office hours: - Tuesday/Thursday 2-3pm - Or by appointment - Come with specific questions

TA sessions: - Wednesday evenings - Great for lab prep

Resources: - Scikit-learn documentation for GridSearchCV - Chapter companion notebooks - Previous lab solutions (for reference)

Any Final Questions?

About today’s concepts?
About Thursday’s lab?
About upcoming assignments?
About rest of the term?

Common final questions:

Q: “Do I need to memorize the GridSearchCV syntax?” A: “No, but understand what it does. You can always look up syntax. Focus on when and why to use it.”

Q: “Will the quiz cover all three chapters equally?” A: “Yes, roughly equal coverage. Some questions integrate concepts (like ‘when to scale AND use CV’).”

Q: “How long will Thursday’s lab take?” A: “Plan for 2-3 hours. Some finish faster, some take longer. No rush - learning matters more than speed.”

Q: “Can I use GridSearchCV on the midterm project?” A: “Absolutely! In fact, I encourage it. Shows you understand proper workflow.”

Q: “What if my CV scores are really different across folds?” A: “High variance in CV scores suggests model instability. Might need more data, simpler model, or different features. We’ll discuss in lab.”

Q: “Do I always need to use pipelines?” A: “For any real project: yes. Only exception is exploratory analysis with no preprocessing. When in doubt, use a pipeline.”

Closing message: “This week’s topics - CV, hyperparameter tuning, feature engineering - are what separate textbook ML from production ML. Master these, and you’ll be doing professional-quality data science. See you Thursday!”

See you Thursday for hands-on practice!