Week 9: Correlation & Regression Foundations
Quick overview of today’s plan:
🎯 Supervised learning
We have a target to learn from (an “answer key”)
Regression — predict a number
How much will this customer spend next month?
Classification — predict a category
Will this customer cancel?
🔍 Unsupervised learning
No target: the algorithm finds structure on its own
Clustering — discover groups
What types of shoppers do we have?
Tip
This week we go deep on the first one: regression.
What’s one regression or classification model you interacted with this past week?
What was it predicting? Was the output a number or a category?
A couple of mine:
In business, we rarely care about a single number in isolation. Leaders ask questions like:
Today’s Goal: Learn to measure and model these relationships quantitatively.
Think about your work experience, internships, or daily life:
Share your examples with your partner!
Then we’ll take a few responses…
Correlation measures how strongly two variables move together in a linear relationship.
Key Insight: Correlation is descriptive, it tells you variables move together, but not why.
Your turn
Give me a business or everyday example of a positive, a negative, and no correlation.
Setup & Game Link
Teams: I’m dividing the room into 2-3 teams
Game: We’ll use the interactive game at guessthecorrelation.com
How it works:
Scoring: Keep track of your team’s average accuracy
Prize: Winning team gets extra credit bragging rights! 🎉
Let’s play 5-10 rounds!
Congratulations to our winning team! 🏆
Key Takeaways from the Challenge:
Next: How do we move from measuring relationships to making predictions?
Your turn! Let’s analyze some real grocery chain data.
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
# Example dataset: advertising spend vs. weekly sales
data = pd.DataFrame({
"ad_spend": [400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900, 2000, 2100, 2200, 2300],
"weekly_sales": [4200, 4400, 4100, 4800, 5600, 5200, 4900, 5500, 5300, 5900, 5700, 6300, 6900, 6200, 5800, 6600, 7100, 6800, 7300, 7800]
})
# Visualize the relationship
plt.figure(figsize=(8, 5))
plt.scatter(data["ad_spend"], data["weekly_sales"], alpha=0.7, s=60)
plt.xlabel("Advertising Spend ($)")
plt.ylabel("Weekly Sales")
plt.title("Ad Spend vs. Weekly Sales")
plt.grid(True, alpha=0.3)
plt.show()Question for you: What do you think the correlation is? Write down your guess!
Let’s see how close your guess was!
Correlation coefficient: 0.941
Full correlation matrix:
| ad_spend | weekly_sales | |
|---|---|---|
| ad_spend | 1.000000 | 0.941372 |
| weekly_sales | 0.941372 | 1.000000 |
Interpretation:
We found a strong correlation (~0.9) between advertising and sales.
But now the business question is: Can we use advertising spend to predict future sales?
Question: If you could draw a line to represent this relationship, what would it look like?
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
# Visualize the relationship
plt.figure(figsize=(8, 5))
plt.scatter(data["ad_spend"], data["weekly_sales"], alpha=0.7, s=60)
plt.xlabel("Advertising Spend ($)")
plt.ylabel("Weekly Sales")
plt.title("Ad Spend vs. Weekly Sales")
plt.grid(True, alpha=0.3)
plt.show()Remember from high school? The equation of a line:
\[ y = mx + b \]
With linear regression, we follow a similar approach but the typical equation you’ll see is:
\[ y = \beta_0 + \beta_1 x \]
Where:
Important
The goal: Find the best values for slope and intercept that minimize prediction errors!
from sklearn.linear_model import LinearRegression
# Prepare the data
X = data[["ad_spend"]] # Feature matrix (note the double brackets)
y = data["weekly_sales"] # Target variable
# Fit the model
model = LinearRegression()
model.fit(X, y)
# Extract the results
intercept = model.intercept_
slope = model.coef_[0]
print(f"Intercept (β₀): ${intercept:.0f}")
print(f"Slope (β₁): ${slope:.2f}")Intercept (β₀): $3552
Slope (β₁): $1.68
Our fitted equation: \[ \text{Weekly Sales} = 3552 + 1.68 \times \text{Ad Spend} \]
Interpretation: For every $1 increase in advertising, we expect weekly sales to increase by $1.68!
# Create the regression line visualization
plt.figure(figsize=(10, 6))
plt.scatter(data["ad_spend"], data["weekly_sales"], alpha=0.7, s=60, label="Actual data")
# Add the fitted regression line
plt.plot(data["ad_spend"], model.predict(X), color="red", linewidth=3, label="Regression line")
# Show specific predictions
pred_x = [1500, 2000]
for x_val in pred_x:
pred_y = model.predict([[x_val]])[0]
plt.scatter(x_val, pred_y, color="orange", s=120, zorder=5)
plt.annotate(f"${x_val} → ${pred_y:.0f}",
xy=(x_val, pred_y), xytext=(5, -15),
textcoords='offset points', fontsize=10)
plt.xlabel("Advertising Spend ($)")
plt.ylabel("Weekly Sales")
plt.title("Linear Regression: Predictions on the Line")
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()Key insight: Every prediction falls exactly on our regression line!
Business value: Now we can forecast sales for any advertising budget!
# Make predictions for different scenarios
scenarios = pd.DataFrame({
'ad_spend': [1500, 2500, 3000]
})
predictions = model.predict(scenarios)
print("Business Predictions:")
for spend, pred in zip(scenarios['ad_spend'], predictions):
print(f"Spend ${spend:,} → Predicted Sales: ${pred:,.0f}")
# Manual calculation to verify
print(f"\nManual check for $1,500:")
manual_pred = 1.68 * 1500 + 3552
print(f"1.68 × 1500 + 3552 = ${manual_pred:.0f}")Business Predictions:
Spend $1,500 → Predicted Sales: $6,072
Spend $2,500 → Predicted Sales: $7,752
Spend $3,000 → Predicted Sales: $8,592
Manual check for $1,500:
1.68 × 1500 + 3552 = $6072
We now have a model that can make predictions…
Question 🤔
How do we know if this is a GOOD model?
How would you assess prediction quality?
Two analysts forecast weekly sales for your store. Here’s their track record:
| Week | Actual Sales | Forecaster A | Forecaster B |
|---|---|---|---|
| 1 | $6,100 | $5,900 | $6,100 |
| 2 | $5,400 | $5,700 | $5,350 |
| 3 | $6,800 | $6,600 | $6,850 |
| 4 | $5,900 | $6,100 | $5,900 |
| 5 | $7,200 | $6,900 | $8,300 |
With your neighbor:
To decide, you compared predictions with what actually happened, then summarized the misses. That’s all model evaluation is.
🔮 Predict
Have the model make predictions
➜
⚖️ Compare
Line them up with the actual outcomes
➜
📏 Summarize
Condense the misses into a single metric
➜
✅ Decide
Deploy it, improve it, or pick a better model
Why bother?
Root Mean Squared Error turns a list of misses into one number: ① find each error (actual − forecast), ② square it, ③ take the mean, ④ take the square root.
| Forecaster A | Forecaster B | ||||||
|---|---|---|---|---|---|---|---|
| Week | Actual | Forecast | ① Error | ② Error² | Forecast | ① Error | ② Error² |
| 1 | $6,100 | $5,900 | +200 | 40,000 | $6,100 | 0 | 0 |
| 2 | $5,400 | $5,700 | −300 | 90,000 | $5,350 | +50 | 2,500 |
| 3 | $6,800 | $6,600 | +200 | 40,000 | $6,850 | −50 | 2,500 |
| 4 | $5,900 | $6,100 | −200 | 40,000 | $5,900 | 0 | 0 |
| 5 | $7,200 | $6,900 | +300 | 90,000 | $8,300 | −1,100 | 1,210,000 |
| ③ Mean | 60,000 | ③ Mean | 243,000 | ||||
| ④ RMSE | $245 | ④ RMSE | $493 |
Tip
Both forecasters miss by $240 per week on average, but squaring makes B’s one big miss count heavily. Lower RMSE is better, and the square root puts it back in the same units as what you’re predicting ($).
\[ R^2 = 1 - \frac{\text{model's squared error}}{\text{baseline's squared error}} \]
Our model: R² ≈ 0.89. Using ad spend removes about 89% of the error we’d make by guessing the average every week. In other words, it explains 89% of the variation in weekly sales.
You won’t compute these by hand. Scikit-learn does it in one line each:
R²: 0.89
RMSE: $347
Note
Both functions take the actual values first, then the predictions.
📏 RMSE
Asks: How far off are my predictions, typically?
Units: same as the target (dollars, units, minutes…)
Better: lower (0 = perfect)
Our model: “Forecasts are typically off by about $347 per week”
📊 R²
Asks: How much better is my model than guessing the average?
Units: none (a proportion, usually 0 to 1)
Better: higher (1 = perfect)
Our model: “Ad spend explains about 89% of the week-to-week variation in sales”
Tip
Use both: R² tells you how much of the pattern you’ve captured; RMSE tells the business what size of miss to plan for. Your readings add two more: MAE and MAPE.
The store manager says: “I can plan staffing and inventory as long as forecasts are within about $500 a week.”
Our model: R² = 0.89, RMSE = $347
With your neighbor:
We just scored our model on the same 20 weeks it learned from.
That’s like…
Getting the exact exam questions the night before. A perfect score proves you memorized the answers, not that you learned the material.
The manager doesn’t care how well the model “predicts” weeks that already happened.
The real question is: how will it do on next week’s sales, which it has never seen?
Suppose we hide 6 of our 20 weeks and train two models on the other 14:
Based on training RMSE, which model would you deploy?
Now let’s bring back the 6 hidden weeks:
The “perfect” model is worse on weeks it hasn’t seen. Its $0 training score told us nothing about the future. Only data the model never saw reveals what it actually learned.
🗂️ Full dataset
All the historical data we have
➜
🏋️ Training set (70–80%)
The model learns the pattern here
➜
🔒 Test set (20–30%)
Locked away until the very end, then used once to score the model
➜
🔮 Test score
Our best preview of how the model will do on future data
The golden rule
The model never sees the test set while it’s learning. The test set stands in for the future.
Training weeks: 14
Test weeks: 6
test_size=0.3: hold out 30% of the rows for testingrandom_state=11: rows are assigned randomly; fixing the seed gives the same split every time you run it# Learn ONLY from the training weeks
model = LinearRegression()
model.fit(X_train, y_train)
# Score the model on both sets
train_preds = model.predict(X_train)
test_preds = model.predict(X_test)
print(f"Training → R²: {r2_score(y_train, train_preds):.2f} | RMSE: ${root_mean_squared_error(y_train, train_preds):,.0f}")
print(f"Test → R²: {r2_score(y_test, test_preds):.2f} | RMSE: ${root_mean_squared_error(y_test, test_preds):,.0f}")Training → R²: 0.90 | RMSE: $322
Test → R²: 0.83 | RMSE: $411
The test numbers are the ones we report to the manager. A test score slightly worse than the training score is normal: the model was tuned to the training weeks.
✅ Train ≈ Test
The model generalizes: it learned a real pattern
Trust the test score
⚠️ Train great, Test much worse
The model memorized the training data (overfitting)
Like our connect-the-dots model
❌ Train poor, Test poor
The model is too simple or missing key information (underfitting)
Try better features or a different model
Note
Overfitting and underfitting get a deeper treatment in your readings, and we’ll keep coming back to them for the rest of the semester.
What happens if we change only the random_state?
random_state= 1 → Test RMSE: $490
random_state= 3 → Test RMSE: $523
random_state= 5 → Test RMSE: $447
random_state=11 → Test RMSE: $411
random_state=21 → Test RMSE: $238
Why does the test score bounce around?
With only 6 test weeks, the score depends heavily on which weeks land in the test set. More data gives more stable estimates. In Week 12, cross-validation solves this by testing on several different splits and averaging the results.
Let’s see the complete machine learning workflow using a fresh dataset:
# Step 1: Import a dataset
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, root_mean_squared_error
# Load the classic Advertising dataset
advertising = pd.read_csv("../data/Advertising.csv")
print("Dataset shape:", advertising.shape)
advertising.head()Dataset shape: (200, 4)
| TV | radio | newspaper | sales | |
|---|---|---|---|---|
| 0 | 230.1 | 37.8 | 69.2 | 22.1 |
| 1 | 44.5 | 39.3 | 45.1 | 10.4 |
| 2 | 17.2 | 45.9 | 69.3 | 9.3 |
| 3 | 151.5 | 41.3 | 58.5 | 18.5 |
| 4 | 180.8 | 10.8 | 58.4 | 12.9 |
Step 1 complete: We have our data loaded and ready!
# Step 2: Split data into features (X) and target (y), then train/test
X = advertising[['TV', 'radio', 'newspaper']] # Multiple predictors
y = advertising['sales'] # Target variable
# Split into train/test sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
print(f"Training set: {len(X_train)} samples")
print(f"Test set: {len(X_test)} samples")Training set: 160 samples
Test set: 40 samples
Step 2 complete: Data is properly split for honest evaluation!
Model coefficients:
TV: 0.045
radio: 0.189
newspaper: 0.003
Intercept: 2.979
Step 3 complete: Model is trained and ready to make predictions!
# Step 4: Evaluate model performance
train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)
# Calculate metrics
train_r2 = r2_score(y_train, train_predictions)
test_r2 = r2_score(y_test, test_predictions)
train_rmse = root_mean_squared_error(y_train, train_predictions)
test_rmse = root_mean_squared_error(y_test, test_predictions)
print("Performance Summary:")
print(f"Training R²: {train_r2:.3f} | Training RMSE: {train_rmse:.2f}")
print(f"Test R²: {test_r2:.3f} | Test RMSE: {test_rmse:.2f}")Performance Summary:
Training R²: 0.896 | Training RMSE: 1.64
Test R²: 0.899 | Test RMSE: 1.78
Complete workflow achieved! 🎉 We can now trust our model’s performance estimates.
📖 Multiple regression
Using many predictors at once
📊 Categorical variables
Turning categories like “Region” into numbers
⚖️ Overfitting vs. underfitting
Finding the right amount of model complexity
💼 MAE & MAPE
More metrics, and how to pick the one that fits the business
This Week’s Lab Preview
In Thursday’s lab, you’ll get hands-on practice with:
Homework
This week’s Lab will serve as your homework!
BANA 4080 | Week 9