Creates Valid Probabilities

Magic Properties:

• Always between 0 and 1 - perfect for probabilities!
• S-shaped curve - captures realistic business relationships
• Smooth transitions - gradual probability changes, not jumps
• 50% at center - when z = 0, probability = exactly 50%

Building a Logistic Regression Model

Step 1: Load and explore the ISLP Default dataset

from ISLP import load_data
import pandas as pd

# Load the Default dataset
Default = load_data('Default')

Default[['default', 'balance', 'income']].head()

from sklearn.linear_model import LogisticRegression

# Prepare features (balance and income) and target
X = Default[['balance', 'income']]
y = (Default['default'] == 'Yes').astype(int)

# Fit logistic regression model
model = LogisticRegression()
model.fit(X, y)

# Extract the coefficients
intercept = model.intercept_[0]
balance_coef = model.coef_[0][0]
income_coef = model.coef_[0][1]

print(f"Intercept: {intercept:.6f}")
print(f"Balance coefficient: {balance_coef:.6f}")
print(f"Income coefficient: {income_coef:.6f}")

Intercept: -11.540468
Balance coefficient: 0.005647
Income coefficient: 0.000021

Interpreting Our Coefficients

Now we can understand what our coefficients mean:

Intercept (-11.540468):

• When balance = $0 and income = $0, the log-odds of default = -11.54
• This means very low baseline probability of default (~0.000973%)

Balance coefficient (0.005647):

• For each $1 increase in balance, log-odds increase by 0.005647
• For each $1,000 increase in balance, log-odds increase by ~5.6 Probability increases ~95%
• Positive coefficient = higher balance increases default risk ✓

Income coefficient (0.000021):

• For each $1 increase in income, log-odds increase by 0.000021
• For each $10,000 increase in income, log-odds increase by ~0.21
• Probability increases by only ~5%
• Tiny positive coefficient = income has minimal effect on default risk

Business Translation:

• Balance is the primary driver of default risk
• Income has almost no effect once we account for balance
• A customer with $2,000 balance is much riskier than one with $1,000 balance, regardless of income

From Coefficients to Predictions

Step 1: Use coefficients in the logistic function

import numpy as np

# Manual calculation for $1,500 balance, $50,000 income
balance = 1500
income = 50000
log_odds = -11.540468 + 0.005647 * balance + 0.000021 * income
probability_manual = 1 / (1 + np.exp(-log_odds))

print(f"Manual calculation: {probability_manual:.4f}")

Manual calculation: 0.1171

Step 2: Compare with scikit-learn

# Scikit-learn prediction (must include both features)
customer_df = pd.DataFrame({'balance': [1500], 'income': [50000]})
probability_sklearn = model.predict_proba(customer_df)[:, 1]

print(f"Scikit-learn prediction: {probability_sklearn[0]:.4f}")

Scikit-learn prediction: 0.1162

Both give similar results for a customer with $1,500 balance and $50,000 income

Evaluating Model Performance

Now that we’ve built our model, let’s evaluate how well it performs on new data:

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Split data into training and test sets
X = Default[['balance', 'income']]
y = (Default['default'] == 'Yes').astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Train the model on training data
model = LogisticRegression()
model.fit(X_train, y_train)

# Make predictions on test data
y_pred = model.predict(X_test)

# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy:.1%}")

Model Accuracy: 97.1%

Understanding Classification Errors

The Confusion Matrix: Where Our Model Goes Wrong

Why These Errors Matter:

• False Positives (FP) are harmful because:
  • Good customers get denied credit or charged higher rates
  • Customer frustration leads to churn and lost revenue
  • Wasted resources investigating “false alarms”
• False Negatives (FN) are harmful because:
  • Risky customers get approved and actually default
  • Direct financial losses from unpaid debts
  • Missed opportunities to offer risk-appropriate pricing

Key Insight: Better performance metrics focus on the errors that matter most to your business!

Scenario 1: Medical Cancer Screening

You work for a hospital developing an AI system to screen for early-stage cancer.

Business Context:

• System analyzes medical scans to flag potential cancer cases
• Doctors use predictions to decide whether to order additional tests
• Early detection dramatically improves patient outcomes

Error Costs:

• False Positive: Healthy patient flagged as having cancer → $1,500 in unnecessary follow-up tests, patient anxiety
• False Negative: Cancer patient marked as healthy → $50,000+ in delayed treatment costs, potentially life-threatening

Scenario 2: Credit Card Fraud Detection

You work for a bank building a fraud detection system for credit card transactions.

Business Context:

• System analyzes transactions in real-time to flag potential fraud
• Flagged transactions are automatically blocked from processing
• Customer satisfaction and transaction volume are key business metrics

Error Costs:

• False Positive: Legitimate transaction blocked → Angry customer at checkout, potential customer churn, lost transaction fees
• False Negative: Fraudulent transaction approved → $200 average fraud loss (relatively small compared to transaction volume)

Scenario 3: Marketing Campaign Targeting

You work for a company planning targeted email marketing campaigns for product promotions.

Business Context:

• System predicts which customers are likely to purchase based on promotions
• Marketing budget is limited, so targeting efficiency matters
• Missing potential customers means lost sales opportunities

Error Costs:

• False Positive: Send promotion to non-buyer → $2 wasted marketing cost per customer
• False Negative: Miss a potential buyer → $5 lost profit opportunity per customer

Scenario Challenge: Answers

Business-aligned metric selection:

1. Airport Security → Recall

• Missing dangerous items = catastrophic
• False alarms inconvenient but acceptable