Welcome to Week 8

  • Quick overview of today’s plan:

    • Midterm Project Feedback & Discussion
    • Peer Review Reminders
    • Introduction to Machine Learning Concepts
    • Key Considerations Before Building ML Models

Discussion: Midterm Project & Peer Reviews

Midterm Project Feedback

Let’s start with some feedback and discussion on the midterm project:

  • How did you find the project overall?
  • What aspects were most challenging?
  • What did you learn that surprised you?
  • Any questions about the analysis or technical implementation?

Activity

Converse with your neighbor and identify…

  • 1 thing you learned from the midterm project that you’ll use in future data projects
  • 1 aspect of the project that was more challenging than expected

Peer Review Reminder

Important: Your peer reviews are due by end of day Thursday

Peer Review Guidelines

  • Review your assigned classmate’s project thoughtfully
  • Focus on both technical execution and business insights
  • Provide constructive feedback on:
    • Data analysis approach
    • Visualization effectiveness
    • Business recommendations
    • Code organization and clarity
  • Be specific and helpful in your comments

Questions about the peer review process?

Introduction to Machine Learning

What is Machine Learning?

You’ve mastered exploratory data analysis - describing what happened in your data. Now we move to machine learning - predicting what might happen next!

  • Artificial Intelligence (AI): Building systems that mimic human intelligence
  • Machine Learning (ML): Algorithms that learn patterns from data
  • Data Mining: Discovering insights and patterns (includes both EDA and ML)

EDA vs ML: Understanding the Difference

Exploratory Data Analysis (EDA)

✅ Advantages:

  • Understand what happened in the past
  • Identify patterns and trends
  • Summarize and visualize data
  • Quick insights and reporting
  • No complex algorithms needed

❌ Limitations:

  • Cannot predict future events
  • Limited to descriptive insights
  • Requires manual interpretation
  • Cannot automate decisions

EDA vs ML: Understanding the Difference

Machine Learning (ML)

✅ What it can do:

  • Predict future outcomes
  • Automate decision-making
  • Handle complex patterns
  • Scale to large datasets
  • Improve over time

Business Questions ML Answers:

  • “Which customers will churn next month?”
  • “What price should we set for this product?”
  • “Is this transaction fraudulent?”
  • “What content should we recommend?”

ML is Everywhere Around You!

You interact with machine learning dozens of times every day:

Think-Pair-Share: ML vs EDA

Let’s practice distinguishing between problems that need ML vs. basic data analysis:

Scenarios to Discuss:

  • A retail company wants to understand which regions had the highest sales last quarter
  • Netflix wants to automatically recommend movies to users based on viewing history
  • A hospital wants to identify which patients might be readmitted within 30 days
  • A marketing team wants to create a dashboard showing customer demographics

Then we’ll take a few responses…

Two Main Types of Machine Learning

Supervised Learning

  • Has labeled examples (input → output)
  • Learn from “teacher” providing correct answers
  • Goal: Make predictions on new data

Example: Learning to recognize spam emails by studying emails already labeled as “spam” or “not spam”

Unsupervised Learning

  • No labeled outputs
  • Discover hidden patterns
  • Goal: Find structure in data

Example: Analyzing customer data to discover natural groupings without knowing what those groups should be

Supervised Learning: Two Main Types

In supervised learning, we have historical data with known inputs and outputs. There are two main types:

Regression: Predicting continuous numbers

📈 Examples:

  • House prices ($150,000, $275,000, $420,000)
  • Customer lifetime value ($1,200, $3,500)
  • Stock prices ($45.32, $67.89)
  • Sales revenue ($12,000, $25,000)

Business Question: “How much will this house sell for?”

Classification: Predicting categories

🏷️ Examples:

  • Email spam (Spam, Not Spam)
  • Credit approval (Approved, Denied)
  • Customer churn (Will Leave, Will Stay)
  • Product reviews (Positive, Negative, Neutral)

Business Question: “Is this email spam or not?”

🤔 What are examples of regression & classification in your life?

Unsupervised Learning: Finding Hidden Patterns

In unsupervised learning, we only have inputs - no known “correct answers.” The goal is to discover hidden structures.

Clustering: Finding natural groups

👥 Examples:

  • Customer segments (hi/med/low loyalty)
  • Market segments (Urban, Suburban, Rural)
  • Product categories (Tech, Fashion, Home)
  • Website user types (Browsers vs. Buyers)

Business Question: “What natural customer groups exist in our data?”

How it works:

  • Algorithm analyzes customer data
  • Identifies customers that behave similarly
  • Groups them based on similarities
  • Reveals segments you might never have considered

🤔 How would we cluster people in this class?

ML Application Scenarios

Business Context Activity

Scenario Analysis

For each business scenario below, discuss in your groups:

  1. Is ML required? Could this be solved with basic exploratory data analysis (EDA) instead?
  2. If ML is needed: What type of ML problem is it?
    • Supervised (regression/classification) or unsupervised (clustering)?

Scenarios:

  • An e-commerce site wants to identify their top-performing product categories
  • A bank needs to automatically detect fraudulent credit card transactions
  • A streaming service wants to discover different types of viewing behavior patterns
  • A retailer wants to predict how much revenue each customer will generate next year
  • A hospital wants to understand which departments have the highest patient satisfaction scores
  • A marketing team wants to automatically personalize email subject lines for each customer

You have 5 minutes.

Before We Build: Critical Considerations

Why Planning Matters

Building ML models is like building a house - you need the right foundation:

🏗️ Foundation First

  • Clear problem definition
  • Well-defined success metrics

📊 Quality Materials

  • Clean, relevant data
  • Proper data validation

📋 Proper Permits

  • Ethical considerations
  • Privacy and fairness

🔧 Right Tools

  • Appropriate algorithms
  • Proper evaluation methods

⚠️ Reality Check

  • Many ML projects fail due to poor planning
  • Technical sophistication ≠ Business success
  • Planning prevents costly mistakes

Case Study: StreamFlix Customer Retention

Let’s walk through the planning process with a real business scenario:

The Business Request

Company: StreamFlix (streaming service with 2M+ subscribers)
Stakeholder: VP of Customer Success
Request: “Our customer churn is killing us! Can you build an AI model to predict which customers will cancel so we can save them? We need this ASAP for next quarter’s retention campaign.”

Context:

  • Losing 5% of subscribers monthly ($15M+ annual impact)
  • Customer service team overwhelmed with retention efforts
  • Current retention campaigns are poorly targeted

How do we transform this vague request into a successful ML project?

Step 1: Problem Framing & Success Metrics

The original request: “Build an AI model to predict which customers will cancel”

❌ Problems with this framing:

  • No specific time horizon
  • Unclear business action
  • No success criteria
  • Vague about model use

⚠️ Risk: Building a technically perfect model that doesn’t solve the business problem

✅ Better problem statement: “Predict which customers are likely to cancel within the next 30 days so our retention team can proactively contact them with targeted offers”

Success Metrics:

  • Technical: 80%+ precision on churn prediction
  • Business: Reduce monthly churn from 5% to 4%
  • Operational: Enable team to contact 1,000 highest-risk customers monthly

Step 2: Data Readiness Assessment

Available Data Sources:

  • Customer demographics & subscription history
  • Viewing behavior (watch time, content preferences)
  • Customer service interactions
  • Payment/billing information

Step 2: Data Readiness Assessment


🔍 Data Quality Concerns:

  • Missing values: Some customers lack demographic data
  • Inconsistent definitions: What counts as “active viewing”?
  • Freshness: Viewing data updated daily, billing monthly
  • Completeness: Customer service notes vary in detail

🎯 Train/Test Split Strategy:

  • Why not random? We need to predict future churn
  • Recommended approach: Time-based split
    • Train: Jan 2023 - Dec 2023
    • Test: Jan 2024 - Mar 2024
  • Mimics reality: Use past data → predict future behavior

Step 3: Identifying Data Leakage Risks

⚠️ Common leakage pitfalls in churn prediction:

Temporal Leakage Examples:

  • ❌ “customer_satisfaction_score_march” when predicting February churn
  • ❌ “support_tickets_after_cancellation”
  • ❌ “final_bill_amount” (only available after cancellation decision)

Target Leakage Examples:

  • ❌ “account_status” field containing “cancelled”
  • ❌ “retention_campaign_sent” (only sent to customers planning to leave)

✅ Safe features to use:

  • ✅ Viewing patterns from previous 30-90 days
  • ✅ Historical payment behavior
  • ✅ Customer service interactions before churn window
  • ✅ Subscription tenure and plan details

🚨 Red flags:

  • Features with perfect correlations
  • Performance “too good to be true” (99%+ accuracy)
  • Features that wouldn’t exist at prediction time

Step 4: Ethical Considerations

Fairness & Privacy Concerns:


🤔 Fairness Questions:

  • Could the model discriminate by age, location, or content preferences?
  • Are we unfairly targeting certain demographic groups?
  • Does retention budget get allocated fairly across customer segments?

🔒 Privacy Considerations:

  • Viewing behavior reveals personal interests
  • Location data from streaming devices
  • Payment information sensitivity
  • Customer service conversation content

Step 4: Ethical Considerations

Fairness & Privacy Concerns:


⚖️ Ethical Safeguards:

  • Audit for bias: Test model performance across demographic groups
  • Data minimization: Only use necessary features
  • Transparency: Clear retention offer policies
  • Human oversight: Review high-stakes decisions
  • Customer control: Easy opt-out from retention communications

💡 Interpretability needs:

  • Retention team needs to understand why customers are flagged
  • Required for crafting personalized retention offers

Step 5: Implementation Plan

From Vague Request to Actionable Project:

✅ What we accomplished:

  1. Clear problem definition: 30-day churn prediction for retention targeting
  2. Measurable success criteria: Technical + business + operational metrics
  3. Data readiness plan: Time-based splitting strategy identified
  4. Leakage prevention: Identified risky features to avoid
  5. Ethical framework: Fairness, privacy, and interpretability considerations

🚀 Clear Next steps:

  • Partner with data owners to validate data quality
  • Implement time-based train/test split
  • Build baseline model with safe features
  • Test for demographic bias across customer segments
  • Design interpretable model outputs for retention team

Timeline: 6-8 weeks for responsible development vs. rushed 2-week timeline

Key insight: Planning time is never wasted - it prevents costly mistakes and ensures business impact.

Key Considerations Checklist


  • Clear problem definition: What exactly are you trying to predict? How will you measure success?
  • Data Readiness: Is your data clean, complete, and appropriate for the problem?
  • Data Leakage: Are you accidentally using “future” information that won’t be available when making real predictions?
  • Fairness & Ethics: Could your model discriminate against certain groups? What are the privacy implications?
  • Interpretability: Do stakeholders need to understand how the model makes decisions?

Mini-Exercise: Business Scenario Planning

Scenario: HealthFirst Insurance Risk Assessment

Company: HealthFirst Insurance
Stakeholder Request: “We want to use AI to automatically approve or deny health insurance applications to speed up our process and reduce costs. Can you build a model that predicts whether an applicant is high-risk or low-risk?”

Context:

  • Currently takes 2-3 weeks for manual underwriting review
  • Receiving 1,000+ applications per week
  • Want to auto-approve low-risk applicants instantly
  • Available data: Age, medical history, lifestyle factors, previous claims, zip code, employment

Your Discussion Points:

  1. Problem Framing: How would you clearly define this business question?
  2. Data Readiness: What data quality issues might you encounter?
  3. Data Leakage: What information might accidentally “leak” future knowledge?
  4. Fairness/Privacy: What ethical concerns should guide your decisions?

Time: 10 minutes - We’ll do more of these exercises in Thursday’s lab!

Discussion: What Did You Discover?

Let’s see what insights you came up with! Here are some key considerations:

🎯 Problem Framing Issues:

  • “High-risk” vs “low-risk” needs clear definition
  • Missing success metrics (approval rate? cost savings?)
  • Should model recommend decisions or make them automatically?
  • What happens to borderline cases?

📊 Data Readiness Challenges:

  • Medical history: missing values, inconsistent formats
  • Previous claims: sparse for new customers
  • Lifestyle factors: self-reported reliability issues
  • Employment: may change between application and decision

⚠️ Data Leakage Risks:

  • Using claims data that occurs after application date
  • Medical test results ordered after application submitted
  • Employment verification completed after initial application

⚖️ Major Ethical Concerns:

  • Fairness: Age, zip code, employment discrimination
  • Privacy: Medical information protection (HIPAA)
  • Legal compliance: State insurance regulations, ADA
  • Transparency: Applicants’ right to understand denials

Key Takeaways

Today’s Learning Objectives

  • Machine Learning Concepts – ML predicts the future; EDA describes the past
  • ML Types – Supervised learning uses labeled data; unsupervised finds hidden patterns
  • Business Applications – Not every problem needs ML; sometimes basic analysis is better
  • Planning Importance – Successful ML requires careful upfront consideration of data, ethics, and problem framing

Connection to Thursday’s Lab

This Week’s Lab Preview

In Thursday’s lab, you’ll get hands-on practice with:

  • More business scenario analysis exercises
  • Identifying appropriate ML approaches for different problems
  • Practicing problem framing and success metric definition
  • Exploring real datasets to understand data readiness challenges

Come prepared to apply today’s concepts to practical business problems!

Midterm & Next Steps

Midterm Project Reminders

  • Peer Reviews – Due by end of day Thursday
  • Final Reflection – Consider what you learned and how you’ll apply it
  • Grading Timeline – Expect feedback by end of next week

Any final questions about the midterm project or peer review process?

Questions & Next Steps

Looking Ahead


Between now & Thursday: Chapter readings

Thursday Lab: More business scenario planning and problem framing practice

Future Weeks: Building and evaluating actual ML models in Python

Any Final Questions?

  • About today’s ML concepts?
  • About the difference between supervised and unsupervised learning?
  • About Thursday’s lab?
  • About the peer review process?

Office Hours & Resources

  • Office hours: Thursday labs act as office hours!
  • Thursday lab: Extended practice with business scenarios
  • Course resources: Additional readings on ML fundamentals
See you Thursday for hands-on practice with ML problem framing!