Interactive Session

9-Step Data Science Workflow

Every data science project follows the same systematic workflow. Whether you’re analyzing Netflix recommendations, climate research, social media trends, or working on your final project, you’ll use these 9 steps:

flowchart LR
    A["1. Import<br/>📂"] --> B["2. Explore<br/>🔍"] --> C["3. Clean<br/>🧹"]
    C --> D["4. Filter<br/>🎯"] --> E["5. Sort<br/>📊"]
    E --> F["6. Transform<br/>🔄"] --> G["7. Group<br/>👥"]
    G --> H["8. Aggregate<br/>📈"] --> I["9. Visualize<br/>📊"]

    style A fill:#e1f5fe
    style B fill:#e8f5e8
    style C fill:#fff3e0
    style D fill:#f3e5f5
    style E fill:#e0f2f1
    style F fill:#fce4ec
    style G fill:#e8eaf6
    style H fill:#f1f8e9
    style I fill:#fff8e1

Why This Workflow Matters

Today: See all 9 steps in action with ocean temperature analysis
Days 4-7: Master each step individually with detailed sessions
Your final project: Apply this exact workflow to answer your research question!

Course Integration

Almost all pandas functions and dataframe methods fit into one of these 9 categories. For reference, here is a cheatsheet that maps common pandas functions to our workflow steps.

Ocean Temperature Analysis: Complete Workflow Demo

In this session, we’ll systematically work through every step of the data science workflow using ocean temperature data. You’ll see exactly how professional data scientists approach problems, and by the end, you’ll have completed your first full data science project!

Research Question: Which ocean has the warmest average temperatures, and how do temperatures change between seasons?

Let’s systematically work through our 9-step workflow!

Setting up our environment

First, let’s import the libraries we know from previous sessions:

Code

import pandas as pd
import matplotlib.pyplot as plt

Libraries We’re Using

pandas (pd): For working with DataFrames (from Sessions 4a & 4b)
matplotlib (plt): For creating charts and graphs (from Session 4c)

Workflow Progress Tracker

As we work through each step, we’ll track our progress through the complete data science workflow:

Workflow Progress

Ocean Temperature Analysis - Workflow Steps

☐ Step 1: Import - Load our ocean data
☐ Step 2: Explore - Discover what we have
☐ Step 3: Clean - Fix any problems
☐ Step 4: Filter - Focus on specific data
☐ Step 5: Sort - Find temperature patterns
☐ Step 6: Transform - Create new insights
☐ Step 7: Group - Organize by categories
☐ Step 8: Aggregate - Calculate summaries
☐ Step 9: Visualize - Present our results

Goal: Complete systematic data science analysis

📂 Step 1: Import Data

✅ Workflow Step 1: Getting our data into Python

The first step in every data science project is getting your data into Python. We’ll use pd.read_csv() - the same function you learned in Session 4a!

Code

# Step 1: Import our ocean temperature data
df = pd.read_csv('ocean_temperatures_simple.csv')
print("✅ Step 1 Complete: Data imported successfully!")
print(f"📊 Loaded {len(df)} rows of ocean temperature data")

✅ Step 1 Complete: Data imported successfully!
📊 Loaded 30 rows of ocean temperature data

Real Data Science Connection

Professional data scientists start every project the same way - importing data! Whether it’s: - Climate data from NASA - User behavior from websites
- Financial data from banks - Your final project data

You always start with: pd.read_csv() or similar import functions

🔮 Coming Attractions: Later in the course, you’ll learn to import Excel files, JSON data, and even data from databases!

🔍 Step 2: Explore Data

✅ Workflow Step 2: Discovering what we have

Before we can analyze data, we need to understand what we’re working with. Let’s use the exploration methods you learned in Session 4a:

Code

print("🔍 EXPLORING OUR OCEAN DATA")
print("=" * 40)

print("\n📋 First few rows:")
print(df.head())

print(f"\n📊 DataFrame info:")
df.info()

print(f"\n📈 Summary statistics:")
print(df.describe())

print("\n❓ Missing values check:")
print(df.isna().sum())

print("\n✅ Step 2 Complete: We now understand our data!")

🔍 EXPLORING OUR OCEAN DATA
========================================

📋 First few rows:
         date  location  temperature  salinity  depth
0  2021-01-15   Pacific         18.5      34.2     50
1  2021-01-15  Atlantic         22.1      35.1      0
2  2021-01-15    Indian         20.0      34.8    100
3  2021-01-15  Southern         15.2      34.0    200
4  2021-01-15    Arctic         12.1      33.5     50

📊 DataFrame info:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30 entries, 0 to 29
Data columns (total 5 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   date         30 non-null     object 
 1   location     30 non-null     object 
 2   temperature  30 non-null     float64
 3   salinity     30 non-null     float64
 4   depth        30 non-null     int64  
dtypes: float64(2), int64(1), object(2)
memory usage: 1.3+ KB

📈 Summary statistics:
       temperature   salinity       depth
count    30.000000  30.000000   30.000000
mean     19.283333  34.433333   80.000000
std       4.621843   0.616068   68.982756
min      11.500000  33.300000    0.000000
25%      15.400000  34.025000   50.000000
50%      19.200000  34.350000   50.000000
75%      22.925000  35.000000  100.000000
max      27.100000  35.400000  200.000000

❓ Missing values check:
date           0
location       0
temperature    0
salinity       0
depth          0
dtype: int64

✅ Step 2 Complete: We now understand our data!

What We Discovered

Our ocean dataset contains: - 5 oceans: Pacific, Atlantic, Indian, Southern, Arctic - Temperature measurements in degrees Celsius
- Salinity measurements (salt content) - Depth measurements where samples were taken - 30 total measurements across different dates

This is exactly what real data scientists do first!

🔮 Coming Attractions: In Day 5, you’ll learn advanced exploration techniques like correlation analysis and custom statistics!

🧹 Step 3: Clean Data

✅ Workflow Step 3: Fixing problems in our data

Good news! Our ocean data is already clean - no missing values to worry about. But let’s see what cleaning looks like:

Code

print("🧹 CLEANING OUR DATA")
print("=" * 30)

# Check for missing values (we already did this, but let's confirm)
missing_data = df.isna().sum()
print("Missing values per column:")
print(missing_data)

if missing_data.sum() == 0:
    print("\n🎉 Great news! Our data is already clean!")
    df_cleaned = df.copy()  # Make a copy for consistency
else:
    print(f"\n🔧 Cleaning needed...")
    df_cleaned = df.dropna().copy()  # Remove rows with missing values
    print(f"Removed {len(df) - len(df_cleaned)} rows with missing data")

print(f"\n✅ Step 3 Complete: Clean dataset with {len(df_cleaned)} rows ready for analysis!")

🧹 CLEANING OUR DATA
==============================
Missing values per column:
date           0
location       0
temperature    0
salinity       0
depth          0
dtype: int64

🎉 Great news! Our data is already clean!

✅ Step 3 Complete: Clean dataset with 30 rows ready for analysis!

Why Cleaning Matters

In real data science projects, you’ll spend 50-80% of your time cleaning data! Common problems include: - Missing values (what we just checked for) - Duplicate entries - Incorrect data types - Outliers and errors

The .dropna() method you just learned will be one of your most-used tools!

🔮 Coming Attractions: In Day 5, you’ll learn advanced cleaning techniques like handling duplicates and fixing data types!

🎯 Step 4: Filter Data

✅ Workflow Step 4: Focusing on what matters for our question

Let’s focus on specific data to answer our research question. We’ll filter for just the Pacific Ocean to start:

Code

print("🎯 FILTERING OUR DATA")
print("=" * 30)

# Filter for just Pacific Ocean data (using boolean indexing from Session 4b)
pacific_data = df_cleaned[df_cleaned['location'] == 'Pacific']

print("Pacific Ocean measurements:")
print(pacific_data)
print(f"\n📊 Found {len(pacific_data)} Pacific Ocean measurements")

# Let's also look at summer data (June measurements)
summer_data = df_cleaned[df_cleaned['date'].str.contains('06-15')]
print(f"\n🌞 Summer measurements (June): {len(summer_data)} rows")

print("\n✅ Step 4 Complete: Focused on specific data for our analysis!")

🎯 FILTERING OUR DATA
==============================
Pacific Ocean measurements:
          date location  temperature  salinity  depth
0   2021-01-15  Pacific         18.5      34.2     50
5   2021-06-15  Pacific         24.3      34.5     50
10  2021-12-15  Pacific         19.1      34.3     50
15  2022-01-15  Pacific         18.2      34.1     50
20  2022-06-15  Pacific         24.8      34.6     50
25  2022-12-15  Pacific         19.3      34.4     50

📊 Found 6 Pacific Ocean measurements

🌞 Summer measurements (June): 10 rows

✅ Step 4 Complete: Focused on specific data for our analysis!

Filtering in Real Data Science

Professional data scientists constantly filter data to focus on specific questions: - Netflix: “Show me viewing data for comedy movies” - Climate research: “Focus on temperature data from Arctic regions”
- Your final project: “Filter for data relevant to your specific question”

The boolean indexing you just used (df[df['column'] == value]) is a fundamental skill!

🔮 Coming Attractions: In Day 5, you’ll learn complex filtering with multiple conditions using & and | operators!

📊 Step 5: Sort Data

✅ Workflow Step 5: Organizing data to find patterns

Sorting helps us find the highest and lowest values. Let’s find the warmest and coldest ocean measurements:

Code

print("📊 SORTING OUR DATA")
print("=" * 30)

# Sort by temperature (warmest first) using .sort_values() from Session 4b
sorted_by_temp = df_cleaned.sort_values('temperature', ascending=False)

print("🔥 TOP 5 WARMEST measurements:")
print(sorted_by_temp[['location', 'temperature', 'date']].head())

print("\n🧊 TOP 5 COLDEST measurements:")
print(sorted_by_temp[['location', 'temperature', 'date']].tail())

print("\n✅ Step 5 Complete: Found temperature patterns by sorting!")

📊 SORTING OUR DATA
==============================
🔥 TOP 5 WARMEST measurements:
    location  temperature        date
21  Atlantic         27.1  2022-06-15
6   Atlantic         26.8  2021-06-15
22    Indian         25.5  2022-06-15
7     Indian         25.2  2021-06-15
20   Pacific         24.8  2022-06-15

🧊 TOP 5 COLDEST measurements:
   location  temperature        date
9    Arctic         14.8  2021-06-15
4    Arctic         12.1  2021-01-15
19   Arctic         11.9  2022-01-15
29   Arctic         11.8  2022-12-15
14   Arctic         11.5  2021-12-15

✅ Step 5 Complete: Found temperature patterns by sorting!

Insights from Sorting

What we discovered: - 🔥 Warmest: Atlantic Ocean (27.1°C in summer) - 🧊 Coldest: Arctic Ocean (11.5°C in winter) - 📈 Pattern: Atlantic and Pacific are warmest, Arctic is coldest

This is how data scientists find patterns - sorting reveals extremes and trends!

🔮 Coming Attractions: In Day 6, you’ll learn to sort by multiple columns and create hierarchical sorting!

🔄 Step 6: Transform Data

✅ Workflow Step 6: Creating new insights from existing data

Let’s create new information that will help answer our research question:

Code

print("🔄 TRANSFORMING OUR DATA")
print("=" * 35)

# Create a new column: temperature in Fahrenheit (simple math from Session 4b)
df_cleaned['temperature_f'] = (df_cleaned['temperature'] * 9/5) + 32

# Create a season category based on the date
def get_season(date_str):
    if '01-15' in date_str or '12-15' in date_str:
        return 'Winter'
    elif '06-15' in date_str:
        return 'Summer'
    else:
        return 'Other'

df_cleaned['season'] = df_cleaned['date'].apply(get_season)

# Show our new columns
print("New columns added:")
print(df_cleaned[['location', 'temperature', 'temperature_f', 'season']].head())

print(f"\n📈 Original columns: 5")
print(f"📈 After transformation: {len(df_cleaned.columns)} columns")
print("\n✅ Step 6 Complete: Created new insights from our data!")

🔄 TRANSFORMING OUR DATA
===================================
New columns added:
   location  temperature  temperature_f  season
0   Pacific         18.5          65.30  Winter
1  Atlantic         22.1          71.78  Winter
2    Indian         20.0          68.00  Winter
3  Southern         15.2          59.36  Winter
4    Arctic         12.1          53.78  Winter

📈 Original columns: 5
📈 After transformation: 7 columns

✅ Step 6 Complete: Created new insights from our data!

Why Transform Data?

Transformation creates new insights: - 🌡️ Temperature in Fahrenheit: Makes data accessible to different audiences - 🗓️ Season categories: Helps us compare winter vs summer patterns - 📊 New calculations: Ratios, categories, derived metrics

Real data scientists spend lots of time creating these “feature engineering” transformations!

🔮 Coming Attractions: In Day 6, you’ll learn advanced transformations and custom functions!

👥 Step 7: Group Data

✅ Workflow Step 7: Organizing by categories to find patterns

Now we’ll group our data by categories to compare different oceans and seasons:

Code

print("👥 GROUPING OUR DATA")
print("=" * 30)

# Group by ocean location (using .groupby() from Session 4b)
by_ocean = df_cleaned.groupby('location')

print("📊 Number of measurements per ocean:")
print(by_ocean.size())

# Group by season to compare winter vs summer
by_season = df_cleaned.groupby('season')

print("\n📊 Number of measurements per season:")
print(by_season.size())

print("\n✅ Step 7 Complete: Data organized by meaningful categories!")

👥 GROUPING OUR DATA
==============================
📊 Number of measurements per ocean:
location
Arctic      6
Atlantic    6
Indian      6
Pacific     6
Southern    6
dtype: int64

📊 Number of measurements per season:
season
Summer    10
Winter    20
dtype: int64

✅ Step 7 Complete: Data organized by meaningful categories!

Why Group Data?

Grouping reveals patterns: - 🌊 By ocean: Compare Pacific vs Atlantic vs Arctic temperatures - 🗓️ By season: See how temperatures change winter to summer
- 📊 By categories: Any categorical variable can create groups

This sets up the next step - calculating summary statistics for each group!

🔮 Coming Attractions: In Day 6, you’ll learn to group by multiple columns simultaneously and create complex hierarchical groups!

📈 Step 8: Aggregate Data

✅ Workflow Step 8: Calculating summary statistics to answer our question

Now for the exciting part - let’s calculate averages to answer “Which ocean is warmest?”

Code

print("📈 AGGREGATING OUR DATA")
print("=" * 35)

# Calculate average temperature by ocean (using .mean() from Session 4b)
avg_temp_by_ocean = df_cleaned.groupby('location')['temperature'].mean()

print("🌊 AVERAGE TEMPERATURE BY OCEAN:")
print(avg_temp_by_ocean.sort_values(ascending=False))

# Calculate average temperature by season
avg_temp_by_season = df_cleaned.groupby('season')['temperature'].mean()

print("\n🗓️ AVERAGE TEMPERATURE BY SEASON:")
print(avg_temp_by_season.sort_values(ascending=False))

# Answer our research question!
warmest_ocean = avg_temp_by_ocean.max()
warmest_ocean_name = avg_temp_by_ocean.idxmax()

print(f"\n🎉 RESEARCH QUESTION ANSWERED!")
print(f"🏆 Warmest ocean: {warmest_ocean_name} ({warmest_ocean:.1f}°C)")

print("\n✅ Step 8 Complete: Found the answer through aggregation!")

📈 AGGREGATING OUR DATA
===================================
🌊 AVERAGE TEMPERATURE BY OCEAN:
location
Atlantic    24.083333
Indian      22.133333
Pacific     20.700000
Southern    16.616667
Arctic      12.883333
Name: temperature, dtype: float64

🗓️ AVERAGE TEMPERATURE BY SEASON:
season
Summer    22.100
Winter    17.875
Name: temperature, dtype: float64

🎉 RESEARCH QUESTION ANSWERED!
🏆 Warmest ocean: Atlantic (24.1°C)

✅ Step 8 Complete: Found the answer through aggregation!

Key Discovery!

Our Research Results: - 🥇 Warmest Ocean: Atlantic (24.1°C average) - 🥈 Second Warmest: Pacific (20.7°C average)
- 🥉 Coldest: Arctic (12.9°C average) - 🌞 Summer is warmer than winter (as expected!)

This is exactly how real data science works - use aggregation to answer research questions!

🔮 Coming Attractions: In Day 6, you’ll learn advanced aggregation functions like .agg() to calculate multiple statistics at once!

📊 Step 9: Visualize Data

✅ Workflow Step 9: Telling our story with charts

The final step is creating a chart to communicate our findings clearly:

Code

print("📊 VISUALIZING OUR RESULTS")
print("=" * 35)

# Create average temperature data for plotting
avg_temps = df_cleaned.groupby('location')['temperature'].mean().sort_values(ascending=False)

# Create a bar chart (using matplotlib from Session 4c)
plt.figure(figsize=(10, 6))
avg_temps.plot(kind='bar', color=['red', 'orange', 'blue', 'green', 'purple'])
plt.title('🌊 Average Ocean Temperatures: Research Results', fontsize=16, fontweight='bold')
plt.xlabel('Ocean Location', fontsize=12)
plt.ylabel('Average Temperature (°C)', fontsize=12)
plt.xticks(rotation=45)
plt.grid(True, alpha=0.3)
plt.tight_layout()

# Add our research conclusion to the plot
plt.figtext(0.5, 0.02, '🏆 Research Conclusion: Atlantic Ocean is the warmest on average!', 
            ha='center', fontsize=12, fontweight='bold')

plt.show()

print("\n✅ Step 9 Complete: Story told through visualization!")

📊 VISUALIZING OUR RESULTS
===================================

/var/folders/bs/x9tn9jz91cv6hb3q6p4djbmw0000gn/T/ipykernel_55814/1715597996.py:15: UserWarning: Glyph 127754 (\N{WATER WAVE}) missing from font(s) DejaVu Sans.
  plt.tight_layout()
/Users/kellycaylor/mambaforge/envs/eds217_2025/lib/python3.11/site-packages/IPython/core/pylabtools.py:170: UserWarning: Glyph 127942 (\N{TROPHY}) missing from font(s) DejaVu Sans.
  fig.canvas.print_figure(bytes_io, **kw)
/Users/kellycaylor/mambaforge/envs/eds217_2025/lib/python3.11/site-packages/IPython/core/pylabtools.py:170: UserWarning: Glyph 127754 (\N{WATER WAVE}) missing from font(s) DejaVu Sans.
  fig.canvas.print_figure(bytes_io, **kw)


✅ Step 9 Complete: Story told through visualization!

🎉 CONGRATULATIONS! 🎉

You just completed your first full data science project!

🌊 Research Question: Which ocean has the warmest average temperatures?
📊 Answer: Atlantic Ocean (24.1°C average)
🏆 Method: Complete 9-step data science workflow!

You’ve completed the full workflow - you’re officially a data scientist! 🎓

🎯 What You Accomplished Today

✅ Complete Workflow Mastery

You just used the exact same process that professional data scientists use every day:

✅ Imported real ocean temperature data
✅ Explored to understand what you had
✅ Cleaned (lucky us - data was already clean!)
✅ Filtered to focus on specific questions
✅ Sorted to find temperature patterns
✅ Transformed data to create new insights
✅ Grouped by meaningful categories
✅ Aggregated to calculate summary statistics
✅ Visualized results with a professional chart

🔮 Your Data Science Journey Continues

Next Week - Individual Step Mastery: - Day 5: Advanced filtering and transformation techniques - Day 6: Complex grouping and aggregation methods
- Day 7: Professional data visualization with seaborn

Your Final Project: Use this exact 9-step workflow to answer your own research question!

🔄 The Workflow You Can Always Apply

Whenever you encounter a new dataset or research question, systematically work through these 9 steps: 1. Import → 2. Explore → 3. Clean → 4. Filter → 5. Sort → 6. Transform → 7. Group → 8. Aggregate → 9. Visualize

This is your systematic approach to data science success! 🎯

🎉 End interactive session 4C - You’re now a data scientist! 🎉