Live Coding Session

Instructor run-through for Session 5D. Student-facing page: 5d_loops_over_groups.qmd. Headings below match the markdown scaffold students set up, so everyone stays in the same place.

Total: 32 minutes of coding, 13 minutes of buffer and questions.

The three blocks on the student page are these sections added up:

Student page block	Sections	Minutes
Looping over groups	1, 2, 3, 4	15
Reading comprehensions	5, 6	10
Choosing	7	5

The student page lists those 30 minutes. The two framing minutes sit on top of them, which is why the coding total here is 32 rather than 30.

Scope discipline

Session 5D teaches iteration over a groupby object at adapt level: students should be able to modify a for name, group in grouped: loop, not design one from nothing. The end-of-day practice does not require a loop.

Comprehensions are read only. Sections 5 and 6 are a prediction exercise. Do not set an exercise that asks a student to write one, do not use one in an answer key, and if a student writes one unprompted, say it is correct and move on rather than teaching the syntax to the room.

There is no new aggregation in this session. Everything numeric here was taught this morning.

Do not use lambda anywhere. It is not in this course.

Framing (2 min)

Say something close to this:

This morning you learned a pattern that turns six piles of rows into six numbers. It is the right tool for almost every grouped question you will ask, and I want to be clear that nothing this afternoon replaces it.

But it does one thing. One number per group, arithmetic only. If what you want per group is a line of text, or a file, or a plot, or a yes-or-no check, the pattern cannot help you and you have to open the groups up and walk through them.

The good news is that you already know how. You wrote your first for loop on Tuesday.

Setup

Rebuild the clean survey table. Students have the cleaning code from 5A, so paste it rather than retyping it line by line, and say so.

Code

import pandas as pd

url = 'https://eds-217-essential-python.github.io/data/messy_field_survey.csv'
survey = pd.read_csv(url)

survey = survey.drop_duplicates()
survey['site'] = survey['site'].str.strip()
survey['site'] = survey['site'].str.lower()
survey['site'] = survey['site'].str.replace('-', '_')
survey['pH'] = survey['pH'].str.replace(',', '.')
survey['pH'] = survey['pH'].astype(float)
survey = survey.dropna(subset=['temperature_c', 'dissolved_oxygen_mg_L', 'conductivity_uS_cm'])
survey['n_replicates'] = survey['n_replicates'].fillna(1)
survey['n_replicates'] = survey['n_replicates'].astype(int)
survey = survey[survey['temperature_c'] > -100].copy()
survey = survey.rename(columns={'collection date': 'collection_date'})

survey.shape

(255, 7)

1. What groupby is holding (2 min)

Make the split object again and refuse to aggregate it, on purpose:

Code

grouped = survey.groupby('site')
grouped

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x118954290>

This morning we said this object is holding six piles of rows and waiting. Let us actually look at the piles.

Code

grouped.ngroups

Code

list(grouped.groups.keys())

['site_a', 'site_b', 'site_c', 'site_d', 'site_e', 'site_f']

Points to make:

.ngroups is how many piles
.groups is a dictionary. The keys are the group labels, the values are the row labels in each pile. This is the dictionary from Tuesday, doing a real job.
Say out loud: the split has already been done. It is sitting there.

Pull one pile out by name:

Code

site_c = grouped.get_group('site_c')
site_c.shape

(45, 7)

That is a DataFrame. Forty-five rows, seven columns, all the same columns as survey. Hold on to the idea that one group is a whole DataFrame, because it is what the next section is built on.

2. Looping over the groups (5 min)

The loop pattern. Write the two-variable version first and name both variables out loud, then run it:

Code

for name, group in grouped:
    print(name, group.shape)

site_a (44, 7)
site_b (41, 7)
site_c (45, 7)
site_d (39, 7)
site_e (43, 7)
site_f (43, 7)

Points to make, slowly, because this is the moment the session either works or does not:

name is the group label, one of the six site strings
group is the DataFrame of rows in that pile
the loop body runs six times, once per pile
the order is the sorted order of the keys, which is why site_a comes first

Compare it directly to Tuesday’s loop over a list. Put both on the screen:

Code

sites = ['site_a', 'site_b', 'site_c']

for s in sites:
    print(s)

site_a
site_b
site_c

One variable when you loop over a list, two when you loop over a groupby. The reason is that each item in a groupby has two halves: what it is called, and what is in it.

The classic error, run it on purpose:

for group in grouped:
    print(group.shape)

AttributeError: 'tuple' object has no attribute 'shape'

Ask them to guess why before you explain. With one variable, group catches the pair, not the DataFrame.

Now something the pattern cannot do. A formatted report line per site:

Code

for name, group in grouped:
    print(f"{name}: {len(group)} samples, mean DO {group['dissolved_oxygen_mg_L'].mean():.2f} mg/L")

site_a: 44 samples, mean DO 9.49 mg/L
site_b: 41 samples, mean DO 8.60 mg/L
site_c: 45 samples, mean DO 6.87 mg/L
site_d: 39 samples, mean DO 10.05 mg/L
site_e: 43 samples, mean DO 7.41 mg/L
site_f: 43 samples, mean DO 6.18 mg/L

Six lines of prose. .groupby('site')['dissolved_oxygen_mg_L'].mean() cannot produce those, because it can only produce numbers. This is the first thing today that genuinely needs the loop.

f-strings, if they ask

f"{value:.2f}" rounds to two decimal places inside the string. Students met f-strings on Day 1. If nobody asks, do not stop for it.

3. One group is a DataFrame (4 min)

Hammer the point from section 1. Inside the loop, group is a real DataFrame and every method you know works on it:

Code

for name, group in grouped:
    warm = group[group['temperature_c'] > 20]
    print(name, len(warm), 'of', len(group), 'samples above 20 C')

site_a 1 of 44 samples above 20 C
site_b 5 of 41 samples above 20 C
site_c 42 of 45 samples above 20 C
site_d 0 of 39 samples above 20 C
site_e 20 of 43 samples above 20 C
site_f 43 of 43 samples above 20 C

That is the filter pattern from Wednesday, running inside a loop, six times. Nothing new. The loop just handed you a smaller table each time round.

One more, to show it is not only filtering:

Code

for name, group in grouped:
    hottest = group['temperature_c'].idxmax()
    print(name, 'warmest on', group.loc[hottest, 'collection_date'])

site_a warmest on 2025-06-06
site_b warmest on 2025-07-08
site_c warmest on 2025-08-29
site_d warmest on 2025-06-30
site_e warmest on 2025-08-03
site_f warmest on 2025-09-02

Points to make:

.idxmax() and .loc[] are Wednesday’s tools, unchanged
the answer here is a date, not a number, and a grouped aggregation cannot return a date

If you are running behind

Cut this second example. Section 4 matters more.

4. Building something up as you go (4 min)

The other reason to loop: you want to collect results rather than print them.

Start with the empty list from Tuesday and .append():

Code

records = []

for name, group in grouped:
    records.append({
        'site': name,
        'n': len(group),
        'mean_do': group['dissolved_oxygen_mg_L'].mean()
    })

records[0]

{'site': 'site_a', 'n': 44, 'mean_do': np.float64(9.485909090909091)}

A list of dictionaries. Both containers from Tuesday, in one place, doing the thing they are for.

And then the payoff, which lands well:

Code

pd.DataFrame(records)

	site	n	mean_do
0	site_a	44	9.485909
1	site_b	41	8.601220
2	site_c	45	6.873111
3	site_d	39	10.053333
4	site_e	43	7.410930
5	site_f	43	6.180233

That is a DataFrame you built yourself, row by row, out of Python containers. Day 2 you read one in from a file. Today you made one.

Now the honest comparison. Put the one-liner underneath it:

Code

survey.groupby('site')['dissolved_oxygen_mg_L'].agg(['count', 'mean'])

	count	mean
site
site_a	44	9.485909
site_b	41	8.601220
site_c	45	6.873111
site_d	39	10.053333
site_e	43	7.410930
site_f	43	6.180233

Same table. Nine lines versus one. Use the one line. The loop is for when the thing you want in each row is not something .agg() can compute.

A dictionary instead of a list, because they will see both:

Code

means = {}

for name, group in grouped:
    means[name] = group['dissolved_oxygen_mg_L'].mean()

means

{'site_a': np.float64(9.485909090909091),
 'site_b': np.float64(8.60121951219512),
 'site_c': np.float64(6.87311111111111),
 'site_d': np.float64(10.053333333333333),
 'site_e': np.float64(7.41093023255814),
 'site_f': np.float64(6.180232558139535)}

5. Reading a list comprehension (5 min)

Transition explicitly:

The square-bracket shorthand you have probably seen in other people’s code has a name, and this is the part of the course where we read some. You are not going to write these. You are going to be able to look at one and say what it does, because they are everywhere.

Start from a loop that builds a list, written the long way:

Code

site_names = []

for name, group in grouped:
    site_names.append(name.upper())

site_names

['SITE_A', 'SITE_B', 'SITE_C', 'SITE_D', 'SITE_E', 'SITE_F']

Then the same thing, one line:

Code

site_names = [name.upper() for name, group in grouped]
site_names

['SITE_A', 'SITE_B', 'SITE_C', 'SITE_D', 'SITE_E', 'SITE_F']

How to teach the reading. Put this on the board:

[  what to keep     for  each thing  in  the collection  ]

Then read the line out loud in that order, pointing: “name.upper(), for each name and group, in grouped.” Say that the loop body moved to the front, and that is the only real difference.

Do a plain one over a list so the groupby is not a distraction:

Code

temps_c = [12.1, 18.4, 21.9, 7.2]

temps_f = [c * 9 / 5 + 32 for c in temps_c]
temps_f

[53.78, 65.12, 71.42, 44.96]

Prediction drill. Write this on the screen and get an answer from the room before running it:

Code

[len(s) for s in ['site_a', 'bb', 'c']]

[6, 2, 1]

Expect [6, 2, 1]. If more than a couple of people miss it, do one more; if not, move on.

6. Comprehensions you will meet in the wild (5 min)

Three patterns, read only, one prediction each.

A dictionary comprehension. Curly braces, and a key: value pair at the front:

Code

{name: len(group) for name, group in grouped}

{'site_a': 44,
 'site_b': 41,
 'site_c': 45,
 'site_d': 39,
 'site_e': 43,
 'site_f': 43}

Same six numbers as .value_counts(). Ask them which they would rather read.

A comprehension with a filter. The if goes at the end:

Code

[name for name, group in grouped if len(group) > 42]

['site_a', 'site_c', 'site_e', 'site_f']

Predict first. Point at the counts from section 2 while they think.

A comprehension over a DataFrame column, which is the one they are most likely to meet in inherited code:

Code

[c.upper() for c in survey.columns]

['SITE',
 'COLLECTION_DATE',
 'TEMPERATURE_C',
 'PH',
 'DISSOLVED_OXYGEN_MG_L',
 'CONDUCTIVITY_US_CM',
 'N_REPLICATES']

Then, immediately, the pandas way to do the same job:

Code

survey.columns.str.upper()

Index(['SITE', 'COLLECTION_DATE', 'TEMPERATURE_C', 'PH',
       'DISSOLVED_OXYGEN_MG_L', 'CONDUCTIVITY_US_CM', 'N_REPLICATES'],
      dtype='object')

Both are correct. The second is the one to write, because pandas is faster at it and because the first stops being readable the moment the expression gets long.

Do not go further than this

Nested comprehensions, comprehensions with else, generator expressions and lambda are all out of scope. If a student raises one, confirm it exists, point at the cheatsheet, and move on. The goal of these ten minutes is recognition, not fluency.

7. Which one should I reach for? (5 min)

Build this table on the board with the room, rather than showing it finished:

What you want per group	Reach for
one number	`.groupby()...agg()`
several numbers	`.agg(['count', 'mean'])`
a formatted line of text	a `for` loop
a saved file, a plot, a check that passes or fails	a `for` loop
a small list built from something simple	a comprehension, if you like them

Close with the three questions:

Before you write a loop over groups, ask: is the answer a number? If yes, use the pattern. Is it one number per group? If yes, definitely use the pattern. Would the loop body be one line of arithmetic? If yes, you are writing nine lines to avoid learning one.

Loop when the answer is not a number. That is nearly the whole rule.

Timing checkpoints

Elapsed	You should be at
0:02	framing done, setup cell running
0:09	end of section 2, the two-variable loop is on everyone’s screen
0:17	end of section 4, the `pd.DataFrame(records)` payoff
0:22	end of section 5, first prediction drill done
0:32	end of section 7
0:32+	questions, and the trailing exercise below if there is appetite

If you are at 0:20 and not yet at section 5, cut section 3’s second example and the dictionary at the end of section 4. Sections 5 and 6 are the ten minutes of reading the student page promises in print, so they cannot be the ones you drop.

If you finish before 0:32, ask the room to write the loop from section 3 for a different threshold, or to predict {name: group['pH'].max() for name, group in grouped} before you run it.

What the end-of-day practice needs from this session

The end-of-day exercise is entirely .groupby() patterns. No loop and no comprehension appears in it. If a student asks whether they should be looping over the OpenAQ stations, the answer is no, and the reason is section 7: every answer the practice needs is a number, one per group.

Framing (2 min)

Setup

1. What groupby is holding (2 min)

2. Looping over the groups (5 min)

3. One group is a DataFrame (4 min)

4. Building something up as you go (4 min)

5. Reading a list comprehension (5 min)

6. Comprehensions you will meet in the wild (5 min)

7. Which one should I reach for? (5 min)

Timing checkpoints

What the end-of-day practice needs from this session

Resources