A crowd of pandas, waiting to be sorted into piles. MidJourney 5
The past four days we have been asking questions about a whole table. Today we will learn to manipulate datasets to allow us to examine slices and subsets.
Day 5 is all about one pattern, and it is one that forms the basis for many, many data science analyses. It has a common name, split-apply-combine, which defines the steps necessary to break a table into groups, run a calculation on each group, and then summarize the results across each group. Almost every real question about data turns out to be that shape: not βwhat is the averageβ, but βwhat is the average for eachβ. This approach is similar to the even more general map-reduce pattern of data analysis, where the map step corresponds to split-apply, and reduce takes the place of combine. Generally, reduce encompasses a larger range of more sophisticated analyses than combine, and the map phase is implicitly structured around parallel processing workflows.
This afternoon you will address some questions the pattern cannot answer, and learn some older, slower tools that can handle them.
Class materials
| Session | Session 1 | Session 2 |
|---|---|---|
| day 5 / morning | ποΈ The Split-Apply-Combine Pattern | π Several Answers at Once |
| day 5 / afternoon | π The Comparison You Made Yesterday, Properly | π When a Pattern Is Not Enough |
End-of-day practice
Later today you will use a month of hourly air quality readings from three monitoring stations on the South Coast to explore patterns across the dataset and reveal a new result that depends on the split-apply-combine workflow we have learned today.