Coding Colab

Two distributions, being distributions. Artwork by Allison Horst

Between 2007 and 2009, researchers at the Palmer Station on the Antarctic Peninsula measured 344 penguins of three species: Adélie, Chinstrap and Gentoo. For each bird they recorded four numbers, the island it was caught on, and its sex.

The file is small, clean by the standards of anything you have opened this week, and it contains a result that you cannot get to without a figure. Not a result that is easier with a figure. One that a table of summary statistics will actively point you away from.

Your job this afternoon is to find it.

Work in pairs, in one shared notebook, taking turns at the keyboard. Swap every time you finish a numbered task. The person not typing says out loud what they expect the figure to look like before the cell is run, and is allowed to be wrong.

You have 45 minutes.

The sentences you need

All from this morning.

sns.scatterplot(data=df, x='col', y='col')             # one measurement against another
sns.histplot(data=df, x='col')                         # the distribution of one column
sns.barplot(x=series.values, y=series.index)           # a grouped Series, as bars

hue='col'                                              # works on all three

Plus the matplotlib frame:

plt.figure(figsize=(w, h))
plt.xlabel(...) ; plt.ylabel(...) ; plt.title(...)
plt.tight_layout()

And these, from earlier in the week:

df.dropna()
df.groupby('key')['column'].agg(['count', 'mean'])
df['column'].value_counts()
df.sort_values('column', ascending=False)

Setup

Create a notebook named Colab_7C_Penguins.ipynb, with both partners’ names in the title cell, then read the file:

Code

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

url = 'https://eds-217-essential-python.github.io/data/penguins.csv'
penguins = pd.read_csv(url)

Part 1: What is in the file (about 8 minutes)

Answer each question with code, then write the answer in a markdown cell underneath, in a complete sentence with the numbers in it.

How many rows and columns? What are the column names, and which are numeric?
Count the rows for each species, and separately for each island. Which species is rarest, and which island has the most birds?
Run .isnull().sum(). Two different things are going on in that output. Say what each one probably is.
Build a clean table called birds by dropping every row with any missing value, and report how many rows you lost. Then re-run the species counts on birds and say whether the losses fell evenly across the three species.

Part 2: Distributions (about 8 minutes)

Draw a histogram of flipper_length_mm for the whole table. Label the x-axis with units and give the figure a title. Describe the shape in one sentence.
Draw it again, split by species. In a markdown cell, say what the two humps in question 5 turned out to be, and whether “the average penguin has a flipper of about 201 mm” is a useful sentence.
Pick one of bill_length_mm, bill_depth_mm or body_mass_g and do the same pair of figures for it. Does that measurement separate the species as cleanly as flipper length does?

Part 3: Relationships (about 15 minutes)

This is the part with the result in it. Do not skip ahead to the answer key, and do not skip question 9.

Draw a scatter plot of body_mass_g against flipper_length_mm, with flipper length across the bottom. Label both axes and title it. Describe the relationship in one sentence.
Draw the same figure with hue='species'. Does adding the species change what you would say about the relationship, or just make it prettier? One sentence.
Now a different pair of columns. Draw a scatter plot of bill_depth_mm against bill_length_mm, with bill length across the bottom, and no hue=. Label it.

Before you go on, write down in a markdown cell what this figure says: as a penguin’s bill gets longer, does it get deeper or shallower?
Draw it once more with hue='species'.
Read your answer to question 10 back, then look at question 11. Within each of the three species, longer bills go with deeper bills. Across all three together, longer bills go with shallower ones.

In a markdown cell of four or five sentences: explain how both of those can be true at once. Your explanation should mention where the three species sit relative to each other, and it should not use the word “wrong” about either figure.

This has a name

A relationship that runs one way inside every group and the other way across the pooled data is called Simpson’s paradox. It is not a curiosity. It is one of the most common ways that a perfectly correct summary of a dataset produces a conclusion that is the reverse of the truth, and it happens whenever the groups differ both in what you are measuring and in how many of them there are.

You found it here in one argument, hue='species'. Without the figure you would have needed to suspect it first.

Part 4: Summaries, and what they hide (about 10 minutes)

Build a Series of mean body_mass_g by species, sorted from heaviest to lightest, and draw it as horizontal bars using the .values and .index idiom. Label both axes and title it.
Two of those three bars are nearly the same height. Look back at your figure from question 11. In one sentence: would you say those two species are similar birds?
Run .agg(['count', 'mean']) on the same grouping and print it. Then build a Series of the number of birds on each island with .value_counts(), and draw that as horizontal bars too.
Draw a bar chart of mean body_mass_g by species, split by sex. This one is easier with the data= form, because you want two bars per species: sns.barplot(data=birds, x='species', y='body_mass_g', hue='sex').

Then say in one sentence what the sex split adds to your answer to question 14.
Last one, and it is a trap worth walking into. Build a wide table with species down the rows, island across the columns, and a count of body_mass_g in the cells. This is yesterday morning’s pivot sentence with aggfunc='count'.

In a markdown cell, two or three sentences: given that table, what would go wrong if somebody used this dataset to compare Biscoe island with Dream island?

Before you stop

Check that:

every figure you made has an x-label, a y-label and a title, and the labels carry units
you can say out loud what hue= did in each of the three kinds of plot you used it in
your answer to question 12 does not describe either figure as wrong
your notebook reads top to bottom as a document, not as a pile of cells