Interactive Session

πŸ“† A Date Is Not a Number

A cartoon panda is visiting Stonehenge. MidJourney 5

You owe two lines of code an explanation.

On Thursday, an exercise handed you .dt.year inside a boxed aside and told you not to worry about it. Yesterday, another one handed you aq['datetimeLocal'].str[11:13] and told you the same thing. Both were doing the same job by different means: pulling a piece out of a date so you could group by it.

This session is where that stops being a borrowed line.

The reason dates get their own hour is not that the syntax is hard. It is that a date is one of the few things in a data file that is almost never stored as what it is. It arrives as text, or as an integer, or as five separate columns, and every one of those disguises works well enough to fool you until the moment it doesn’t.

By the end of this session you will be able to:

Getting Started

  1. Create the file. In the Explorer, hover over the EDS217 heading and click New File…, then type the name in full, extension included: Session_6C_Dates.ipynb

  2. Check the kernel. The Kernel Selector in the notebook’s action bar should read Python 3.11.15 (Conda: eds217). If it reads anything else, click it, choose Change Kernel, and pick the eds217 entry.

  3. Add a title cell. Click + Markdown in the action bar, and give it this content, with today’s date:

# Day 6: Session 6C - A Date Is Not a Number

[Session Webpage](https://eds-217-essential-python.github.io/course-materials/interactive-sessions/6c_dates.html)

Date: 09/08/2026

  1. Save with Ctrl + S (Cmd + S on macOS), and keep saving as you go.

  2. Read in a file you first met on the very first afternoon of this course:

Code 
import pandas as pd

url = 'https://eds-217-essential-python.github.io/data/toolik_weather.csv'
toolik = pd.read_csv(url)

toolik[['Year', 'Month', 'Date', 'Daily_AirTemp_Mean_C']].head()
Year Month Date Daily_AirTemp_Mean_C
0 1988 6 19880601 8.4
1 1988 6 19880602 6.0
2 1988 6 19880603 5.8
3 1988 6 19880604 1.8
4 1988 6 19880605 6.8

Daily weather from the Toolik Field Station on the North Slope of Alaska. One row per day, 11,171 of them.

The disguise

Look at the Date column, and then at what pandas thinks it is:

Code 
toolik['Date'].dtype
dtype('int64')

int64. The first row’s date is not the first of June 1988; it is the integer nineteen million, eight hundred and eighty thousand, six hundred and one.

That works better than it has any right to. Sorting by it gives the right order, because %Y%m%d happens to sort the same way as text does. Filtering to a year works, because 19880000 < Date < 19890000 catches exactly 1988. You could get quite a long way like this.

Then you subtract two of them:

Code 
19880701 - 19880630
71

Seventy-one. The thirtieth of June and the first of July are one day apart, and the integer says seventy-one, because integers do not know that months end.

Everything else that makes a date useful fails the same way. Which day of the week was it? How many days between sampling visits? Is this row in the growing season? None of those questions can be answered by arithmetic on 19880601, and none of them will raise an error when you try. They will just be wrong.

The parsing sentence

pd.to_datetime() takes a column of dates-in-disguise and hands back a column of real dates. You tell it the disguise with format=:

Code 
toolik['date'] = pd.to_datetime(toolik['Date'], format='%Y%m%d')

toolik[['Date', 'date']].head()
Date date
0 19880601 1988-06-01
1 19880602 1988-06-02
2 19880603 1988-06-03
3 19880604 1988-06-04
4 19880605 1988-06-05 
pd.to_datetime(column, format='%Y%m%d')
#              ↑               ↑
#              what to parse   how it is laid out

The format string is a picture of the text you have. %Y stands where the four-digit year sits, %m where the two-digit month sits, %d where the two-digit day sits, and any punctuation in between is typed literally.

Your data looks like Your format string is
19880601 '%Y%m%d'
1988-06-01 '%Y-%m-%d'
06/01/1988 '%m/%d/%Y'
01/06/1988 '%d/%m/%Y'

Look at the last two rows for as long as it takes to be alarmed. The same eight characters mean the first of June in most of the world and the sixth of January in the United States, and the only thing standing between those two readings is the format string you typed. Half of all date bugs in data science live in that one row of the table.

format= is not optional, it is a safety net

pd.to_datetime() will usually guess correctly if you leave format= out. Supply it anyway.

A format string is an assertion about your data. If one row of your file is malformed, or the first five hundred rows are ISO dates and the rest are American ones, a supplied format raises an error and a guessed format silently produces a column of confidently wrong dates.

Loud failure now beats quiet nonsense later. Type the format.

Check what you got:

Code 
print(toolik['date'].dtype)
print(toolik['date'].min())
print(toolik['date'].max())
datetime64[ns]
1988-06-01 00:00:00
2018-12-31 00:00:00

datetime64[ns], running from 1 June 1988 to 31 December 2018. Thirty years of Arctic weather, and now pandas knows it.

✏️ Test your knowledge

The date_data.csv file at https://eds-217-essential-python.github.io/data/date_data.csv has a Date column stored as text in the form 2023-11-02. Read it in, write the parsing sentence for it with the correct format string, and print the dtype of the result to prove it worked.

The .dt accessors

A parsed date column knows what year, month and day each of its values belongs to, and .dt is how you ask:

Code 
toolik['year'] = toolik['date'].dt.year
toolik['month'] = toolik['date'].dt.month
toolik['day'] = toolik['date'].dt.day

toolik[['date', 'year', 'month', 'day']].head()
date year month day
0 1988-06-01 1988 6 1
1 1988-06-02 1988 6 2
2 1988-06-03 1988 6 3
3 1988-06-04 1988 6 4
4 1988-06-05 1988 6 5

.dt works the same way .str does. Both are doorways: .str gives you the string operations for a column of text, and .dt gives you the date operations for a column of dates. Neither one works on the wrong kind of column, which is a feature. If .dt.year raises an AttributeError, your column is not parsed yet.

Note

🐍 Other things .dt knows, for when you need them. You are only responsible for the first three in this course:

.dt.year Β· .dt.month Β· .dt.day Β· .dt.hour Β· .dt.dayofyear Β· .dt.dayofweek Β· .dt.quarter Β· .dt.date

Proving it worked

This file is unusually generous: it already has its own Year and Month columns, written by whoever prepared the data. So you can check your parse against theirs.

Code 
print((toolik['year'] == toolik['Year']).all())
print((toolik['month'] == toolik['Month']).all())
True
True

Two Trues. Every one of the 11,171 dates you parsed agrees with the year and month the station recorded independently.

That check took two lines and it is worth doing every time you have anything to check against, because a wrong format string produces a column that looks completely normal. '%d%m%Y' on this file would have failed loudly, but on a file of American dates it would have quietly given you a year of 2019 for every row and never said a word.

✏️ Test your knowledge

Add a dayofyear column to toolik using the .dt accessor for it. Then use the filter pattern to show the rows where dayofyear is 366, and say in one sentence what those rows have in common.

Dates as grouping keys

Here is what all of this was for. A date component is an ordinary column, so it is an ordinary grouping key, and Friday’s sentence works on it unchanged:

Code 
toolik.groupby('month')['Daily_AirTemp_Mean_C'].mean().round(2)
month
1    -22.89
2    -20.70
3    -20.69
4    -11.76
5     -0.80
6      8.59
7     11.22
8      7.23
9     -0.11
10   -10.54
11   -18.34
12   -21.44
Name: Daily_AirTemp_Mean_C, dtype: float64

Thirty years of daily temperatures reduced to the twelve numbers that describe the Arctic year. January at βˆ’22.9 Β°C, July at +11.2 Β°C, and a thirty-four degree swing between them.

Ask for the count alongside it, the way you now always do:

Code 
toolik.groupby('year')['Daily_AirTemp_Mean_C'].agg(['count', 'mean']).head(3).round(2)
count mean
year
1988 214 -4.88
1989 365 -7.94
1990 365 -8.54

1988 has 214 rows because the station opened on the first of June. Its annual mean is warmer than every year that follows it, not because 1988 was warm but because 1988 has no January in it. That is the .count() lesson from Friday, arriving in a new costume.

✏️ Test your knowledge

Group toolik by month and take the mean of Daily_Precip_Total_mm instead of temperature. Which three months get the most precipitation? Then check .isnull().sum() on that column and say whether it changes how much you trust the answer.

The question that needs a month column

Toolik has thirty years of data, and the obvious question to ask of thirty years of Arctic weather is whether it is getting warmer.

Split the record into its first eleven years and its last ten, using Wednesday’s filter pattern:

Code 
early = toolik[toolik['year'] <= 1998]
late = toolik[toolik['year'] >= 2009]

print(early.shape)
print(late.shape)
(3866, 25)
(3652, 25)

Then ask each half the same question, and put the two answers side by side:

Code 
comparison = pd.DataFrame({
    'early': early.groupby('month')['Daily_AirTemp_Mean_C'].mean(),
    'late': late.groupby('month')['Daily_AirTemp_Mean_C'].mean(),
})

comparison['change'] = comparison['late'] - comparison['early']

comparison.round(2)
early late change
month
1 -24.72 -21.26 3.46
2 -21.86 -19.38 2.48
3 -18.62 -20.64 -2.02
4 -10.40 -11.86 -1.45
5 0.37 -0.54 -0.91
6 8.43 8.19 -0.24
7 12.05 11.28 -0.77
8 7.48 7.18 -0.30
9 -0.85 -0.01 0.84
10 -12.26 -7.99 4.26
11 -19.84 -17.47 2.37
12 -22.32 -21.12 1.20

Read the change column top to bottom.

January is 3.5 Β°C warmer. October is 4.3 Β°C warmer. February, November and December are all up by one to two and a half degrees. And July, the month everybody thinks of when they think about warming, is 0.8 Β°C cooler, as are June, August and May.

If you had computed a single annual mean for each half of the record you would have got a small positive number and concluded, correctly but uselessly, that Toolik has warmed a little. The month column is what turns that into the actual finding: this site’s warming is almost entirely a cold-season phenomenon. That is a well-documented Arctic pattern, and it has physical causes you can reason about: snow arriving later, sea ice forming later, and both of them changing how much heat the land loses in the dark half of the year.

You could not have seen it without a month, you did not have a month until you parsed the date, and the file had been sitting there the whole time with 19880601 in it.

Note

🐍 Two honest caveats, because this is a real result and real results have them. Eleven years and ten years are short records for a climate claim, and one station is one station. What you have found is a pattern in this dataset, which is the correct thing to say about it.

Key points

  • A date stored as text or as an integer is not a date. It will sort correctly and then fail at arithmetic, without complaining.
  • The parsing sentence is pd.to_datetime(column, format='...').
  • A format string is a picture of your data: %Y for a four-digit year, %m for a two-digit month, %d for a two-digit day, punctuation typed literally.
  • Always supply format=. It converts a silent wrong answer into a loud error.
  • .dt is to dates what .str is to text. .dt.year, .dt.month, .dt.day.
  • If .dt raises an AttributeError, the column has not been parsed.
  • Check your parse against anything you can: another column, a known date range, a row count.
  • Date components are ordinary columns, so they are ordinary grouping keys, and that is usually the point of extracting them.

Resources