Code
import pandas as pd
url = 'https://eds-217-essential-python.github.io/data/openaq_goleta_measurments.csv'
goleta = pd.read_csv(url)
goleta.shape(1962, 15)
A panda, sorting the wheat from the chaff. MidJourney 5
An hour ago you wrote a question and asked it of an entire column. What came back was a boolean mask: a True or a False for every row.
A mask by itself is not very interesting. What you actually want is the rows it says True about. Getting them takes one line, and that line is the most useful thing you will learn this week.
By the end of this session you will be able to:
- write the filter sentence,
df[df['col'] > value], without looking it up - filter on text as well as on numbers
- combine two conditions with
&and|, and get the parentheses right - invert a condition with
~ - match against a list of values with
.isin() - say what
.copy()is for, and use it when you intend to keep a result
Getting Started
Create a new notebook:
- Open the Command Palette with
Ctrl + Shift + P(Cmd + Shift + Pon macOS) and run Create: New Jupyter Notebook. - Confirm the Kernel Selector in the notebook’s action bar reads eds217_2026.
- Open the Command Palette with
Save your notebook (
Ctrl + S, orCmd + Son macOS) as:Session_3B_Filtering_Data.ipynbAdd a title cell (Markdown), updating the date to today:
# Day 3: Session 3B - The Filter Sentence
[Session Webpage](https://eds-217-essential-python.github.io/course-materials/interactive-sessions/3b_filtering_data.html)
Date: 09/02/2026- Read in the Goleta air quality data you explored yesterday:
Caution
Save your work frequently with Ctrl+S (Cmd+S on macOS).
The sentence
Here it is, whole:
Code
goleta[goleta['value'] > 20]| location_id | location_name | parameter | value | unit | datetimeUtc | datetimeLocal | timezone | latitude | longitude | country_iso | isMobile | isMonitor | owner_name | provider | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 720 | 1186 | Goleta | pm10 | 21.0 | µg/m³ | 2024-07-17T14:00:00+00:00 | 2024-07-17T07:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 746 | 1186 | Goleta | pm10 | 21.0 | µg/m³ | 2024-07-22T20:00:00+00:00 | 2024-07-22T13:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 795 | 1186 | Goleta | pm10 | 38.0 | µg/m³ | 2024-07-24T21:00:00+00:00 | 2024-07-24T14:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 796 | 1186 | Goleta | pm10 | 36.0 | µg/m³ | 2024-07-24T22:00:00+00:00 | 2024-07-24T15:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 797 | 1186 | Goleta | pm10 | 23.0 | µg/m³ | 2024-07-24T23:00:00+00:00 | 2024-07-24T16:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1226 | 1186 | Goleta | pm10 | 27.0 | µg/m³ | 2024-08-11T23:00:00+00:00 | 2024-08-11T16:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1227 | 1186 | Goleta | pm10 | 23.0 | µg/m³ | 2024-08-12T00:00:00+00:00 | 2024-08-11T17:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1376 | 1186 | Goleta | pm25 | 22.0 | µg/m³ | 2024-07-18T08:00:00+00:00 | 2024-07-18T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1712 | 1186 | Goleta | pm25 | 22.0 | µg/m³ | 2024-08-01T08:00:00+00:00 | 2024-08-01T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1873 | 1186 | Goleta | pm25 | 21.0 | µg/m³ | 2024-08-08T08:00:00+00:00 | 2024-08-08T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
115 rows × 15 columns
Read it from the inside out. goleta['value'] > 20 is the mask you learned to write this morning: True on every row where the value is above 20, False everywhere else. Wrapping it in goleta[ ... ] says “give me the rows where that was True.”
That is the filter sentence. The general form is:
df[df['column'] <comparison> value]The df appears twice, and that trips people up at first. The inner one builds the question. The outer one applies the answer. They are the same table both times.
Assign it a name if you want to use the result:
Code
high_readings = goleta[goleta['value'] > 20]
high_readings.shape(115, 15)115 rows out of 1,962. Compare the shapes and you can see the filter did something:
Code
print('before:', goleta.shape)
print('after: ', high_readings.shape)before: (1962, 15)
after: (115, 15)
Note
🐍 Filtering never changes the original. goleta still has all 1,962 rows. What you get back is a new, smaller table, and if you don’t give it a name it is displayed and then discarded. This is the same rule you met with .rename() yesterday.
✏️ Test your knowledge
Filter goleta to the rows where value is below zero. How many are there? A concentration below zero is not physically possible, so make a note of what you find; we will deal with values like these on Day 4.
Filtering on text
The same sentence works on text columns. Use == for an exact match:
Code
ozone = goleta[goleta['parameter'] == 'o3']
ozone.head()| location_id | location_name | parameter | value | unit | datetimeUtc | datetimeLocal | timezone | latitude | longitude | country_iso | isMobile | isMonitor | owner_name | provider | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1186 | Goleta | o3 | 0.025 | ppm | 2024-07-12T01:00:00+00:00 | 2024-07-11T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1 | 1186 | Goleta | o3 | 0.028 | ppm | 2024-07-12T02:00:00+00:00 | 2024-07-11T19:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 2 | 1186 | Goleta | o3 | 0.029 | ppm | 2024-07-12T03:00:00+00:00 | 2024-07-11T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 3 | 1186 | Goleta | o3 | 0.027 | ppm | 2024-07-12T04:00:00+00:00 | 2024-07-11T21:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 4 | 1186 | Goleta | o3 | 0.026 | ppm | 2024-07-12T05:00:00+00:00 | 2024-07-11T22:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
Code
ozone.shape(711, 15)711 ozone readings. Text comparisons are exact and case-sensitive, so 'O3' would have matched nothing at all:
Code
goleta[goleta['parameter'] == 'O3'].shape(0, 15)An empty result is not an error. It is an answer, and it is usually a spelling problem.
✏️ Test your knowledge
Filter goleta to the pm25 readings and store them in a variable called pm25. How many rows? Check your number against goleta['parameter'].value_counts() from yesterday.
Why this matters here
Yesterday you found that goleta['value'] stacks three different pollutants, in two different units, in one column. That made .describe() on the whole column meaningless.
Filtering is the fix:
Code
goleta['value'].describe()count 1962.000000
mean 6.378173
std 7.313763
min -4.000000
25% 0.027000
50% 5.000000
75% 10.000000
max 40.000000
Name: value, dtype: float64Code
goleta[goleta['parameter'] == 'pm25']['value'].describe()count 734.000000
mean 6.480926
std 3.651091
min -4.000000
25% 4.000000
50% 6.000000
75% 8.000000
max 22.000000
Name: value, dtype: float64The first summary averages micrograms per cubic metre together with parts per million. The second one describes 734 particulate readings, all in µg/m³. Only the second one is a fact.
Two conditions at once
& means “and”. Every comparison gets its own parentheses:
Code
goleta[(goleta['parameter'] == 'pm25') & (goleta['value'] > 12)]| location_id | location_name | parameter | value | unit | datetimeUtc | datetimeLocal | timezone | latitude | longitude | country_iso | isMobile | isMonitor | owner_name | provider | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1261 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-13T10:00:00+00:00 | 2024-07-13T03:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1331 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-16T08:00:00+00:00 | 2024-07-16T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1376 | 1186 | Goleta | pm25 | 22.0 | µg/m³ | 2024-07-18T08:00:00+00:00 | 2024-07-18T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1460 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-21T20:00:00+00:00 | 2024-07-21T13:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1465 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-22T01:00:00+00:00 | 2024-07-21T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1496 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-23T08:00:00+00:00 | 2024-07-23T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1520 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-24T08:00:00+00:00 | 2024-07-24T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1529 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-24T17:00:00+00:00 | 2024-07-24T10:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1533 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-24T21:00:00+00:00 | 2024-07-24T14:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1534 | 1186 | Goleta | pm25 | 17.0 | µg/m³ | 2024-07-24T22:00:00+00:00 | 2024-07-24T15:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1536 | 1186 | Goleta | pm25 | 16.0 | µg/m³ | 2024-07-25T00:00:00+00:00 | 2024-07-24T17:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1537 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-25T01:00:00+00:00 | 2024-07-24T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1556 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-25T20:00:00+00:00 | 2024-07-25T13:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1586 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-27T02:00:00+00:00 | 2024-07-26T19:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1587 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-27T03:00:00+00:00 | 2024-07-26T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1592 | 1186 | Goleta | pm25 | 17.0 | µg/m³ | 2024-07-27T08:00:00+00:00 | 2024-07-27T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1600 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-27T16:00:00+00:00 | 2024-07-27T09:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1601 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-27T17:00:00+00:00 | 2024-07-27T10:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1602 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-27T18:00:00+00:00 | 2024-07-27T11:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1603 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-27T19:00:00+00:00 | 2024-07-27T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1608 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-28T00:00:00+00:00 | 2024-07-27T17:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1609 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-07-28T01:00:00+00:00 | 2024-07-27T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1610 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-28T02:00:00+00:00 | 2024-07-27T19:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1630 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-28T22:00:00+00:00 | 2024-07-28T15:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1640 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-07-29T08:00:00+00:00 | 2024-07-29T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1651 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-07-29T19:00:00+00:00 | 2024-07-29T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1712 | 1186 | Goleta | pm25 | 22.0 | µg/m³ | 2024-08-01T08:00:00+00:00 | 2024-08-01T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1802 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-08-05T02:00:00+00:00 | 2024-08-04T19:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1803 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-05T03:00:00+00:00 | 2024-08-04T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1837 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-06T19:00:00+00:00 | 2024-08-06T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1838 | 1186 | Goleta | pm25 | 19.0 | µg/m³ | 2024-08-06T20:00:00+00:00 | 2024-08-06T13:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1839 | 1186 | Goleta | pm25 | 16.0 | µg/m³ | 2024-08-06T21:00:00+00:00 | 2024-08-06T14:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1840 | 1186 | Goleta | pm25 | 16.0 | µg/m³ | 2024-08-06T23:00:00+00:00 | 2024-08-06T16:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1842 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-07T01:00:00+00:00 | 2024-08-06T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1844 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-07T03:00:00+00:00 | 2024-08-06T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1859 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-08-07T18:00:00+00:00 | 2024-08-07T11:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1860 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-07T19:00:00+00:00 | 2024-08-07T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1861 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-07T20:00:00+00:00 | 2024-08-07T13:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1862 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-07T21:00:00+00:00 | 2024-08-07T14:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1863 | 1186 | Goleta | pm25 | 19.0 | µg/m³ | 2024-08-07T22:00:00+00:00 | 2024-08-07T15:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1873 | 1186 | Goleta | pm25 | 21.0 | µg/m³ | 2024-08-08T08:00:00+00:00 | 2024-08-08T01:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1928 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-10T15:00:00+00:00 | 2024-08-10T08:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1929 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-10T16:00:00+00:00 | 2024-08-10T09:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1930 | 1186 | Goleta | pm25 | 20.0 | µg/m³ | 2024-08-10T17:00:00+00:00 | 2024-08-10T10:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1932 | 1186 | Goleta | pm25 | 15.0 | µg/m³ | 2024-08-10T19:00:00+00:00 | 2024-08-10T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1934 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-10T21:00:00+00:00 | 2024-08-10T14:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1940 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-11T03:00:00+00:00 | 2024-08-10T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1953 | 1186 | Goleta | pm25 | 16.0 | µg/m³ | 2024-08-11T16:00:00+00:00 | 2024-08-11T09:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1954 | 1186 | Goleta | pm25 | 16.0 | µg/m³ | 2024-08-11T17:00:00+00:00 | 2024-08-11T10:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1955 | 1186 | Goleta | pm25 | 13.0 | µg/m³ | 2024-08-11T18:00:00+00:00 | 2024-08-11T11:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1956 | 1186 | Goleta | pm25 | 14.0 | µg/m³ | 2024-08-11T19:00:00+00:00 | 2024-08-11T12:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
51 hours where fine particulate matter was above 12 µg/m³, which is the level the US EPA uses as its annual air quality standard.
| means “or”:
Code
particulates = goleta[(goleta['parameter'] == 'pm25') | (goleta['parameter'] == 'pm10')]
particulates.shape(1251, 15)
The parentheses are not optional
Leave them out and you get an error that has nothing to do with what you meant:
goleta[goleta['parameter'] == 'pm25' & goleta['value'] > 12]& binds more tightly than == and >, so Python tries to combine the wrong pieces. The rule is absolute: wrap every comparison in its own parentheses. Do it even when there is only one condition if it helps you build the habit.
✏️ Test your knowledge
Find the ozone readings above 0.04 ppm. Then find the readings that are either ozone above 0.04 ppm or pm25 above 20 µg/m³. How many rows does each give you?
Turning a condition around with ~
~ inverts a mask: every True becomes False and every False becomes True.
Code
not_ozone = goleta[~(goleta['parameter'] == 'o3')]
not_ozone.shape(1251, 15)1,251 rows, which is 1,962 minus the 711 ozone readings. You could have written != here and gotten the same answer:
Code
goleta[goleta['parameter'] != 'o3'].shape(1251, 15)~ earns its keep when the condition is more complicated than a single comparison, and you would rather say “not that” than rewrite the whole thing backwards.
Matching a list of values with .isin()
Writing (col == 'a') | (col == 'b') | (col == 'c') gets tedious fast. .isin() takes a list and matches any of its members:
Code
particulates = goleta[goleta['parameter'].isin(['pm25', 'pm10'])]
particulates.shape(1251, 15)Same 1,251 rows as the | version above, in a line that stays readable when the list grows.
The argument is an ordinary Python list, so you can build it first and pass it by name:
Code
of_interest = ['pm25', 'pm10']
goleta[goleta['parameter'].isin(of_interest)].shape(1251, 15)
Note
🐍 .isin() hangs off the column, not off the DataFrame: df['col'].isin([...]). It returns a mask, exactly like a comparison does, so it goes in the same place inside the brackets.
✏️ Test your knowledge
Build a list called keep_parameters holding 'o3' and 'pm10', then use .isin() to filter goleta down to those two pollutants. Check your row count against the two numbers in goleta['parameter'].value_counts(); they should add up.
Keeping a result with .copy()
Once you have a subset you like, you will usually want to keep working on it, and often that means adding a column to it later this week.
When you plan to keep and modify a filtered result, say .copy():
Code
pm25 = goleta[goleta['parameter'] == 'pm25'].copy()
pm25.shape(734, 15).copy() gives you a table of your own, with no remaining connection to goleta. Without it, pandas may hand you back something that is still tied to the original table. When you then try to change it, you get either a warning you don’t understand or a change that quietly does not stick.
The habit is easy to state:
If the filtered result is going to be a thing you keep and change, end the line with
.copy(). If you are just looking at it, don’t bother.
Code
pm25.head()| location_id | location_name | parameter | value | unit | datetimeUtc | datetimeLocal | timezone | latitude | longitude | country_iso | isMobile | isMonitor | owner_name | provider | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1228 | 1186 | Goleta | pm25 | 3.0 | µg/m³ | 2024-07-12T01:00:00+00:00 | 2024-07-11T18:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1229 | 1186 | Goleta | pm25 | 8.0 | µg/m³ | 2024-07-12T02:00:00+00:00 | 2024-07-11T19:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1230 | 1186 | Goleta | pm25 | 6.0 | µg/m³ | 2024-07-12T03:00:00+00:00 | 2024-07-11T20:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1231 | 1186 | Goleta | pm25 | 4.0 | µg/m³ | 2024-07-12T04:00:00+00:00 | 2024-07-11T21:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
| 1232 | 1186 | Goleta | pm25 | 9.0 | µg/m³ | 2024-07-12T05:00:00+00:00 | 2024-07-11T22:00:00-07:00 | America/Los_Angeles | 34.445301 | -119.827797 | NaN | NaN | NaN | Unknown Governmental Organization | AirNow |
A warning you will meet
SettingWithCopyWarning is pandas telling you it isn’t sure whether you meant to change the subset or the original. It is not always a real problem, but it is always worth reading. Adding .copy() at the point where you made the subset is the fix nine times out of ten.
✏️ Test your knowledge
Make a .copy() of the ozone readings called ozone, then confirm with .shape and .head() that you have what you expect. Keep it in your notebook; you’ll use it in the next session.
Putting it together
A real filter is usually two or three of these ideas in one line:
Code
summer_particulates = goleta[
goleta['parameter'].isin(['pm25', 'pm10']) & (goleta['value'] > 15)
].copy()
summer_particulates.shape(246, 15)Code
summer_particulates[['parameter', 'value', 'unit', 'datetimeLocal']].head()| parameter | value | unit | datetimeLocal | |
|---|---|---|---|---|
| 711 | pm10 | 20.0 | µg/m³ | 2024-07-16T22:00:00-07:00 |
| 720 | pm10 | 21.0 | µg/m³ | 2024-07-17T07:00:00-07:00 |
| 721 | pm10 | 20.0 | µg/m³ | 2024-07-17T08:00:00-07:00 |
| 746 | pm10 | 21.0 | µg/m³ | 2024-07-22T13:00:00-07:00 |
| 748 | pm10 | 16.0 | µg/m³ | 2024-07-22T15:00:00-07:00 |
Notice that the line is broken across several lines inside the square brackets. Python doesn’t mind, and a long filter is much easier to read that way.
Key points
- The filter sentence is
df[df['col'] <comparison> value]. The innerdfbuilds the mask; the outer one applies it. - Filtering returns a new DataFrame. Assign it to a name to keep it.
==matches text exactly, and is case-sensitive. An empty result is an answer, not an error.- Combine conditions with
&for and,|for or,~for not. Every comparison needs its own parentheses. df['col'].isin([...])matches any value in a list, and replaces a pile of|.- End a filter with
.copy()when you plan to keep the result and change it later.