Day 4: Tasks & activities

Reading, Filtering, and Visualizing Data in Pandas

Introduction

This end-of-day session is focused on using pandas for loading, visualizing, and analyzing marine microplastics data. This session is designed to help you become more comfortable with the pandas library, equipping you with the skills needed to perform data analysis effectively.

The National Oceanic and Atmospheric Administration, via its National Centers for Environmental Information has an entire section related to marine microplastics – that is, microplastics found in water β€” at https://www.ncei.noaa.gov/products/microplastics.

We will be working with a recent download of the entire marine microplastics dataset. The url for this data is located here:

Code 
url = 'https://ucsb.box.com/shared/static/dnnu59jsnkymup6o8aaovdywrtxiy3a9.csv'

Objective: Write your own notebook that contains a simple DataFrame exploration as well as some basic grouping, filtering, and aggregation, and visualization… all within the pandas library.

1. Loading the Data

Objective: Learn to load data into a pandas DataFrame and display the first few records.

Task 1.1 Import the pandas library.

Task 1.2: Load the dataset into a dataframe named df from the provided URL into a pandas DataFrame.

Tip

I’ve already taken a look at this data set and noticed there was a column with sample date called Date. We can use the parse_date option of the read_csv() function to convert values in the Dates column of the csv into datetime objects in pandas while reading the file.

Task 1.3:

  • Display the first five rows of the DataFrame to get an initial understanding of the data structure.

2. Exploring the Data

Task 2.1:

  • Display summary statistics of the dataset to understand the central tendency and variability.

Task 2.2:

  • Check the data types of each column and identify if there are any missing values.
  • Remove any records that are missing a value in the Oceans
Using ~ to invert built-in function results

The ~ operator inverts a list of Boolean values (switches True to False and vice versa).

This operator isn’t useful for most selection operations because you can just use == and != to invert selection criteria. However, the ~ operator becomes very handy when there is a need to invert the results of a built-in function.

For example, the use of the ~ operator and isnull() combine to create an efficient way to filter dataframes where the value of a df[column] is not isnull():

df_valid = df[~(df['column'].isnull())].copy()

Note that the results of the built-in function - df['column'].isnull() need to be wrapped in ( ) for the ~ operator to work properly.

3. Grouping the Data

Task 3.1:

  • Create a groupby object called oceans that groups the data in df according to the value of the Oceans column.

Task 3.2:

  • Determine the total number of Measurements taken from each Ocean.

Task 3.3:

  • Determine the average value of Measurement taken from each Ocean.

4. Filtering the Data and Aggregating the data

Task 4.1:

  • Filter the data to a new df (called df2) that only contains rows where the Unit of measurement is pieces/m3

Task 4.2:

  • Use the groupby and the max() command to determine the Maximum value of pieces/m3 measured for each Ocean

5. Simple Plots in pandas

Task 5.1:

  • Make a histogram of the latitude of every sample in your filtered dataframe using the DataFrame plot command.

Task 5.2:

  • Make a new dataframe (df3) from your filtered dataframe (df2) that contains only rows where Measurement is greater than zero.
Important

Using .copy() when filtering a dataframe ensures that you’re working with a new DataFrame, not a view of the original. This is especially important when you’re filtering data and then modifying the result, which is common in data science workflows.

Task 5.3

  • Create a new column in df3 that contains the log10 of Measurements.
Tip

The numpy library has a log10() function that you will find useful for this step!

Task 5.4

  • Make a histogram of the log-transformed values in df3

Conclusion

πŸŽ‰ Congratulations, you’re officially doing python data science! πŸŽ‰

Be sure to save your notebook and add comments and reflections at the end of your notebook before heading out for the day.

End Activity Session (Day 4)