Day 1: Tasks & activities

An Example Python Data Science Workflow

Toolik from the boardwalk (source)[https://media.arcus.org/album/polartrec-2019-alejandra-martinez/30679]

Objective

In this exercise, you will work with climate data using the Python data science workflow. You’ll load the data into a pandas DataFrame, perform basic exploration and cleaning, and create visualizations. This hands-on practice will help you understand how Python can be used for data analysis, with comparisons to similar tasks in R.

Background and Data Source

Our data comes from the Arctic Long Term Ecological Research station. The Arctic Long Term Ecological Research (ARC LTER) site is part of a network of sites established by the National Science Foundation to support long-term ecologicalLooking South of Toolik Field Station research in the United States. The research site is located in the foothills region of the Brooks Range, North Slope of Alaska (68° 38’N, 149° 36.4’W, elevation 720 m). The Arctic LTER project’s goal is to understand and predict the effects of environmental change on arctic landscapes, both natural and anthropogenic. Researchers at the site use long-term monitoring and surveys of natural variation of ecosystem characteristics, experimental manipulation of ecosystems (years to decades) and modeling at ecosystem and watershed scales to gain an understanding of the controls of ecosystem structure and function. The data and insights gained are provided to federal, Alaska state and North Slope Borough officials who regulate the lands on the North Slope and through this web site.

We will be using some basic weather data downloaded from Toolik Station:

  • Toolik Station Meteorological Data: toolik_weather.csv Shaver, G. 2019. A multi-year DAILY weather file for the Toolik Field Station at Toolik Lake, AK starting 1988 to present. ver 4. Environmental Data Initiative. https://doi.org/10.6073/pasta/ce0f300cdf87ec002909012abefd9c5c (Accessed 2021-08-08).

I have already downloaded this data and placed in our course repository, where we can access it easily using its github raw url.

Let’s dive into the exercise!

Instructions

Setup and Data Loading

  1. Open JupyterLab and Start a New Notebook

  1. Import Libraries

    • Import the necessary libraries to work with data (pandas) and create plots (matplotlib.pyplot). Use the standard python conventions that import pandas as pd and import matplotlib.pyplot as plt

  1. Load the Data

    Our data is located at:

    https://raw.githubusercontent.com/environmental-data-science/eds217-day0-comp/main/data/raw_data/toolik_weather.csv

    • Create a variable called url that stores the URL provided above as a string.
    • Use the pandas library’s read_csv() function from pandas to load the data from the URL into a new DataFrame called df. Any pandas function will always be called using the pd object and dot notation: pd.read_csv().
Note

The read_csv() function can do a ton of different things, but today all you need to know is that it can take a url to a csv file as it’s only input.

Syntax Similarities

The read_csv() function in pandas is similar to read.csv() in R. In python, the function is part of the pandas library, which we imported as pd. So we call the function using dot notation: pd.read_csv()

Data Exploration

  1. Preview the Data

    • Use the head() method to display the first few rows of the DataFrame df.
Note

Because the head() function is a method of a DataFrame, you will call it using dot notation and the dataframe you just created: df.head()

Tip

Syntax Similarities: In R, you would use head(df) to view the first few rows.

  1. Check for Missing Values

    • Use the isnull() method combined with sum() to count missing values in each column.
Syntax Similarities

In R, you might use sum(is.na(df$column)) to check for missing values.

Important

You should see that the Daily_AirTemp_Mean_C doesn’t have any missing values. This means we can skip the usual step of dealing with missing data. We’ll learn these tools in Python and Pandas later in the course.

  1. Data Description

    • Use the describe() method to generate summary statistics for numerical columns.
    • Use the info() method to get an overview of the DataFrame, including data types and non-null counts. Just like the head() function, these are methods associated with your df object, so you call them with dot notation.
Syntax Similarities

The commands summary(df) and str(df) are R equivalents for summarizing and checking structure. Notice a pattern forming… Other than differences in function names (i.e. “Boot” vs. “Boot” in American/British English), a major “grammar” difference between R and Python is Python’s frequent use of dot notation for calling methods of objects!

Data Analysis

  1. Calculate Monthly Average Temperature

    • Use the groupby() method to group the data by the ‘Month’ column and save this as a new variable called monthly.
    • Calculate the mean of the ‘Daily_AirTemp_Mean_C’ column for each month in the monthly using the mean() function. Save this result to a new variable called monthly_means.
Note

You can do analysis on a specific column in a dataframe using [column_nanme] notation: my_df["column A"].mean() would give the average value of “column A” (if there was a column with that name in the dataframe). In the coming days, we will spend a lot of time learning how to select and subset data in dataframes!

Syntax Similarities

This analysis is similar to using group_by() and summarize() in dplyr.

  1. Plot Monthly Average Temperature

    • Use the bar() method to create a bar plot of the monthly average temperature.
    • Set appropriate titles and labels for the plot. The bar() function is a method of the plt library you imported at the start of your code.
Syntax Similarities

Use plt.plot() or plot.bar() to create plots. In R, you would use ggplot().

  1. Analyze Temperature Trends Over Years

    • Use groupby() to explore how temperature trends change over the years.
    • Create a line plot using the plot() command of the yearly average temperature trend.
Tip

Similar to calculating monthly averages, group by the ‘Year’ column.

Saving Analyses and Figures

Of course, we can always just re-run our notebook code to re-generate our analyses and figures. However, for complicated analyses and long-running processes, it is helpful to save intermediate or final outputs into files that can be re-loaded or used elsewhere. Let’s look as some ways to export our work.

  1. Saving Analyses and Data

To write a pandas.Series to a CSV file, you can use the .to_csv() method (just like you would with a pandas.DataFrame). Here’s an example of how to do it:

# Writing the Series to a CSV file
monthly_means.to_csv("monthly_means.csv", header=True)

Key Points:

  • The index of the Series will be written as the first column in the CSV file.
  • By setting header=True, the name of the Series will be written as the header in the CSV file.
  • The CSV file will be saved in the current working directory unless you specify a different path.
Syntax Similarities

To create a csv from dataframe you use the dataframe’s built-in method .to_csv(). In R, you would use a write.csv() function.

Important

A major difference between Python and R is the extensive use of object methods in Python and the extensive use of global functions in R.**

Example to_csv() Output:

If you inspect the monthly_means.csv file using the file browser in JupyterLab, it will look something like this:

Month,Daily_AirTemp_Mean_C
1,-20.561290322580643
2,-23.94107142857143
3,-17.806451612903224
4,-15.25294117647059
5,-0.8758190327613105
6,8.76624
  1. Saving Figures

Use the plt.savefig() command to save your figure to a file. This functions takes a set of keyword options that determine the output image format, resolution (DPI, or dots per inch) and the size of the image. Here’s an example of a command that produces a jpeg file with 300 dots per inch and with the size of the output image cropped closely around the figure:

plt.savefig("sample_plot.jpg", format='jpeg', dpi=300, bbox_inches='tight')

Conclusion

We will spend the rest of the course learning more about each of the steps we just went through. And of course, we have a lot more to learn about the essentials of the Python programming language over the next 8 days of class.

Take some time now to reflect on what you’ve learned today, and to add some additional comments and notes in your code to follow up on in the coming days.

By the end of the course you will be writing your own Python data science workflows just like this one… hopefully many of the “code strangers” you’ve just met will have become good friends!

🎉🎉 Congratulations! You made it to the end of a Python data science workflow…🎉🎉

🎉🎉..and the end of the first day of EDS 217!! 🎉🎉

End Activity Session (Day 1)