The Final Project

Over Days 8 and 9 you will work in a small team to build a notebook. On Wednesday afternoon you will choose a dataset and then on Thursday and Friday morning you will progress through the ten data science workflow steps. The final product will be a figure (or figures) and an evidence claim based on your analysis. On Friday, you will present your data and findings.

This page is a general description of what you are aiming to create. The project kickoff covers choosing a team, a dataset and a question. Day 8 and Day 9 contain the hour-by-hour plan.

What you are building

One Jupyter notebook, per team, named Project_<team name>.ipynb.

Your notebook should be self-contained: somebody who has your notebook and nothing else should be able to restart the kernel, run every cell from the top, and get your figures back.

Repeatable, shareable, and consistent code is sometimes difficult in notebooks because they can be run asynchronously. You can define a variable in one cell, run the cell, and then later delete it. However, the variable stays alive in the kernel, and the notebook will still work in your session. However, it will not work again for anybody else or if you restart your session from scratch. Therefore, restarting and running notebooks from the top is the best way to ensure you have a consistent and well-structured set of analyses.

Your code should be documented: markdown cells should explain what each step was for and what it found, and code comments should explain anything a reader needs to know to understand your code.

The analysis uses pandas, numpy, matplotlib and seaborn. The only other import you may need is requests, and only if your file lives in Google Drive rather than at a public URL, which the code recipe at the bottom of this page explains. For this exercise, please avoid analyses that require mapping.

The ten steps

Every one of the ten steps should appear in your notebook under its own markdown heading, in this order, even if the content of a step is one sentence.

1. Import Data
2. Explore Data
3. Clean Data
4. Filter Data
5. Sort Data
6. Transform Data
7. Group Data
8. Aggregate Data
9. Join / Reshape Data
10. Visualize Data

These are the same ten steps described on The Data Science Workflow.

Step 9 is the one most likely to be short. If your project uses a single table and needs neither a join nor a reshape, write that under the heading and move on. Step 3 is the next most likely: a file that arrives clean has still been cleaned, and “.isnull().sum() is zero in every column and there are no duplicate rows” is that step done properly. Skipping steps silently is always a bad idea because then nobody reading the notebook knows whether the work was unnecessary or merely undone.

When am I “done”?

This is a checklist, not a grading scheme. As with the rest of the course, nothing you will turn in or present is scored. However, this is the list we would use if you asked me whether your notebook was finished:

The notebook runs.

• Restart the kernel, run all, no errors.
• Every dataset loads from a URL or a path that will still work on somebody else’s machine.

The ten steps are all there.

• Ten markdown headings, in order, matching the list above.
• Under each, either the work or a sentence saying why the step needed none.

The analysis answers a question you wrote down.

• The question appears near the top, in a complete sentence.
• The two questions you set aside are named, with a sentence on why.
• One sentence says what one row of the dataset represents.

The figure contains the evidence for your analysis.

• At least one figure made with seaborn or matplotlib.
• Axis labels with units where units exist.
• A title that states the finding rather than naming the variables.
• A legend, if you used multiple markers/colors.

The analysis conclusion is stated and supported.

• Again, this is usually found in two or three sentences near the top/bottom: what you asked, what you found, what it depends on or assumes.
• The counts behind any group comparison are visible somewhere in the notebook.
• The word significant does not appear, unless you have run an actual hypothesis test.

Somebody else can read it.

• No cells left over from an approach you abandoned.
• Consistent, well-formatted markdown between the code blocks, not just at the top.
• Every team member’s name in the first cell.

Handing it in

The notebook is due before lunch on Friday, September 11. The whole morning is available to finish and polish, and presentations are in the afternoon. There is nothing to submit after class.

The submission form

Submit here. One submission per team. You will need to sign in with your UCSB Google account because the form is restricted to UCSB emails.

1. Put the project notebook in its own repository, separate from your coursework repository.
2. Add a short README.md: the team, the dataset, its source URL, and the question.
3. Push, then open the repository in a browser. GitHub renders .ipynb directly, so you should be able to confirm in ten seconds that the figures are showing.
4. Provide the repository URL in the Google Form.

The presentation

Ten minutes, then three for questions. You will probably present from the notebook itself, and see Day 9 for a walkthrough plan that fits the twelve minutes. Slides are allowed but not expected, and building them could eat up time that you could be using for additional analyses and figure polish.

Extra figures are welcome, and you are encouraged to make the notebook engaging to read. Place figures where they are most helpful; exploratory figures make sense in Step 2, for example.

Finding a dataset

The five tests for a workable dataset are in the project kickoff, so keep them in mind as you look for a suitable dataset: one CSV, more than a couple of hundred rows, at least two numeric columns, at least one categorical column, and you can say what one row represents.

It does not need to be environmental data! Let your curiosity and creativity loose!!

General collections

• Kaggle datasets, filtered to CSV
• Data is Plural
• US Data.gov
• Zenodo
• TidyTuesday

Oddly specific, and all fine choices

• Central Park Squirrel Census
• Harry Potter Characters
• Spotify Tracks
• Lego Dataset

Loading a CSV from Google Drive

If your file is not already at a public URL, put it in a Google Drive folder in your UCSB account and load it from there:

1. Save the file to a Google Drive folder in your UCSB account.
2. Right-click the file and choose Share.
3. Under General access, change Restricted to Anyone with the link.
4. Click Copy link, then Done.
5. Put the code below in the Import Data section at the top of your notebook, with your own link in place of the example.

import pandas as pd
import requests

def extract_file_id(url):
    """Extract file id from Google Drive Sharing URL."""
    return url.split("/")[-2]

def df_from_gdrive_csv(url):
    """ Get the CSV file from a Google Drive Sharing URL."""
    file_id = extract_file_id(url)
    URL = "https://docs.google.com/uc?export=download"
    session = requests.Session()
    response = session.get(URL, params={"id": file_id}, stream=True)
    return pd.read_csv(response.raw)

# Example of how to use:
# Note: your sharing link will be different, but should look like this:
sharing_url = "https://drive.google.com/file/d/1RlilHNG7BtvXT2Pm4OpgNvEjVJJZNaps/view?usp=share_link"
df = df_from_gdrive_csv(sharing_url)
df.head()

Check this on Thursday morning, not Friday

A Drive link that has not been shared correctly will look like it is working fine until someone else tries to access it. Have a teammate run the loading cell on their own machine so you know your file is available correctly.

Resources

• The Data Science Workflow
• Project kickoff
• Putting your work on GitHub
• Workflow methods cheatsheet
• seaborn cheatsheet