Building a Python Data Science Workflow

A panda, one marble away from a data science result. MidJourney 5

Yesterday afternoon you (hopefully!) ended up with a team, a dataset, three questions you’re interested in asking, and one rough figure.

Today you will keep working to determine which of your three questions the data can actually answer. We will cover no new python in the next two days, although it is likely that you will move into new areas as a team. However, at a fundamental level, every method your projects require has already been covered. The goal now is to put together the individual steps of our workflow, customize your analyses to the particular aspects of your project and data, and then put everything together into a notebook that you can present and share with your classmates.

Today’s agenda (Note: your actual progress will go however it goes!)

When	What
10:00 to 11:30	Choose your question! Steps 1 and 2, import and explore.
11:30	Checkpoint 1
11:30 to 12:00	Begin Step 3, clean.
12:00 to 1:00	Lunch
1:00 to 2:30	Steps 3 to 6: clean, filter, sort, transform.
2:30	Checkpoint 2
2:30 to 4:30	Steps 7 to 9: group, aggregate, join or reshape. A first figure.
4:30	Checkpoint 3

The checkpoints exist so that we can make sure that you aren’t getting too bogged down and/or identify any needs to rethink or rescope team plans. Better to find that out early than to be scrambling on Friday morning to read a dataset that is still not loading!

The three checkpoints

Checkpoint 1, 11:30(ish). Do you have data in hand and a question in mind?

By late morning your notebook should hold:

One question, chosen from the three questions you wrote yesterday (or a new contender!), written near the top of the notebook in a complete sentence. Make sure you’ve mapped your question to specific columns in your dataset.
A sentence saying why the other two (or three, etc…) questions were set aside. Usually this is some combination of data or coding knowledge limitations. You don’t need to use this project to teach yourself an entire new python course, so “we didn’t know how to do that” is a legitimate answer. You can also track down Cella or Kelly to get some help on a question you are not sure about how to address.
Step 1, Import, under its own heading: the read, the URL, and one sentence on what one row represents.
Step 2, Explore, under its own heading: .shape, .dtypes, .isnull().sum(), .describe(), and a written note on anything surprising.

If you are still choosing between datasets at 11:30, stop choosing and take whichever one loaded cleanly. A dull dataset you can open beats an exciting one you cannot.

Checkpoint 2, about 2:30. Your dataframe is in the shape the question requires

By mid afternoon, steps 3 to 6 should be done(ish) and written up:

Step 3, Clean. What was missing, what you did about it, and how many rows that cost you. If the file arrived clean, write that under the Clean heading and move on. Writing it down is how Step 3 still gets done when there is nothing to clean (Remember: Absence of evidence is not evidence of absence!)
Step 4, Filter. Which rows your question is about, and which it is not.
Step 5, Sort. Even if sorting is only how you looked at the result.
Step 6, Transform. Any column you derived, with the reason you needed it.

The test is whether you can point at one DataFrame and say, in a sentence, what question that table is about. The amount of code it took to get there does not matter.

Checkpoint 3, 4:30. The analysis is finished(ish)

By the end of the day an answer should exist, even if it is ugly:

Steps 7 and 8, Group and Aggregate. The split, apply and combine that produces the numbers your claim rests on, with the counts each group is built from.
Step 9, Join or Reshape. If your project uses a single table and needs neither, write that under the heading and move on.
A first version of your figure, with axis labels, showing the thing you are going to claim. Tomorrow morning we refine this same figure into Step 10. It does not have to be good today, it has to exist.
One sentence in a markdown cell, saying what you found.

The one-sentence finding is the point of the whole exercise. Tomorrow morning is for making it presentable, and there is nothing to make presentable until you have written it down!

If you are behind at a checkpoint

Being behind at a checkpoint is normal and expected. The fix is almost never to work faster, but rather to ask for help and to try and de-escalate your plans to get to a place where you can move forward.

Behind at Checkpoint 1. The dataset may be the problem. Change it! Go back to the checklist headed “What makes a dataset workable in two days” in the project kickoff, and pick a file that passes all five of the heuristic tests. You lose an hour, but you might save the day!
Behind at Checkpoint 2. Your question is probably too big. Narrow it, usually by dropping a comparison: two groups instead of six, one year instead of a decade.
Behind at Checkpoint 3. Stop adding and start writing. A finished small analysis presents well. An unfinished large one does not.

Come and find Cella or Kelly at any time, but especially if you are nearing a checkpoint and find yourself behind.

Class materials

Session	Session 1	Session 2
day 8 / morning	📝 Final project: choose the question, then Steps 1 to 3	Checkpoint 1 at 11:30
day 8 / afternoon	📝 Final project: Steps 3 to 9, and one figure	Checkpoint 2 at 2:30, Checkpoint 3 at 4:30
day 8 / any time	🔗 Putting your work on GitHub, which you will need before Friday

End-of-day practice

There is no end-of-day activity today. The project is the only thing we are working on.