EDS 217 Cheatsheet

Using pd.read_csv and Its Key Options

Basic Usage of pd.read_csv

Reading a Simple CSV File

Code 
import pandas as pd
from io import StringIO

# Create a simple CSV string
csv_data = """
name,age,city
Alice,28,New York
Bob,35,San Francisco
Charlie,42,Chicago
"""

# Read the CSV data
df = pd.read_csv(StringIO(csv_data))
print(df)
      name  age           city
0    Alice   28       New York
1      Bob   35  San Francisco
2  Charlie   42        Chicago

Selecting Specific Columns

Using the usecols Parameter

Code 
# Read only specific columns
df = pd.read_csv(StringIO(csv_data), usecols=['name', 'city'])
print(df)
      name           city
0    Alice       New York
1      Bob  San Francisco
2  Charlie        Chicago

Naming Columns

Using the names Parameter

Code 
# Rename columns while reading
df = pd.read_csv(StringIO(csv_data), names=['full_name', 'years', 'location'])
print(df)
  full_name years       location
0      name   age           city
1     Alice    28       New York
2       Bob    35  San Francisco
3   Charlie    42        Chicago

Specifying an Index

Using the index_col Parameter

Code 
# Set 'name' column as index
df = pd.read_csv(StringIO(csv_data), index_col='name')
print(df)
         age           city
name                       
Alice     28       New York
Bob       35  San Francisco
Charlie   42        Chicago

Parsing Dates

Automatic Date Parsing

Code 
csv_data_with_dates = """
date,event
2023-01-15,Conference
2023-02-28,Workshop
2023-03-10,Seminar
"""

df = pd.read_csv(StringIO(csv_data_with_dates), parse_dates=['date'])
print(df.dtypes)
print(df)
date     datetime64[ns]
event            object
dtype: object
        date       event
0 2023-01-15  Conference
1 2023-02-28    Workshop
2 2023-03-10     Seminar

Custom Date Parsing

Code 
csv_data_custom_dates = """
date,event
15/01/2023,Conference
28/02/2023,Workshop
10/03/2023,Seminar
"""

df = pd.read_csv(StringIO(csv_data_custom_dates), parse_dates=['date'], date_format='%d/%m/%Y')
print(df.dtypes)
print(df)
date     datetime64[ns]
event            object
dtype: object
        date       event
0 2023-01-15  Conference
1 2023-02-28    Workshop
2 2023-03-10     Seminar

Handling Headers

CSV with Multi-line Header

Code 
csv_data_with_header = """
# This file contains employee data
# Created by: HR Department, Last updated: 2023-08-23
Employee ID,Name,Department,Salary
101,John Doe,Marketing,50000
102,Jane Smith,Engineering,60000
103,Mike Johnson,Sales,55000
"""

# Read CSV ignoring the first two lines
df = pd.read_csv(StringIO(csv_data_with_header), header=2)
print(df)

# Read CSV treating the first row as header and skipping the next two
df_alt = pd.read_csv(StringIO(csv_data_with_header), header=0, skiprows=2)
print("\nAlternative method:")
print(df_alt)
   Employee ID          Name   Department  Salary
0          101      John Doe    Marketing   50000
1          102    Jane Smith  Engineering   60000
2          103  Mike Johnson        Sales   55000

Alternative method:
                         # Created by: HR Department  Last updated: 2023-08-23
Employee ID Name                          Department                    Salary
101         John Doe                       Marketing                     50000
102         Jane Smith                   Engineering                     60000
103         Mike Johnson                       Sales                     55000

CSV with No Header

Code 
csv_data_no_header = """
Alice,28,New York
Bob,35,San Francisco
Charlie,42,Chicago
"""

df = pd.read_csv(StringIO(csv_data_no_header), header=None, names=['name', 'age', 'city'])
print(df)
      name  age           city
0    Alice   28       New York
1      Bob   35  San Francisco
2  Charlie   42        Chicago

Dealing with Missing Data

Customizing NA Values

Code 
csv_data_missing = """
name,age,city
Alice,28,New York
Bob,N/A,San Francisco
Charlie,42,Unknown
"""

df = pd.read_csv(StringIO(csv_data_missing), na_values=['N/A', 'Unknown'])
print(df)
      name   age           city
0    Alice  28.0       New York
1      Bob   NaN  San Francisco
2  Charlie  42.0            NaN

Coercing Columns to Specific Data Types

Using the dtype Parameter

Code 
csv_data_types = """
id,name,score
1,Alice,85.5
2,Bob,92.0
3,Charlie,78.5
"""

df = pd.read_csv(StringIO(csv_data_types), dtype={'id': int, 'name': str, 'score': float})
print(df.dtypes)
print(df)
id         int64
name      object
score    float64
dtype: object
   id     name  score
0   1    Alice   85.5
1   2      Bob   92.0
2   3  Charlie   78.5

Reading Large CSV Files

Using chunksize for Memory Efficiency

Code 
import numpy as np

# Generate a large CSV string (100,000 rows)
np.random.seed(0)
large_csv_data = "id,value\n" + "\n".join([f"{i},{np.random.rand()}" for i in range(100000)])

# Read the large CSV in chunks
chunk_size = 20000
chunks = pd.read_csv(StringIO(large_csv_data), chunksize=chunk_size)

# Process each chunk
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(chunk.head())
    print(f"Chunk shape: {chunk.shape}")
    print("\n")
Chunk 1:
   id     value
0   0  0.548814
1   1  0.715189
2   2  0.602763
3   3  0.544883
4   4  0.423655
Chunk shape: (20000, 2)


Chunk 2:
          id     value
20000  20000  0.392173
20001  20001  0.041157
20002  20002  0.923301
20003  20003  0.406235
20004  20004  0.944282
Chunk shape: (20000, 2)


Chunk 3:
          id     value
40000  40000  0.369256
40001  40001  0.211326
40002  40002  0.476905
40003  40003  0.082234
40004  40004  0.237659
Chunk shape: (20000, 2)


Chunk 4:
          id     value
60000  60000  0.927955
60001  60001  0.902937
60002  60002  0.427617
60003  60003  0.510806
60004  60004  0.583200
Chunk shape: (20000, 2)


Chunk 5:
          id     value
80000  80000  0.011097
80001  80001  0.001770
80002  80002  0.155055
80003  80003  0.316761
80004  80004  0.651845
Chunk shape: (20000, 2)

Remember, these examples use StringIO to simulate reading from a file. When working with actual CSV files, you would replace StringIO(csv_data) with the file path or URL.