Day 3: Missing Data

Welcome to class!

Gratitude Journal

Announcements

Questions 1 and 2

What issues are we still running into?

How to work with missing data

What counts as missing data?

How to identify missing data

df.isnull().sum()
df.describe()
df.column.value_counts(dropna=False)

pd.crosstab()

Option 1: Remove missing values

Be careful with .dropna(), and make sure you know what it is doing to your data!

Let’s use the pandas example:

df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
                   "toy": [np.nan, 'Batmobile', 'Bullwhip'],
                   "born": [pd.NaT, pd.Timestamp("1940-04-25"),
                            pd.NaT]})

Q: When would we ever use dropna()?

A: Almost never! Why do you think it is a bad idea? df.dropna()

Q: What argument do we use to drop rows where all values are NA?

A: df.dropna(how='all') reference

Q: What if we want to drop NA rows based on one column?

A: df.dropna(subset=['toy']) reference

Option 2: Replacing missing values

Again, let’s use the pandas example:

df = pd.DataFrame([[np.nan, 2, np.nan, 0],
                   [3, 4, np.nan, 1],
                   [np.nan, np.nan, np.nan, 5],
                   [np.nan, 3, np.nan, 4]],
                  columns=list("ABCD"))

Q: What if we want to replace all the NA in the wt column with the mean weight?

A: fillna() reference

Q: What if we want to replace all the 999 with a 4?

A: replace() reference

Q: What if we want to replace all the NAs with a linear interpolation?

A: interpolate() reference

Question 3

What columns do we need to use for question 3 (total number of flights delayed by weather)?

num_of_delays_weather
num_of_delays_late_aircraft
num_of_delays_nas

weather = flights.assign(
    severe = #????,
    mild_late = #????,
    mild_nas = np.where(#????),
    total_weather = # add up severe and mild,
).filter(['airport_code','month','severe','mild_late','mild_nas',
    'total_weather', 'num_of_delays_total'])