Section 2.3 Outliers
Looking at frequency tables, it is easy to identify the shape of the distribution and the most common quantities, but rare quantities are not always visible. Before going on, it is a good idea to check for outliers, which are extreme values that might be measurement or recording errors, or might be accurate reports of rare events.
To identify outliers, the following function takes a
FreqTab object and an integer n, and uses a slice index to select the n smallest quantities and their frequencies.
def smallest(ftab, n=10):
return ftab[:n]
In the frequency table of
prglngth, here are the 10 smallest values.
$ smallest(ftab_length)
prglngth
0 1
4 1
9 1
13 1
17 2
18 1
19 1
20 1
21 2
22 7
Name: prglngth, dtype: int64
Since we selected the rows for live births, pregnancy lengths less than 10 weeks are certainly errors. The most likely explanation is that the outcome was not coded correctly. Lengths higher than 30 weeks are probably legitimate. Between 10 and 30 weeks, it is hard to be sure — some quantities are probably errors, but some are correctly recorded preterm births.
The following function selects the largest values from a
FreqTab object.
def largest(ftab, n=10):
return ftab[-n:]
Here are the longest pregnancy lengths in the dataset.
$ largest(ftab_length)
prglngth
40 1116
41 587
42 328
43 148
44 46
45 10
46 1
47 1
48 7
50 2
Name: prglngth, dtype: int64
Again, some of these values are probably errors. Most doctors recommend induced labor if a pregnancy exceeds 41 weeks, so 50 weeks seems unlikely to be correct. But there is no clear line between values that are certainly errors and values that might be correct reports of rare events.
The best way to handle outliers depends on "domain knowledge" — that is, information about where the data come from and what they mean. And it depends on what analysis you are planning to perform.
In this example, the motivating question is whether first babies tend to be earlier or later than other babies. So we’ll use statistics that are not thrown off too much by a small number of incorrect values.
