Transformation

Section 1.5 Transformation

As another kind of data cleaning, sometimes we have to convert data into different formats, and perform other calculations.

🔗

For example, agepreg contains the mother’s age at the end of the pregnancy. According to the codebook, it is an integer number of centiyears (hundredths of a year), as we can tell if we use the mean method to compute its average.

🔗

Listing 1.5.1. Python Code

🔗

$ preg["agepreg"].mean()
np.float64(2468.8151197039497)

To convert it to years, we can divide through by 100.

🔗

Listing 1.5.2. Python Code

🔗

$ preg["agepreg"] /= 100.0
preg["agepreg"].mean()
np.float64(24.6881511970395)

Now the average is more credible.

🔗

As another example, birthwgt_lb and birthwgt_oz contain birth weights with the pounds and ounces in separate columns. It will be more convenient to combine them into a single column that contains weights in pounds and fractions of a pound.

🔗

First we’ll clean birthwgt_oz as we did with birthwgt_lb.

🔗

Listing 1.5.3. Python Code

🔗

$ preg["birthwgt_oz"].value_counts(dropna=False).sort_index()
birthwgt_oz
0.0     1037
1.0      408
2.0      603
3.0      533
4.0      525
5.0      535
6.0      709
7.0      501
8.0      756
9.0      505
10.0     475
11.0     557
12.0     555
13.0     487
14.0     475
15.0     378
97.0       1
98.0       1
99.0      46
NaN     4506
Name: count, dtype: int64

Listing 1.5.4. Python Code

🔗

preg["birthwgt_oz"] = preg["birthwgt_oz"].replace([97, 98, 99], np.nan)

Now we can use the cleaned values to create a new column that combines pounds and ounces into a single quantity.

🔗

Listing 1.5.5. Python Code

🔗

$ preg["totalwgt_lb"] = preg["birthwgt_lb"] + preg["birthwgt_oz"] / 16.0
preg["totalwgt_lb"].mean()
np.float64(7.265628457623368)

The average of the result seems plausible.

🔗

Prev Top Next