Validation

Section 1.4 Validation

When data is exported from one software environment and imported into another, errors might be introduced. And when you are getting familiar with a new dataset, you might decode data incorrectly or misunderstand its meaning. If you invest time to validate the data, you can save time later and avoid errors.

🔗

One way to validate data is to compute basic statistics and compare them with published results. For example, the NSFG codebook includes tables that summarize each variable. Here is the table for outcome, which encodes the outcome of each pregnancy.

🔗

Table 1.4.1. Codebook values for outcome

🔗

Value	Label	Total
1	LIVE BIRTH	9148
2	INDUCED ABORTION	1862
3	STILLBIRTH	120
4	MISCARRIAGE	1921
5	ECTOPIC PREGNANCY	190
6	CURRENT PREGNANCY	352
Total		13593

The "Total" column indicates the number of pregnancies with each outcome. To check these totals, we’ll use the value_counts method, which counts the number of times each value appears, and sort_index, which sorts the results according to the values in the Index (the left column).

🔗

Listing 1.4.2. Python Code

🔗

$ preg["outcome"].value_counts().sort_index()
outcome
1    9148
2    1862
3     120
4    1921
5     190
6     352
Name: count, dtype: int64

Comparing the results with the published table, we can confirm that the values in outcome are correct. Similarly, here is the published table for birthwgt_lb.

🔗

Table 1.4.3. Codebook values for birthwgt_lb

🔗

Value	Label	Total
.	inapplicable	4449
0-5	UNDER 6 POUNDS	1125
6	6 POUNDS	2223
7	7 POUNDS	3049
8	8 POUNDS	1889
9-95	9 POUNDS OR MORE	799
97	Not ascertained	1
98	REFUSED	1
99	DON’T KNOW	57
Total		13593

Birth weight is only recorded for pregnancies that ended in a live birth. The table indicates that there are 4449 cases where this variable is inapplicable. In addition, there is one case where the question was not asked, one where the respondent did not answer, and 57 cases where they did not know.

🔗

Again, we can use value_counts to compare the counts in the dataset to the counts in the codebook.

🔗

Listing 1.4.4. Python Code

🔗

$ counts = preg["birthwgt_lb"].value_counts(dropna=False).sort_index()
counts
birthwgt_lb
0.0        8
1.0       40
2.0       53
3.0       98
4.0      229
5.0      697
6.0     2223
7.0     3049
8.0     1889
9.0      623
10.0     132
11.0      26
12.0      10
13.0       3
14.0       3
15.0       1
51.0       1
97.0       1
98.0       1
99.0      57
NaN     4449
Name: count, dtype: int64

The argument dropna=False means that value_counts does not ignore values that are "NA" or "Not applicable". These values appear in the results as NaN, which stands for "Not a number" — and the count of these values is consistent with the count of inapplicable cases in the codebook.

🔗

The counts for 6, 7, and 8 pounds are consistent with the codebook. To check the counts for the weight range from 0 to 5 pounds, we can use an attribute called loc — which is short for "location" — and a slice index to select a subset of the counts.

🔗

Listing 1.4.5. Python Code

🔗

$ counts.loc[0:5]
birthwgt_lb
0.0      8
1.0     40
2.0     53
3.0     98
4.0    229
5.0    697
Name: count, dtype: int64

And we can use the sum method to add them up.

🔗

Listing 1.4.6. Python Code

🔗

$ counts.loc[0:5].sum()
np.int64(1125)

The total is consistent with the codebook.

🔗

The values 97, 98, and 99 represent cases where the birth weight is unknown. There are several ways we might handle missing data. A simple option is to replace these values with NaN. At the same time, we will also replace a value that is clearly wrong, 51 pounds.

🔗

We can use the replace method like this:

🔗

Listing 1.4.7. Python Code

🔗

preg["birthwgt_lb"] = preg["birthwgt_lb"].replace([51, 97, 98, 99], np.nan)

The first argument is a list of values to be replaced. The second argument, np.nan, gets the NaN value from NumPy.

🔗

When you read data like this, you often have to check for errors and deal with special values. Operations like this are called data cleaning.

🔗

Prev Top Next