Reading the Data

Section 1.3 Reading the Data

Before downloading NSFG data, you have to agree to the terms of use:

Any intentional identification or disclosure of an individual or establishment violates the assurances of confidentiality given to the providers of the information. Therefore, users will:
🔗

Use the data in this dataset for statistical reporting and analysis only.
🔗

🔗

Make no attempt to learn the identity of any person or establishment included in these data.
🔗

🔗

Not link this dataset with individually identifiable data from other NCHS or non-NCHS datasets.
🔗

🔗

Not engage in any efforts to assess disclosure methodologies applied to protect individuals and establishments or any research on methods of re-identification of individuals and establishments.
🔗

🔗

🔗

If you agree to comply with these terms, instructions for downloading the data are in the notebook for this chapter.

🔗

The data files are available directly from the NSFG web site at cdc.gov/nchs/data_access, but we will download them from the repository for this book, which provides a compressed version of the data file.

🔗

Listing 1.3.1. Python Code

🔗

download("https://github.com/AllenDowney/ThinkStats/raw/v3/data/2002FemPreg.dct")
download("https://github.com/AllenDowney/ThinkStats/raw/v3/data/2002FemPreg.dat.gz")

Listing 1.3.2. Python Code

🔗

try:
    import statadict
except ImportError:
    %pip install statadict

The data is stored in two files, a "dictionary" that describes the format of the data, and a data file.

🔗

Listing 1.3.3. Python Code

🔗

dct_file = "2002FemPreg.dct"
dat_file = "2002FemPreg.dat.gz"

The notebook for this chapter defines a function that reads these files. It is called read_stata because this data format is compatible with a statistical software package called Stata.

🔗

The following function takes these file names as arguments, reads the dictionary, and uses the results to read the data file.

🔗

Listing 1.3.4. Python Code

🔗

from statadict import parse_stata_dict


def read_stata(dct_file, dat_file):
    stata_dict = parse_stata_dict(dct_file)
    resp = pd.read_fwf(
        dat_file,
        names=stata_dict.names,
        colspecs=stata_dict.colspecs,
        compression="gzip",
    )
    return resp

Here’s how we use it.

🔗

Listing 1.3.5. Python Code

🔗

preg = read_stata(dct_file, dat_file)

The result is a DataFrame, which is a Pandas data structure that represents tabular data in rows and columns. This DataFrame contains a row for each pregnancy reported by a respondent and a column for each variable. A variable can contain responses to a survey question or values that are calculated based on responses to one or more questions.

🔗

In addition to the data, a DataFrame also contains the variable names and their types, and it provides methods for accessing and modifying the data. The DataFrame has an attribute called shape that contains the number of rows and columns.

🔗

Listing 1.3.6. Python Code

🔗

$ preg.shape
(13593, 243)

This dataset has 243 variables with information about 13,593 pregnancies. The DataFrame provides a method called head that displays the first few rows.

🔗

Listing 1.3.7. Python Code

🔗

$ preg.head()
   caseid  pregordr  howpreg_n  howpreg_p  moscurrp  nowprgdk  pregend1  \
0       1         1        NaN        NaN       NaN       NaN       6.0   
1       1         2        NaN        NaN       NaN       NaN       6.0   
2       2         1        NaN        NaN       NaN       NaN       5.0   
3       2         2        NaN        NaN       NaN       NaN       6.0   
4       2         3        NaN        NaN       NaN       NaN       6.0   

   pregend2  nbrnaliv  multbrth  ...  poverty_i  laborfor_i  religion_i  \
0       NaN       1.0       NaN  ...          0           0           0   
1       NaN       1.0       NaN  ...          0           0           0   
2       NaN       3.0       5.0  ...          0           0           0   
3       NaN       1.0       NaN  ...          0           0           0   
4       NaN       1.0       NaN  ...          0           0           0   

   metro_i      basewgt  adj_mod_basewgt      finalwgt  secu_p  sest  cmintvw  
0        0  3410.389399      3869.349602   6448.271112       2     9     1231  
1        0  3410.389399      3869.349602   6448.271112       2     9     1231  
2        0  7226.301740      8567.549110  12999.542264       2    12     1231  
3        0  7226.301740      8567.549110  12999.542264       2    12     1231  
4        0  7226.301740      8567.549110  12999.542264       2    12     1231  

[5 rows x 243 columns]

The left column is the index of the DataFrame, which contains a label for each row. In this case, the labels are integers starting from 0, but they can also be strings and other types.

🔗

The DataFrame has an attribute called columns that contains the names of the variables.

🔗

Listing 1.3.8. Python Code

🔗

$ preg.columns
Index(['caseid', 'pregordr', 'howpreg_n', 'howpreg_p', 'moscurrp', 'nowprgdk',
       'pregend1', 'pregend2', 'nbrnaliv', 'multbrth',
       ...
       'poverty_i', 'laborfor_i', 'religion_i', 'metro_i', 'basewgt',
       'adj_mod_basewgt', 'finalwgt', 'secu_p', 'sest', 'cmintvw'],
      dtype='object', length=243)

The column names are contained in an Index object, which is another Pandas data structure. To access a column from a DataFrame, you can use the column name as a key.

🔗

Listing 1.3.9. Python Code

🔗

$ pregordr = preg["pregordr"]
type(pregordr)
pandas.core.series.Series

The result is a Pandas Series, which represents a sequence of values. Series also provides head, which displays the first few values and their labels.

🔗

Listing 1.3.10. Python Code

🔗

$ pregordr.head()
0    1
1    2
2    1
3    2
4    3
Name: pregordr, dtype: int64

The last line includes the name of the Series and dtype, which is the type of the values. In this example, int64 indicates that the values are 64-bit integers.

🔗

The NSFG dataset contains 243 variables in total. Here are some of the ones we’ll use for the explorations in this book.

🔗

caseid is the integer ID of the respondent.
🔗

🔗
pregordr is a pregnancy serial number: the code for a respondent’s first pregnancy is 1, for the second pregnancy is 2, and so on.
🔗

🔗
prglngth is the integer duration of the pregnancy in weeks.
🔗

🔗
outcome is an integer code for the outcome of the pregnancy. The code 1 indicates a live birth.
🔗

🔗
birthord is a serial number for live births: the code for a respondent’s first child is 1, and so on. For outcomes other than live birth, this field is blank.
🔗

🔗
birthwgt_lb and birthwgt_oz contain the pounds and ounces parts of the birth weight of the baby.
🔗

🔗
agepreg is the mother’s age at the end of the pregnancy.
🔗

🔗
finalwgt is the statistical weight associated with the respondent. It is a floating-point value that indicates the number of people in the U.S. population this respondent represents.
🔗

🔗

If you read the codebook carefully, you will see that many of the variables are recodes, which means that they are not part of the raw data collected by the survey — they are calculated using the raw data.

🔗

For example, prglngth for live births is equal to the raw variable wksgest (weeks of gestation) if it is available; otherwise it is estimated using mosgest * 4.33 (months of gestation times the average number of weeks in a month).

🔗

Recodes are often based on logic that checks the consistency and accuracy of the data. In general it is a good idea to use recodes when they are available, unless there is a compelling reason to process the raw data yourself.

🔗

Prev Top Next