Think Stats Exploratory Data Analysis in Python

To work with data effectively, you have to think on two levels at the same time: the level of statistics and the level of context. As an example, let’s select the rows in the pregnancy file with caseid 10229. The query method takes a string that can contain column names, comparison operators, and numbers, among other things.

$ subset = preg.query("caseid == 10229")
subset.shape
(7, 244)

The result is a DataFrame that contains only the rows where the query is True. This respondent reported seven pregnancies — here are their outcomes, which are recorded in chronological order.

$ subset["outcome"].values
array([4, 4, 4, 4, 4, 4, 1])

The outcome code 1 indicates a live birth. Code 4 indicates a miscarriage — that is, a pregnancy loss, usually with no known medical cause.

Statistically this respondent is not unusual. Pregnancy loss is common and there are other respondents who reported as many instances. But remembering the context, this data tells the story of a woman who was pregnant six times, each time ending in miscarriage. Her seventh and most recent pregnancy ended in a live birth. If we consider this data with empathy, it is natural to be moved by the story it tells.

Each row in the NSFG dataset represents a person who provided honest answers to many personal and difficult questions. We can use this data to answer statistical questions about family life, reproduction, and health. At the same time, we have an obligation to consider the people represented by the data, and to afford them respect and gratitude.