Section 2.1 Examining numerical data

In this section we will explore techniques for summarizing numerical variables. For example, consider the loan_amount variable from the loan50 data set, which represents the loan size for all 50 loans in the data set. This variable is numerical since we can sensibly discuss the numerical difference of the size of two loans. On the other hand, area codes and zip codes are not numerical, but rather they are categorical variables.

Throughout this section and the next, we will apply these methods using the loan50 and county data sets, which were introduced in Section 1.2. If you’d like to review the variables from either data set, see the relevant data description figures.

Subsection 2.1.1 Scatterplots for paired data

A scatterplot provides a case-by-case view of data for two numerical variables. In a previous figure, a scatterplot was used to examine the homeownership rate against the fraction of housing units that were part of multi-unit properties (e.g. apartments) in the county data set. Another scatterplot is shown in Figure 2.1.1, comparing the total income of a borrower (total_income) and the amount they borrowed (loan_amount) for the loan50 data set. In any scatterplot, each point represents a single case. Since there are 50 cases in loan50, there are 50 points in Figure 2.1.1.

A scatterplot is shown with "Total Income" along the horizontal axis (range from $0 to $325,000) and "Loan Amount" along the vertical axis (range from $0 to $40,000). The points lie in a range from $2,000 to $33,000 in loan amount when total income is smaller than $150,000 (representing most of the points). The range of loan amounts is higher when total income is greater than $175,000, with the range of observations being about $15,000 to $40,000. — Figure 2.1.1. A scatterplot of `total_income` versus `loan_amount` for the `loan50` data set

Looking at Figure 2.1.1, we see that there are many borrowers with an income below $100,000 on the left side of the graph, while there are a handful of borrowers with income above $250,000.

Example 2.1.2. Nonlinear relationships in scatterplots.

Figure 2.1.3 shows a plot of median household income against the poverty rate for 3,142 counties. What can be said about the relationship between these variables?

Solution.

The relationship is evidently nonlinear, as highlighted by the dashed line. This is different from previous scatterplots we’ve seen, which show relationships that do not show much, if any, curvature in the trend.

A scatterplot of a few thousand points is shown with "Poverty Rate" along the horizontal axis (range from 0% to 55%) and "Median Household Income" along the vertical axis (range from $0 to $130,000). A curved trend line is overlaid on the points starting higher on the left and decreasing as it moves right, but it starts flattening the further right it goes. Below 10% poverty rate, points range from about $40,000 to $130,000. Between 10% to 20%, the range is lower at about $25,000 to close to $100,000. For 20% to 30%, the points range from about $22,000 to just over $60,000. For 30% to 50%, the trend is mostly flat with values ranging from about $20,000 to $50,000. — Figure 2.1.3. A scatterplot of the median household income against the poverty rate for the `county` data set

Checkpoint 2.1.4. Value of scatterplots.

What do scatterplots reveal about the data, and how are they useful?

Solution.

Answers may vary. Scatterplots are helpful in quickly spotting associations relating variables, whether those associations come in the form of simple trends or whether those relationships are more complex.

Checkpoint 2.1.5. Horseshoe-shaped associations.

Describe two variables that would have a horseshoe-shaped association in a scatterplot ($\cap$ or $\frown$).

Solution.

Consider the case where your vertical axis represents something "good" and your horizontal axis represents something that is only good in moderation. Health and water consumption fit this description: we require some water to survive, but consume too much and it becomes toxic and can kill a person.

Subsection 2.1.2 Dot plots and the mean

Sometimes two variables are one too many: only one variable may be of interest. In these cases, a dot plot provides the most basic of displays. A dot plot is a one-variable scatterplot; an example using the interest rate of 50 loans is shown in Figure 2.1.6. A stacked version of this dot plot is shown in Figure 2.1.7.

A dot plot is shown for the variable "Interest Rate". There is a horizontal axis ranging from about 5% to a bit over 25%, and then several points are shown horizontally above the axis, scattered over the range. There is a higher density of points between 5% to 11%, with a moderate density of points from 12% to about 20%, and then a few more observations at about 22%, 25%, and 26%. A red triangle is also shown at approximately 12%. — Figure 2.1.6. A dot plot of `interest_rate` for the `loan50` data set

A stacked dot plot is shown for the variable "Interest Rate". There is a horizontal axis ranging from about 5% to a bit over 25%, and then several stacks of points are shown at values 5%, 6%, 7%, and so on. There are 3 points stacked at 5%, 3 points stacked at 6%, 5 at 7%, 4 at 8%, 5 at 9%, 8 at 10%, 5 at 11%, 1 at 11%, 3 at 12%, then 1 each at 14%, 15%, and 16%, 3 at 17%, 2 at 18%, and then 1 each at 19%, 20%, 21%, 25%, and 26%. A red triangle is also shown at approximately 12%. The rates have been rounded to the nearest percent in this plot. — Figure 2.1.7. A stacked dot plot of `interest_rate` for the `loan50` data set

The mean, often called the average, is a common way to measure the center of a distribution of data. To compute the mean interest rate, we add up all the interest rates and divide by the number of observations:

\begin{gather*} \bar{x} = \frac{10.90\% + 9.92\% + 26.30\% + \cdots + 6.08\%}{50} = 11.57\% \end{gather*}

The sample mean is often labeled $\bar{x}\text{.}$ The letter $x$ is being used as a generic placeholder for the variable of interest, interest_rate, and the bar over the $x$ communicates we’re looking at the average interest rate, which for these 50 loans was 11.57%. It is useful to think of the mean as the balancing point of the distribution, and it’s shown as a triangle in Figure 2.1.6 and Figure 2.1.7.

Mean.

The sample mean can be computed as the sum of the observed values divided by the number of observations:

\begin{gather*} \bar{x} = \frac{x_1 + x_2 + \cdots + x_n}{n} \end{gather*}

where $x_1\text{,}$ $x_2\text{,}$ $\dots\text{,}$ $x_n$ represent the $n$ observed values.

Checkpoint 2.1.8. Understanding mean notation.

Examine the equation for the mean. What does $x_1$ correspond to? And $x_2\text{?}$ Can you infer a general meaning to what $x_i$ might represent?

Solution.

$x_1$ corresponds to the interest rate for the first loan in the sample (10.90%), $x_2$ to the second loan’s interest rate (9.92%), and $x_i$ corresponds to the interest rate for the $i^{th}$ loan in the data set. For example, if $i = 4\text{,}$ then we’re examining $x_4\text{,}$ which refers to the fourth observation in the data set.

Checkpoint 2.1.9. Sample size.

What was $n$ in this sample of loans?

Solution.

The sample size was $n = 50\text{.}$

The loan50 data set represents a sample from a larger population of loans made through Lending Club. We could compute a mean for this population in the same way as the sample mean. However, the population mean has a special label: $\mu\text{.}$ The symbol $\mu$ is the Greek letter mu and represents the average of all observations in the population. Sometimes a subscript, such as $_x\text{,}$ is used to represent which variable the population mean refers to, e.g. $\mu_x\text{.}$ Often times it is too expensive to measure the population mean precisely, so we often estimate $\mu$ using the sample mean, $\bar{x}\text{.}$

Example 2.1.10. Estimating a population mean.

The average interest rate across all loans in the population can be estimated using the sample data. Based on the sample of 50 loans, what would be a reasonable estimate of $\mu_x\text{,}$ the mean interest rate for all loans in the full data set?

Solution.

The sample mean, 11.57%, provides a rough estimate of $\mu_x\text{.}$ While it’s not perfect, this is our single best guess of the average interest rate of all the loans in the population under study. In later chapters, we will develop tools to characterize the accuracy of point estimates like the sample mean. As you might have guessed, point estimates based on larger samples tend to be more accurate than those based on smaller samples.

Example 2.1.11. Using the mean for comparisons.

The mean is useful because it allows us to rescale or standardize a metric into something more easily interpretable and comparable. Provide 2 examples where the mean is useful for making comparisons.

Solution.

We would like to understand if a new drug is more effective at treating asthma attacks than the standard drug. A trial of 1500 adults is set up, where 500 receive the new drug, and 1000 receive a standard drug in the control group. The results show 200 asthma attacks in the new drug group and 300 in the standard drug group. Comparing the raw counts of 200 to 300 asthma attacks would make it appear that the new drug is better, but this is an artifact of the imbalanced group sizes. Instead, we should look at the average number of asthma attacks per patient in each group: New drug: $200 / 500 = 0.4\text{,}$ Standard drug: $300 / 1000 = 0.3\text{.}$ The standard drug has a lower average number of asthma attacks per patient than the average in the treatment group.
Emilio opened a food truck last year where he sells burritos, and his business has stabilized over the last 3 months. Over that 3 month period, he has made $11,000 while working 625 hours. Emilio’s average hourly earnings provides a useful statistic for evaluating whether his venture is, at least from a financial perspective, worth it: $\$11000 / 625 \text{ hours} = \$17.60 \text{ per hour}\text{.}$ By knowing his average hourly wage, Emilio now has put his earnings into a standard unit that is easier to compare with many other jobs that he might consider.

Example 2.1.12. Weighted means.

Suppose we want to compute the average income per person in the US. To do so, we might first think to take the mean of the per capita incomes across the 3,142 counties in the county data set. What would be a better approach?

Solution.

The county data set is special in that each county actually represents many individual people. If we were to simply average across the income variable, we would be treating counties with 5,000 and 5,000,000 residents equally in the calculations. Instead, we should compute the total income for each county, add up all the counties’ totals, and then divide by the number of people in all the counties. If we completed these steps with the county data, we would find that the per capita income for the US is $30,861. Had we computed the simple mean of per capita income across counties, the result would have been just $26,093! This example used what is called a weighted mean. For more information on this topic, check out online supplements regarding weighted means.

Subsection 2.1.3 Histograms and shape

Dot plots show the exact value for each observation. This is useful for small data sets, but they can become hard to read with larger samples. Rather than showing the value of each observation, we prefer to think of the value as belonging to a bin. For example, in the loan50 data set, we created a table of counts for the number of loans with interest rates between 5.0% and 7.5%, then the number of loans with rates between 7.5% and 10.0%, and so on. Observations that fall on the boundary of a bin (e.g. 10.00%) are allocated to the lower bin. This tabulation is shown in Table 2.1.13. These binned counts are plotted as bars in Figure 2.1.14 into what is called a histogram, which resembles a more heavily binned version of the stacked dot plot.

Table 2.1.13. Counts for the binned interest_rate data

Interest Rate	5.0%-7.5%	7.5%-10.0%	10.0%-12.5%	12.5%-15.0%	$\cdots$	25.0%-27.5%
Count	11	15	8	4	$\cdots$	1

A histogram with a horizontal axis of "Interest Rate" and a vertical axis showing the frequency of occurrence of different bins of interest rate. The first bin is from 5%-7.5% with a frequency (count) of 11 observations, 7.5%-10% has a frequency of 15, 10%-12.5% has 8, 12.5%-15% has 4, 15%-17.5% has 5, 17.5%-20% has 4, and then the 20%-22.5%, 22.5%-25%, and 25%-27.5% bins each have a frequency of 1. — Figure 2.1.14. A histogram of `interest_rate`. This distribution is strongly skewed to the right.

Histograms provide a view of the data density. Higher bars represent where the data are relatively more common. For instance, there are many more loans with rates between 5% and 10% than loans with rates between 20% and 25% in the data set. The bars make it easy to see how the density of the data changes relative to the interest rate.

Histograms are especially convenient for understanding the shape of the data distribution. Figure 2.1.14 suggests that most loans have rates under 15%, while only a handful of loans have rates above 20%. When data trail off to the right in this way and has a longer right tail, the shape is said to be right skewed.

Other ways to describe data that are right skewed: skewed to the right, skewed to the high end, or skewed to the positive end.

Data sets with the reverse characteristic — a long, thinner tail to the left — are said to be left skewed. We also say that such a distribution has a long left tail. Data sets that show roughly equal trailing off in both directions are called symmetric.

Long tails to identify skew.

When data trail off in one direction, the distribution has a long tail. If a distribution has a long left tail, it is left skewed. If a distribution has a long right tail, it is right skewed.

Checkpoint 2.1.15. Identifying skew.

Take a look at the dot plots in earlier figures. Can you see the skew in the data? Is it easier to see the skew in this histogram or the dot plots?

Solution.

The skew is visible in all three plots, though the flat dot plot is the least useful. The stacked dot plot and histogram are helpful visualizations for identifying skew.

Checkpoint 2.1.16. Histogram limitations.

Besides the mean (since it was labeled), what can you see in the dot plots that you cannot see in the histogram?

Solution.

The interest rates for individual loans.

In addition to looking at whether a distribution is skewed or symmetric, histograms can be used to identify modes. A mode is represented by a prominent peak in the distribution. There is only one prominent peak in the histogram of loan_amount.

A definition of mode sometimes taught in math classes is the value with the most occurrences in the data set. However, for many real-world data sets, it is common to have no observations with the same value in a data set, making this definition impractical in data analysis.

Figure 2.1.17 shows histograms that have one, two, or three prominent peaks. Such distributions are called unimodal, bimodal, and multimodal, respectively. Any distribution with more than 2 prominent peaks is called multimodal. Notice that there was one prominent peak in the unimodal distribution with a second less prominent peak that was not counted since it only differs from its neighboring bins by a few observations.

Example 2.1.18. Identifying modality.

Figure 2.1.14 reveals only one prominent mode in the interest rate. Is the distribution unimodal, bimodal, or multimodal?

Solution.

Unimodal. Remember that uni stands for 1 (think unicycles). Similarly, bi stands for 2 (think bicycles). We’re hoping a multicycle will be invented to complete this analogy.

Checkpoint 2.1.19. Expected modes in height data.

Height measurements of young students and adult teachers at a K-3 elementary school were taken. How many modes would you expect in this height data set?

Solution.

There might be two height groups visible in the data set: one of the students and one of the adults. That is, the data are probably bimodal.

Looking for modes isn’t about finding a clear and correct answer about the number of modes in a distribution, which is why prominent is not rigorously defined in this book. The most important part of this examination is to better understand your data.

Subsection 2.1.4 Variance and standard deviation

The mean was introduced as a method to describe the center of a data set, and variability in the data is also important. Here, we introduce two measures of variability: the variance and the standard deviation. Both of these are very useful in data analysis, even though their formulas are a bit tedious to calculate by hand. The standard deviation is the easier of the two to comprehend, and it roughly describes how far away the typical observation is from the mean.

We call the distance of an observation from its mean its deviation. Below are the deviations for the 1$^{st}\text{,}$ 2$^{nd}\text{,}$ 3$^{rd}\text{,}$ and 50$^{th}$ observations in the interest_rate variable:

\begin{align*} x_1 - \bar{x} \amp = 10.90 - 11.57 = -0.67\\ x_2 - \bar{x} \amp = 9.92 - 11.57 = -1.65\\ x_3 - \bar{x} \amp = 26.30 - 11.57 = 14.73\\ \amp \vdots\\ x_{50} - \bar{x} \amp = 6.08 - 11.57 = -5.49 \end{align*}

If we square these deviations and then take an average, the result is equal to the sample variance, denoted by $s^2\text{:}$

\begin{align*} s^2 \amp = \frac{(-0.67)^2 + (-1.65)^2 + (14.73)^2 + \cdots + (-5.49)^2}{50-1}\\ \amp = \frac{0.45 + 2.72 + 216.97 + \cdots + 30.14}{49}\\ \amp = 25.52 \end{align*}

We divide by $n - 1\text{,}$ rather than dividing by $n\text{,}$ when computing a sample’s variance; there’s some mathematical nuance here, but the end result is that doing this makes this statistic slightly more reliable and useful.

Notice that squaring the deviations does two things. First, it makes large values relatively much larger, seen by comparing $(-0.67)^2\text{,}$ $(-1.65)^2\text{,}$ $(14.73)^2\text{,}$ and $(-5.49)^2\text{.}$ Second, it gets rid of any negative signs.

The standard deviation is defined as the square root of the variance:

\begin{gather*} s = \sqrt{25.52} = 5.05 \end{gather*}

While often omitted, a subscript of $_x$ may be added to the variance and standard deviation, i.e. $s_x^2$ and $s_x\text{,}$ if it is useful as a reminder that these are the variance and standard deviation of the observations represented by $x_1\text{,}$ $x_2\text{,}$ ..., $x_n\text{.}$

Variance and standard deviation.

The variance is the average squared distance from the mean. The standard deviation is the square root of the variance. The standard deviation is useful when considering how far the data are distributed from the mean.

The standard deviation represents the typical deviation of observations from the mean. Usually about 70% of the data will be within one standard deviation of the mean and about 95% will be within two standard deviations. However, these percentages are not strict rules.

Like the mean, the population values for variance and standard deviation have special symbols: $\sigma^2$ for the variance and $\sigma$ for the standard deviation. The symbol $\sigma$ is the Greek letter sigma.

A dot plot of 50 observations is shown with values ranging from about 5% to 26%. The data set is the same as that shown in earlier dot plots, where the data is more dense from 5% to about 11%, has medium density from about 12% to 20%, and then there are a few more values scattered in the 20% to 27% range. Shading is shown to represent the regions within 1, 2, and 3 standard deviations. The region within 1 standard deviation is from 6.5% to 16.7%, representing 34 of the 50 data points. The region within 2 standard deviations runs left off of the chart (but would be from about 1.4%) to 21.8% and contains 48 of the 50 data points. The third standard deviation is shown to extend out to 26.9%, and all 50 observations are contained within the 3 standard deviations. — Figure 2.1.20. Standard deviations in the interest rate distribution

For the interest_rate variable, 34 of the 50 loans (68%) had interest rates within 1 standard deviation of the mean, and 48 of the 50 loans (96%) had rates within 2 standard deviations. Usually about 70% of the data are within 1 standard deviation of the mean and 95% within 2 standard deviations, though this is far from a hard rule.

Three histograms are shown (upper, middle, lower). Each distribution also shows shading -- dark gray between -1 to 1, lighter gray between -2 and 2, and light gray between -3 and 3, and then very light gray further out. The upper plot shows only two bins with non-zero values and of equal height at -1 and 1. The middle plot shows a bell-shaped curve, where most of the higher bin values are between -1 and 1, middling heights are between -2 to -1 and 1 to 2, and the data trails off in each direction with ever-smaller values further out. The lower histogram shows no data below about -1.6, a quick increase to a peak at about -0.7 and then a slow decline of values to about half the max height at 1 and further trails off to ever smaller values to a horizontal location of 3 and beyond. — Figure 2.1.21. Three very different population distributions with the same mean $\mu=0$ and standard deviation $\sigma=1$

Checkpoint 2.1.22. Importance of shape description.

The concept of shape of a distribution was introduced earlier. A good description of the shape of a distribution should include modality and whether the distribution is symmetric or skewed to one side. Using Figure 2.1.21 as an example, explain why such a description is important.

Solution.

Figure 2.1.21 shows three distributions that look quite different, but all have the same mean, variance, and standard deviation. Using modality, we can distinguish between the first plot (bimodal) and the last two (unimodal). Using skewness, we can distinguish between the last plot (right skewed) and the first two. While a picture, like a histogram, tells a more complete story, we can use modality and shape (symmetry/skew) to characterize basic information about a distribution.

Example 2.1.23. Describing a distribution.

Describe the distribution of the interest_rate variable using the histogram in Figure 2.1.14. The description should incorporate the center, variability, and shape of the distribution, and it should also be placed in context. Also note any especially unusual cases.

Solution.

The distribution of interest rates is unimodal and skewed to the high end. Many of the rates fall near the mean at 11.57%, and most fall within one standard deviation (5.05%) of the mean. There are a few exceptionally large interest rates in the sample that are above 20%.

In practice, the variance and standard deviation are sometimes used as a means to an end, where the "end" is being able to accurately estimate the uncertainty associated with a sample statistic. For example, in later chapters the standard deviation is used in calculations that help us understand how much a sample mean varies from one sample to the next.

Subsection 2.1.5 Box plots, quartiles, and the median

A box plot summarizes a data set using five statistics while also plotting unusual observations. Figure 2.1.24 provides a vertical dot plot alongside a box plot of the interest_rate variable from the loan50 data set.

What is shown is a dot plot adjacent to what is called a "box plot". The data values are the same ones used in past dot plots, where the data shows greatest density from 5% to 11%, moderate density from 12% to 20%, and then a few more values at about 22%, 25%, and 26%. The box plot adjacent to the data shows a box that would encapsulate the middle 50% of the data, from about 8% to 13%. The median is also annotated with a line through the center of the box. From here, the data extend out with "whiskers" up to a distance up to 1.5 times IQR below and above the box to capture as much data as possible. There are two observations that extend beyond this range at 25% and 26%. — Figure 2.1.24. A vertical dot plot next to a labeled box plot for the interest rates of the 50 loans

The first step in building a box plot is drawing a dark line denoting the median, which splits the data in half. Figure 2.1.24 shows 50% of the data falling below the median and other 50% falling above the median. There are 50 loans in the data set (an even number) so the data are perfectly split into two groups of 25. We take the median in this case to be the average of the two observations closest to the 50$^{th}$ percentile, which happen to be the same value in this data set: $(9.93\% + 9.93\%) / 2 = 9.93\%\text{.}$ When there are an odd number of observations, there will be exactly one observation that splits the data into two halves, and in such a case that observation is the median (no average needed).

Median: the number in the middle.

If the data are ordered from smallest to largest, the median is the observation right in the middle. If there are an even number of observations, there will be two values in the middle, and the median is taken as their average.

The second step in building a box plot is drawing a rectangle to represent the middle 50% of the data. The total length of the box, shown vertically in Figure 2.1.24, is called the interquartile range (IQR, for short). It, like the standard deviation, is a measure of variability in data. The more variable the data, the larger the standard deviation and IQR tend to be. The two boundaries of the box are called the first quartile (the 25$^{th}$ percentile, i.e. 25% of the data fall below this value) and the third quartile (the 75$^{th}$ percentile), and these are often labeled $Q_1$ and $Q_3\text{,}$ respectively.

Interquartile range (IQR).

The IQR is the length of the box in a box plot. It is computed as

\begin{gather*} IQR = Q_3 - Q_1 \end{gather*}

where $Q_1$ and $Q_3$ are the 25$^{th}$ and 75$^{th}$ percentiles.

Checkpoint 2.1.25. Data between quartiles.

What percent of the data fall between $Q_1$ and the median? What percent is between the median and $Q_3\text{?}$

Solution.

Since $Q_1$ and $Q_3$ capture the middle 50% of the data and the median splits the data in the middle, 25% of the data fall between $Q_1$ and the median, and another 25% falls between the median and $Q_3\text{.}$

Extending out from the box, the whiskers attempt to capture the data outside of the box. However, their reach is never allowed to be more than $1.5 \times IQR\text{.}$ They capture everything within this reach. In Figure 2.1.24, the upper whisker does not extend to the last two points, which is beyond $Q_3 + 1.5 \times IQR\text{,}$ and so it extends only to the last point below this limit. The lower whisker stops at the lowest value, 5.31%, since there is no additional data to reach; the lower whisker’s limit is not shown in the figure because the plot does not extend down to $Q_1 - 1.5 \times IQR\text{.}$ In a sense, the box is like the body of the box plot and the whiskers are like its arms trying to reach the rest of the data.

Any observation lying beyond the whiskers is labeled with a dot. The purpose of labeling these points — instead of extending the whiskers to the minimum and maximum observed values — is to help identify any observations that appear to be unusually distant from the rest of the data. Unusually distant observations are called outliers. In this case, it would be reasonable to classify the interest rates of 24.85% and 26.30% as outliers since they are numerically distant from most of the data.

Outliers are extreme.

An outlier is an observation that appears extreme relative to the rest of the data.

Examining data for outliers serves many useful purposes, including:

Identifying strong skew in the distribution.
Identifying possible data collection or data entry errors.
Providing insight into interesting properties of the data.

Checkpoint 2.1.26. Estimating quartiles from a box plot.

Using Figure 2.1.24, estimate the following values for interest_rate in the loan50 data set: (a) $Q_1\text{,}$ (b) $Q_3\text{,}$ and (c) IQR.

Solution.

These visual estimates will vary a little from one person to the next: $Q_1 \approx$ 8%, $Q_3 \approx$ 14%, $IQR = Q_3 - Q_1 \approx$ 6%. (The true values: $Q_1 = 7.96\%\text{,}$ $Q_3 = 13.72\%\text{,}$ $IQR = 5.76\%\text{.}$)

Subsection 2.1.6 Robust statistics

How are the sample statistics of the interest_rate data set affected by the observation, 26.3%? What would have happened if this loan had instead been only 15%? What would happen to these summary statistics if the observation at 26.3% had been even larger, say 35%? These scenarios are plotted alongside the original data in Figure 2.1.27, and sample statistics are computed under each scenario in Table 2.1.28.

Three dot plots are shown in the same plot. The largest observation from the original data set (discussed in previous dot plots) at about 26% is moved to 15% in the second dot plot and instead to 35% in the third dot plot. — Figure 2.1.27. Dot plots of the original interest rate data and two modified data sets

Table 2.1.28. Comparison of statistics under different scenarios

Scenario	Robust		Not Robust
	Median	IQR	$\bar{x}$	$s$
Original `interest_rate` data	9.93%	5.76%	11.57%	5.05%
Move 26.3% $\to$ 15%	9.93%	5.76%	11.34%	4.61%
Move 26.3% $\to$ 35%	9.93%	5.76%	11.74%	5.68%

Checkpoint 2.1.29. Comparing robustness of statistics.

(a) Which is more affected by extreme observations, the mean or median? Table 2.1.28 may be helpful. (b) Is the standard deviation or IQR more affected by extreme observations?

Solution.

(a) Mean is affected more. (b) Standard deviation is affected more. Complete explanations are provided below.

The median and IQR are called robust statistics because extreme observations have little effect on their values: moving the most extreme value generally has little influence on these statistics. On the other hand, the mean and standard deviation are more heavily influenced by changes in extreme observations, which can be important in some situations.

Example 2.1.30. Stability of robust statistics.

The median and IQR did not change under the three scenarios in Table 2.1.28. Why might this be the case?

Solution.

The median and IQR are only sensitive to numbers near $Q_1\text{,}$ the median, and $Q_3\text{.}$ Since values in these regions are stable in the three data sets, the median and IQR estimates are also stable.

Checkpoint 2.1.31. Choosing between mean and median.

The distribution of loan amounts in the loan50 data set is right skewed, with a few large loans lingering out into the right tail. If you were wanting to understand the typical loan size, should you be more interested in the mean or median?

Solution.

Answers will vary! If we’re looking to simply understand what a typical individual loan looks like, the median is probably more useful. However, if the goal is to understand something that scales well, such as the total amount of money we might need to have on hand if we were to offer 1,000 loans, then the mean would be more useful.

Subsection 2.1.7 Transforming data (special topic)

When data are very strongly skewed, we sometimes transform them so they are easier to model.

Two histograms are shown side by side. The first histogram has a horizontal axis of Population with possible data ranging from 0 to about 10 million. The first bar representing 0 to 400,000 shows a frequency (bar height) of about 3000, the second bar for 400,000 to 800,000 shows about frequency of about 100. All other bars are sufficiently small that they are virtually indistinguishable from 0. The second histogram shows the horizontal axis represents log-base-10 of the population. The horizontal axis runs from about 2 to 7, and frequency (bin/box height) peaks at a little over 1000. The data show an approximate bell shape, peaking in the middle between 4 to 4.5, then showing lower frequencies the further out from 4-4.5 with frequencies being close to zero outside of 2.5 to 6.5. — Figure 2.1.32. County population distributions: (a) original data showing extreme skew, (b) log-transformed data

Example 2.1.33. Issues with extreme skew.

Consider the histogram of county populations shown in Figure 2.1.32 (left panel), which shows extreme skew. What isn’t useful about this plot?

Solution.

Nearly all of the data fall into the left-most bin, and the extreme skew obscures many of the potentially interesting details in the data.

There are some standard transformations that may be useful for strongly right skewed data where much of the data is positive but clustered near zero. A transformation is a rescaling of the data using a function. For instance, a plot of the logarithm (base 10) of county populations results in the new histogram in Figure 2.1.32 (right panel). This data is symmetric, and any potential outliers appear much less extreme than in the original data set. By reigning in the outliers and extreme skew, transformations like this often make it easier to build statistical models against the data.

Transformations can also be applied to one or both variables in a scatterplot. A scatterplot of the population change from 2010 to 2017 against the population in 2010 is shown in Figure 2.1.34. In the first scatterplot, it’s hard to decipher any interesting patterns because the population variable is so strongly skewed. However, if we apply a log$_{10}$ transformation to the population variable, as shown in the second panel, a positive association between the variables is revealed.

Two scatterplots are shown side by side. The first scatterplot has population on the horizontal axis (ranging from 0 to 10 million) and population change as a percent on the vertical axis (ranging from -35% to positive 40%). The data is particularly concentrated on the left of the graph below 1 million. There is no discernible trend in the data. The second scatterplot has log-base-10 of the population on the horizontal axis (ranging from 2 to 7) and population change as a percent on the vertical axis. The data is well distributed and shows a cloud of points with a slight upward trend. — Figure 2.1.34. Scatterplots of population change vs. population: (a) original data, (b) log-transformed population

Transformations other than the logarithm can be useful, too. For instance, the square root and inverse are commonly used by data scientists. Common goals in transforming data are to see the data structure differently, reduce skew, assist in modeling, or straighten a nonlinear relationship in a scatterplot.

Subsection 2.1.8 Mapping data (special topic)

The county data set offers many numerical variables that we could plot using dot plots, scatterplots, or box plots, but these miss the true nature of the data. Rather, when we encounter geographic data, we should create an intensity map, where colors are used to show higher and lower values of a variable. Figures throughout this book demonstrate a variety of intensity maps for county-level data including poverty rate (poverty), unemployment rate (unemployment_rate), homeownership rate (homeownership), and median household income (median_hh_income). The color key indicates which colors correspond to which values. The intensity maps are not generally very helpful for getting precise values in any given county, but they are very helpful for seeing geographic trends and generating interesting research questions or hypotheses.

An intensity map of the United States showing the poverty rate by county. Darker colors indicate higher poverty rates. Notable patterns show higher poverty rates in the deep south, Arizona, New Mexico, and parts of Kentucky. — (a) (a) Poverty rate (percent)

An intensity map of the United States showing the unemployment rate by county. Darker colors indicate higher unemployment rates. The pattern follows similar trends to the poverty rate map. — (a) (a) Poverty rate (percent)

Example 2.1.36. Features in poverty and unemployment intensity maps.

What interesting features are evident in the poverty and unemployment_rate intensity maps?

Solution.

Poverty rates are evidently higher in a few locations. Notably, the deep south shows higher poverty rates, as does much of Arizona and New Mexico. High poverty rates are evident in the Mississippi flood plains a little north of New Orleans and also in a large section of Kentucky.

The unemployment rate follows similar trends, and we can see correspondence between the two variables. In fact, it makes sense for higher rates of unemployment to be closely related to poverty rates. One observation that stands out when comparing the two maps: the poverty rate is much higher than the unemployment rate, meaning while many people may be working, they are not making enough to break out of poverty.

Checkpoint 2.1.37. Features in median household income map.

What interesting features are evident in the median_hh_income intensity map?

Solution.

Note: answers will vary. There is some correspondence between high earning and metropolitan areas, where we can see darker spots (higher median household income), though there are several exceptions. You might look for large cities you are familiar with and try to spot them on the map as dark spots.

Exercises 2.1.9 Exercises

1. Mammal life spans.

Data were collected on life spans (in years) and gestation lengths (in days) for 62 mammals. A scatterplot of life span versus length of gestation is shown below.

A scatterplot of 62 points is shown. The variable "Gestation" is shown along the horizontal axis with a range of 0 days to about 650 days. The variable "Life Span" is shown along the vertical axis with a range of 0 years to 100 years. A large cluster of points is shown between 0 to 250 gestational days and 0 to 30 years. Outside of this cluster, there is one point at approximately (10, 50). There is another cluster of points between 250 and 450 gestational days and 25 and 50 years. Beyond the points so far described are three points located at (250 days, 100 years), (640 days, 70 years), and (650 days, 45 years). — Figure 2.1.38. Scatterplot of life span versus gestation length for 62 mammals

What type of an association is apparent between life span and length of gestation?
What type of an association would you expect to see if the axes of the plot were reversed, i.e. if we plotted length of gestation versus life span?
Are life span and length of gestation independent? Explain your reasoning.

2. Associations.

Indicate which of the plots show (a) a positive association, (b) a negative association, or (c) no association. Also determine if the positive and negative associations are linear or nonlinear. Each part may refer to more than one plot.

Four scatterplots are shown and are labeled 1, 2, 3, and 4. There are no labeled axes on these plots, as only the patterns of the points in the plots are important for this exercise. In plot 1, the points are moderately clustered in the lower left corner of the plot and remain clustered looking further right in the plot, where the points follow steadily upwards to the top-right corner. In plot 2, the points appear to be scattered almost randomly all around the rectangular plotting region. Plot 3 shows points clustered tightly in the lower left corner and the data points remain clustered even as moving right, with the data trending upwards gradually and then more steeply as it reaches the right side of the plot. Plot 4, when looking on the left portion, shows data moderately clustered in the upper-left corner, which then steadily trends downward to the lower-right corner of the plot. — Figure 2.1.39. Four scatterplots showing different types of associations

3. Reproducing bacteria.

Suppose that there is only sufficient space and nutrients to support one million bacterial cells in a petri dish. You place a few bacterial cells in this petri dish, allow them to reproduce freely, and record the number of bacterial cells in the dish over time. Sketch a plot representing the relationship between number of bacterial cells and time.

4. Office productivity.

Office productivity is relatively low when the employees feel no stress about their work or job security. However, high levels of stress can also lead to reduced employee productivity. Sketch a plot to represent the relationship between stress and productivity.

5. Parameters and statistics.

Identify which value represents the sample mean and which value represents the claimed population mean.

American households spent an average of about $52 in 2007 on Halloween merchandise such as costumes, decorations and candy. To see if this number had changed, researchers conducted a new survey in 2008 before industry numbers were reported. The survey included 1,500 households and found that average Halloween spending was $58 per household.
The average GPA of students in 2001 at a private university was 3.37. A survey on a sample of 203 students from this university yielded an average GPA of 3.59 a decade later.

6. Sleeping in college.

A recent article in a college newspaper stated that college students get an average of 5.5 hrs of sleep each night. A student who was skeptical about this value decided to conduct a survey by randomly sampling 25 students. On average, the sampled students slept 6.25 hours per night. Identify which value represents the sample mean and which value represents the claimed population mean.

7. Days off at a mining plant.

Workers at a particular mining site receive an average of 35 days paid vacation, which is lower than the national average. The manager of this plant is under pressure from a local union to increase the amount of paid time off. However, he does not want to give more days off to the workers because that would be costly. Instead he decides he should fire 10 employees in such a way as to raise the average number of days off that are reported by his employees. In order to achieve this goal, should he fire employees who have the most number of days off, least number of days off, or those who have about the average number of days off?

8. Medians and IQRs.

For each part, compare distributions (1) and (2) based on their medians and IQRs. You do not need to calculate these statistics; simply state how the medians and IQRs compare. Make sure to explain your reasoning.

(1) 3, 5, 6, 7, 9

(2) 3, 5, 6, 7, 20
(1) 3, 5, 6, 7, 9

(2) 3, 5, 7, 8, 9
(1) 1, 2, 3, 4, 5

(2) 6, 7, 8, 9, 10
(1) 0, 10, 50, 60, 100

(2) 0, 100, 500, 600, 1000

9. Means and SDs.

For each part, compare distributions (1) and (2) based on their means and standard deviations. You do not need to calculate these statistics; simply state how the means and the standard deviations compare. Make sure to explain your reasoning. Hint: It may be useful to sketch dot plots of the distributions.

(1) 3, 5, 5, 5, 8, 11, 11, 11, 13

(2) 3, 5, 5, 5, 8, 11, 11, 11, 20
(1) -20, 0, 0, 0, 15, 25, 30, 30

(2) -40, 0, 0, 0, 15, 25, 30, 30
(1) 0, 2, 4, 6, 8, 10

(2) 20, 22, 24, 26, 28, 30
(1) 100, 200, 300, 400, 500

(2) 0, 50, 300, 550, 600

10. Mix-and-match.

Describe the distribution in the histograms below and match them to the box plots.

Six plots are shown, three histograms labeled a, b, and c, and 3 box plots labeled 1, 2, and 3. Plot (a) shows a histogram with horizontal range for the data of 50 to 70. The data are bell-shaped and centered in the plot, with only a little data reaching close to the lower end of 50 and the upper end of 70. Plot (b) shows another histogram, where the horizontal axis extends from 0 to 100, and the histogram bins are relatively steady in their height in the first bin near zero across the plot to the last bin near 100. Plot (c) is a histogram with a horizontal axis running from 0 to about 7. The first few bins rise quickly to a peak at the horizontal location of 1 and then fall until reaching 2 and then decline much more gradually until about 4, where the bins are near zero and stay near zero for larger values. Plot (1) is a box plot. The vertical axis for the box plot spans from 0 to about 7. The lower whisker is at 0, the box spans about 1 to 2, with the center line for the box plot at about 1.4. The upper whisker extends up to about 3.5, and then there are several points marked individually extending further upwards to about 7. Plot (2) is a box plot with a vertical axis spanning about 50 to 70. The box for the plot is centered at 60 and runs from about 58 to 62. The whiskers span about 52 to 68. There are 2 individually points shown below 52 and about 4 points shown above 68. Plot (3) is a box plot spanning from 0 to 100. The box is centered at about 50, and the box spans about 25 to 75. The whiskers extend down to 0 and up to 100. — Figure 2.1.40. Three histograms and three box plots to match

11. Air quality.

Daily air quality is measured by the air quality index (AQI) reported by the Environmental Protection Agency. This index reports the pollution level and what associated health effects might be a concern. The index is calculated for five major air pollutants regulated by the Clean Air Act and takes values from 0 to 300, where a higher value indicates lower air quality. AQI was reported for a sample of 91 days in 2011 in Durham, NC. The relative frequency histogram below shows the distribution of the AQI values on these days.

A histogram of "Daily AQI", where the horizontal axis for the data runs from about 5 to 65. The bin width is 5, there are 12 bins from 5 to 60, and the vertical axis shows proportions. The heights of the 12 bins, in order from left to right, are about 0.02 (for the bin 5 to 10), 0.06, 0.20, 0.06, 0.20, 0.15, 0.07, 0.04, 0.07, 0.08, 0.03, and 0.02 for the last bin for 60 to 65. — Figure 2.1.41. Relative frequency histogram of daily AQI values in Durham, NC

Estimate the median AQI value of this sample.
Would you expect the mean AQI value of this sample to be higher or lower than the median? Explain your reasoning.
Estimate Q1, Q3, and IQR for the distribution.
Would any of the days in this sample be considered to have an unusually low or high AQI? Explain your reasoning.

12. Median vs. mean.

Estimate the median for the 400 observations shown in the histogram, and note whether you expect the mean to be higher or lower than the median.

A histogram is shown, with the horizontal axis for the data runs from 40 to 100, with a bin size width of 5. The frequencies for the bins are as follows, where counts are approximate: 2 (for bin 40 to 45), 4, 2, 10, 20, 25, 50, 75, 70, 85, 45, and 10 for the last bin from 95 to 100. — Figure 2.1.42. Histogram of 400 observations

13. Histograms vs. box plots.

Compare the two plots below. What characteristics of the distribution are apparent in the histogram and not in the box plot? What characteristics are apparent in the box plot but not in the histogram?

Two plots are shown, first a histogram and second a box plot. The data for each plot runs from about 0 to 30. The histogram has bins of width 2. The bins, starting at the lower values, shows an initial peak at about the horizontal location of 5, then declines to near the horizontal axis at 10, before rising again between 10 and 14, and then lower values again for bins between 15 to 30. The box plot has its box centered at 10 and runs from about 5 to 12. The whiskers reach out to about 2 and up to about 22. There are a few points above the upper whisker. — Figure 2.1.43. Histogram and box plot of the same data

14. Facebook friends.

Facebook data indicate that 50% of Facebook users have 100 or more friends, and that the average friend count of users is 190. What do these findings suggest about the shape of the distribution of number of friends of Facebook users?

15. Distributions and appropriate statistics, Part I.

For each of the following, state whether you expect the distribution to be symmetric, right skewed, or left skewed. Also specify whether the mean or median would best represent a typical observation in the data, and whether the variability of observations would be best represented using the standard deviation or IQR. Explain your reasoning.

Number of pets per household.
Distance to work, i.e. number of miles between work and home.
Heights of adult males.

16. Distributions and appropriate statistics, Part II.

Housing prices in a country where 25% of the houses cost below $350,000, 50% of the houses cost below $450,000, 75% of the houses cost below $1,000,000 and there are a meaningful number of houses that cost more than $6,000,000.
Housing prices in a country where 25% of the houses cost below $300,000, 50% of the houses cost below $600,000, 75% of the houses cost below $900,000 and very few houses that cost more than $1,200,000.
Number of alcoholic drinks consumed by college students in a given week. Assume that most of these students don’t drink since they are under 21 years old, and only a few drink excessively.
Annual salaries of the employees at a Fortune 500 company where only a few high level executives earn much higher salaries than all the other employees.

17. Income at the coffee shop.

The first histogram below shows the distribution of the yearly incomes of 40 patrons at a college coffee shop. Suppose two new people walk into the coffee shop: one making $225,000 and the other $250,000. The second histogram shows the new income distribution. Summary statistics are also provided.

Two histograms are shown and are labeled 1 and 2. Plot 1 has a horizontal axis from $60,000 to $70,000. The bins, from left to right, generally rise steadily from frequencies of 2 to 3 at $60,000 to $62,000 and up to a peak of about 7 to 8 between $64,000 to $66,000. From here, the bin counts steadily decline down to about 2 for the last bin, $69,000 to $70,000. Plot (2) shows a histogram, with the horizontal axis running from about $60,000 to $260,000. The width of the bins are $1,000, like in the first plot, and the first 10 bins reflect those described in Plot (1). Two additional bins are shown at about $225,000 and $250,000, each with a bin height of 1. — Figure 2.1.44. Income distributions at a coffee shop before and after two high earners arrive

Table 2.1.45. Summary statistics for income distributions

	(1)	(2)
n	40	42
Min.	60,680	60,680
1st Qu.	63,620	63,710
Median	65,240	65,350
Mean	65,090	73,300
3rd Qu.	66,160	66,540
Max.	69,890	250,000
SD	2,122	37,321

Would the mean or the median best represent what we might think of as a typical income for the 42 patrons at this coffee shop? What does this say about the robustness of the two measures?
Would the standard deviation or the IQR best represent the amount of variability in the incomes of the 42 patrons at this coffee shop? What does this say about the robustness of the two measures?

18. Midrange.

The midrange of a distribution is defined as the average of the maximum and the minimum of that distribution. Is this statistic robust to outliers and extreme skew? Explain your reasoning.

19. Commute times.

The US census collects data on time it takes Americans to commute to work, among many other variables. The histogram below shows the distribution of average commute times in 3,142 US counties in 2010. Also shown below is a spatial intensity map of the same data.

A histogram is shown, where the horizontal axis is for the variable "Mean work travel in minutes" spans approximately 0 to 50, with the vertical axis representing frequency with a peak value of about 200. The bins start with small bin heights on the left side, and the bin heights start increasing at about 10 and then rapidly ascend by 15 before leveling off and reaching a peak at about 22. The bins begin declining again about 24 gradually and then more rapidly around 26 to 29. At 30, the bins continue declining, but at a slower pace, before they level off near a height of 0 at about 35. — (a) Histogram

A spatial intensity map is shown of the United States. The legend for the shading runs from values of 4 to about 33. The shading for the eastern half of the country suggests slightly higher values, while the western portion of the upper midwest (North Dakota, South Dakota, and Nebraska) shows lower values. Other specific regions that show patterns of higher values than surrounding areas are in lower Florida and northern California. — (a) Histogram

Describe the numerical distribution and comment on whether or not a log transformation may be advisable for these data.
Describe the spatial distribution of commuting times using the map above.

20. Hispanic population.

The US census collects data on race and ethnicity of Americans, among many other variables. The histogram below shows the distribution of the percentage of the population that is Hispanic in 3,142 counties in the US in 2010. Also shown is a histogram of logs of these values.

A histogram is shown for the variable "Percent Hispanic", where the horizontal axis runs from 0 to 100. The first bin, from 0 to 5, is dramatically higher than all other bins at about 2000. From here, the bins descend rapidly: about 500 between 5 and 10, 200 between 10 and 15, 100 between 15 and 20, then then trail off with the bins being nearly indistinguishable from a height of 0 for bins about 50%. — (a) Histogram of percent Hispanic

A histogram is shown for the transformed variable, "log-base-e of Percent Hispanic", where the horizontal axis runs from about -2.5 to 4.5. The bins are very close to 0 in frequency until -1, then the rise slightly to about -0.5, before sharply rising to a peak at about 0.5. From here, the bins steadily decline towards a frequency of 0 at the horizontal location of 4.5. — (a) Histogram of percent Hispanic

A spatial intensity map is shown of the United States. The legend for the shading runs from values of 0% to a peak of "greater than 40%". A large portion of the eastern and central portion of the country -- east of Texas, east of Colorado, east of Utah, and east of Idaho -- is shaded mostly with values below 10%. Florida is an exception to this rule, where a handful of counties show higher values. Higher values are particularly prominent in Texas, New Mexico, Arizona, and California, which mostly shows shading for values of at least 20%. Nevada, Idaho, Oregon, and Washington shows values averaging around 10-20%. — Figure 2.1.48. Spatial intensity map of percent Hispanic population by county

Describe the numerical distribution and comment on why we might want to use log-transformed values in analyzing or modeling these data.
What features of the distribution of the Hispanic population in US counties are apparent in the map but not in the histogram? What features are apparent in the histogram but not the map?
Is one visualization more appropriate or helpful than the other? Explain your reasoning.

Prev Top Next

Scenario	Robust		Not Robust
	Median	IQR	\(\bar{x}\)	\(s\)
Original `interest_rate` data	9.93%	5.76%	11.57%	5.05%
Move 26.3% \(\to\) 15%	9.93%	5.76%	11.34%	4.61%
Move 26.3% \(\to\) 35%	9.93%	5.76%	11.74%	5.68%