Introduction to Statistical Thinking

A point estimator of the expectation of a measurement is the sample average of the variable that is associated with the measurement. A confidence interval is an interval of numbers that is likely to contain the parameter value. A natural interval to consider is an interval centered at the sample average \(\bar x\text{.}\) The interval is set to have a width that assures the inclusion of the parameter value in the prescribed probability, namely the confidence level.

Consider the confidence interval for the expectation. The structure of the confidence interval of confidence level 95% is \([\bar x - 1.96 \cdot s/\sqrt{n}, \bar x + 1.96 \cdot s/\sqrt{n}]\text{,}\) where \(s\) is the estimated standard deviation of the measurement (namely, the sample standard deviation) and \(n\) is the sample size. This interval may be expressed in the form:

As an illustration, let us construct a 0.95-confidence interval for the expected price of a car. :

cars <- read.csv("_data/cars.csv")
x.bar <- mean(cars$price,na.rm=TRUE)
s <- sd(cars$price,na.rm=TRUE)
n <- 201

In the first line of code the data in the file “cars.csv" is stored in a data frame called “cars". In the second line the average \(\bar x\) is computed for the variable “price" in the data frame “cars". This average is stored under the name “x.bar". Recall that the variable “price" contains 4 missing values. Hence, in order to compute the average of the non-missing values we should set a “TRUE" value to the argument “na.rm". The sample standard deviation “s" is computed in the third line by the application of the function “sd". We set once more the argument “na.rm=TRUE" in order to deal with the missing values. Finally, in the last line we store the sample size “n", the number of non-missing values.

Let us compute the lower and the upper limits of the confidence interval for the expectation of the price:

x.bar - 1.96*s/sqrt(n)
x.bar + 1.96*s/sqrt(n)

## [1] 12108.47
## [1] 14305.79

The lower limit of the confidence interval turns out to be $12,108.47 and the upper limit is $14,305.79. The confidence interval is the range of values between these two numbers.

Consider, next, a confidence interval for the probability of an event. The estimate of the probability \(p\) is \(\hat p\text{,}\) the relative proportion of occurrences of the event in the sample. Again, we construct an interval about this estimate. In this case, a confidence interval of confidence level 95% is of the form \(\big[\hat p - 1.96 \cdot \sqrt{\hat p(1-\hat p)/n}, \hat p + 1.96 \cdot \sqrt{\hat p(1-\hat p)/n}\big]\text{,}\) where \(n\) is the sample size. Observe that \(\hat p\) replaces \(\bar x\) as the estimate of the parameter and that \(\hat p(1-\hat p)/n\) replace \(s^2/n\) as the estimate of the variance of the estimator. The confidence interval for the probability may be expressed in the form:

As an example, let us construct a confidence interval for the proportion of car types that use diesel fuel. The variable “fuel.type" is a factor that records the type of fuel the car uses, either diesel or gas:

table(cars$fuel.type)

## 
## diesel    gas 
##     20    185

Only 20 of the 205 types of cars are run on diesel in this data set. The point estimation of the probability of such car types and the confidence interval for this probability are:

n <- 205
p.hat <- 20/n
p.hat
p.hat - 1.96*sqrt(p.hat*(1-p.hat)/n)
p.hat + 1.96*sqrt(p.hat*(1-p.hat)/n)

## [1] 0.09756098
## [1] 0.05694226
## [1] 0.1381797

The point estimation of the probability is \(\hat p = 20/205 \approx 0.098\) and the confidence interval, after rounding up, is \([0.057, 0.138]\text{.}\)

In the previous subsection we computed a confidence interval for the expected price of a car and a confidence interval for the probability that a car runs on diesel. In this subsection we explain the theory behind the construction of confidence intervals for the expectation. The theory provides insight to the way confidence intervals should be interpreted. In the next subsection we will discuss the theory behind the construction of confidence intervals for the probability of an event.

Assume one is interested in a confidence interval for the expectation of a measurement \(X\text{.}\) For a sample of size \(n\text{,}\) one may compute the sample average \(\bar X\text{,}\) which is the point estimator for the expectation. The expected value of the sample average is the expectation \(\Expec(X)\text{,}\) for which we are trying to produce the confidence interval. Moreover, the variance of the sample average is \(\Var(X)/n\text{,}\) where \(\Var(X)\) is the variance of a single measurement and \(n\) is the sample size.

The construction of a confidence interval for the expectation relies on the Central Limit Theorem and on estimation of the variance of the measurement. The Central Limit Theorem states that the distribution of the (standardized) sample average \(Z = (\bar X - \Expec(X)/\sqrt{\Var(X)/n}\) is approximately standard Normal for a large enough sample size. The variance of the measurement can be estimated using the sample variance \(S^2\text{.}\)

Supposed that we are interested in a confidence interval with a confidence level of 95%. The value 1.96 is the 0.975-percentile of the standard Normal. Therefore, about 95% of the distribution of the standardized sample average is concentrated in the range \([-1.96,1,96]\text{:}\)

The event, the probability of which is being described in the last display, states that the absolute value of deviation of the sample average from the expectation, divided by the standard deviation of the sample average, is no more than 1.96. In other words, the distance between the sample average and the expectation is at most 1.96 units of standard deviation. One may rewrite this event in a form that puts the expectation within an interval that is centered at the sample average

¹

Observe that \(|\bar X - \Expec(X)| = |\Expec(X) -\bar X |\) and therefore \(\big\{|\bar X - \Expec(X)| \leq 1.96 \cdot \sqrt{\Var(X)/n} \big\} =\big\{| \Expec(X)-\bar X | \leq 1.96 \cdot \sqrt{\Var(X)/n} \big\}\text{.}\) From the definition of the absolute value we obtain that the last expression is equal to \(\big\{-1.96 \cdot \sqrt{\Var(X)/n} \leq \Expec(X)-\bar X \leq 1.96 \cdot \sqrt{\Var(X)/n} \big\}\text{.}\) Moving the average to the other side of the inequality (for both inequalities involved) produces the representation \(\big\{\bar X-1.96 \cdot \sqrt{\Var(X)/n} \leq \Expec(X)\leq \bar X + 1.96 \cdot \sqrt{\Var(X)/n} \big\}\text{.}\)

:

Clearly, the probability of the later event is (approximately) 0.95 since we are considering the same event, each time represented in a different form. The second representation states that the expectation \(\Expec(X)\) belongs to an interval about the sample average: \(\bar X \pm 1.96 \sqrt{\Var(X)/n}\text{.}\) This interval is, almost, the confidence interval we seek.

The difficulty is that we do not know the value of the variance \(\Var(X)\text{,}\) hence we cannot compute the interval in the proposed form from the data. In order to overcome this difficulty we recall that the unknown variance may nonetheless be estimated from the data:

where \(S\) is the sample standard deviation

²

The sample variance, that serves as the estimator of the variance, is computed from the sample using the formula: \(S^2 = \sum_{i=1}^n (X_i-\bar X)^2/(n-1)\text{.}\)

.

When the sample size is sufficiently large, so that \(S\) is very close to the value of the standard deviation of an observation, we obtain that the interval \(\bar X \pm 1.96 \sqrt{\Var(X)/n}\) and the interval \(\bar X \pm 1.96 \cdot S/\sqrt{n}\) almost coincide. Therefore:

Hence, \(\bar X \pm 1.96 \cdot S/\sqrt{n}\) is an (approximate) confidence interval of the (approximate) confidence level 0.95.

Let us demonstrate the issue of confidence level by running a simulation. We are interested in a confidence interval for the expected price of a car. In the simulation we assume that the distribution of the price is \(\mathrm{Exponential}(1/13000)\text{.}\) (Consequently, \(\Expec(X) = 13,000\)). We take the sample size to be equal to \(n=201\) and compute the actual probability of the confidence interval containing the value of the expectation:

lam <- 1/13000
n <- 201
X.bar <- rep(0,10^5)
S <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rexp(n,lam)
  X.bar[i] <- mean(X)
  S[i] <- sd(X)
}
LCL <- X.bar - 1.96*S/sqrt(n)
UCL <- X.bar + 1.96*S/sqrt(n)
mean((13000 >= LCL) & (13000 <= UCL))

## [1] 0.94425

Below we will go over the code and explain the simulation. But, before doing so, notice that the actual probability that the confidence interval contains the expectation is about 0.945, which is slightly below the nominal confidence level of 0.95. Still quoting the nominal value as the confidence level of the confidence interval is not too far from reality.

Let us look now at the code that produced the simulation. In each iteration of the simulation a sample is generated. The sample average and standard deviations are computed and stored in the appropriate locations of the sequences “X.bar" and “S". At the end of all the iterations the content of these two sequences represents the sampling distribution of the sample average \(\bar X\) and the sample standard deviation \(S\text{,}\) respectively.

The lower and the upper end-points of the confidence interval are computed in the next two lines of code. The lower level of the confidence interval is stored in the object “LCL" and the upper level is stored in “UCL". Consequently, we obtain the sampling distribution of the confidence interval. This distribution is approximated by 100,000 random confidence intervals that are generated by the sampling distribution. Some of these random intervals contain the value of the expectation, namely 13,000, and some do not. The proportion of intervals that contain the expectation is the (simulated) confidence level. The last expression produces this confidence level, which turns out to be equal to about 0.945.

The last expression involves a new element, the term “&", which calls for more explanations. Indeed, let us refer to the last expression in the code. This expression involves the application of the function “mean". The input to this function contains two sequences with logical values (“TRUE" or “FALSE"), separated by the character “&". The character “&" corresponds to the logical “AND" operator. This operator produces a “TRUE" if a “TRUE" appears at both sides. Otherwise, it produces a “FALSE". (Compare this operator to the operator “OR", that is expressed in R with the character “|", that produces a “TRUE" if at least one “TRUE" appears at either sides.)

In order to clarify the behavior of the terms “&" and “|" consider the following example:

a <- c(TRUE, TRUE, FALSE, FALSE)
b <- c(FALSE, TRUE, TRUE, FALSE)
a & b
a | b

## [1] FALSE  TRUE FALSE FALSE
## [1]  TRUE  TRUE  TRUE FALSE

The term “&" produces a “TRUE" only if parallel components in the sequences “a" and “b" both obtain the value “TRUE". On the other hand, the term “|" produces a “TRUE" if at least one of the parallel components are “TRUE". Observe, also, that the output of the expression that puts either of the two terms between two sequences with logical values is a sequence of the same length (with logical components as well).

The expression “(13000 >= LCL)" produces a logical sequence of length 100,000 with “TRUE" appearing whenever the expectation is larger than the lower level of the confidence interval. Similarly, the expression “(13000 <= UCL)" produces “TRUE" values whenever the expectation is less than the upper level of the confidence interval. The expectation belongs to the confidence interval if the value in both expressions is “TRUE". Thus, the application of the term “&" to these two sequences identifies the confidence intervals that contain the expectation. The application of the function “mean" to a logical vector produces the relative frequency of TRUE’s in the vector. In our case this corresponds to the relative frequency of confidence intervals that contain the expectation, namely the confidence level.

We calculated before the confidence interval \([12108.47, 14305.79]\) for the expected price of a car. This confidence interval was obtained via the application of the formula for the construction of confidence intervals with a 95% confidence level to the variable “price" in the data frame “cars". Casually speaking, people frequently refer to such an interval as an interval that contains the expectation with probability of 95%.

However, one should be careful when interpreting the confidence level as a probabilistic statement. The probability computations that led to the method for constructing confidence intervals were carried out in the context of the sampling distribution. Therefore, probability should be interpreted in the context of all data sets that could have emerged and not in the context of the given data set. No probability is assigned to the statement “The expectation belongs to the interval \([12108.47, 14305.79]\)". The probability is assigned to the statement “The expectation belongs to the interval \(\bar X \pm 1.96 \cdot S /\sqrt{n}\)", where \(\bar X\) and \(S\) are interpreted as random variables. Therefore the statement that the interval \([12108.47, 14305.79]\) contains the expectation with probability of 95% is meaningless. What is meaningful is the statement that the given interval was constructed using a procedure that produces, when applied to random samples, intervals that contain the expectation with the assigned probability.

The next issue is the construction of a confidence interval for the probability of an event. Recall that a probability \(p\) of some event can be estimated by the observed relative frequency of the event in the sample, denoted \(\hat P\text{.}\) The estimation is associated with the Bernoulli random variable \(X\text{,}\) that obtains the value 1 when the event occurs and the value 0 when it does not. In the estimation problem \(p\) is the expectation of \(X\) and \(\hat P\) is the sample average of this measurement. With this formulation we may relate the problem of the construction of a confidence interval for \(p\) to the problem of constructing a confidence interval for the expectation of a measurement. The latter problem was dealt with in the previous subsection.

Specifically, the discussion regarding the steps in the construction – staring with an application of the Central Limit Theorem in order to produce an interval that depends on the sample average and its variance and proceeding by the replacement of the unknown variance by its estimate – still apply and may be taken as is. However, in the specific case we have a particular expression for the variance of the estimate \(\hat P\text{:}\)

The tradition is to estimate this variance by using the estimator \(\hat P\) for the unknown \(p\) instead of using the sample variance. The resulting confidence interval of significance level 0.95 takes the form:

Let us run a simulation in order to assess the confidence level of the confidence interval for the probability. Assume that \(n=205\) and \(p=0.12\text{.}\) The simulation we run is very similar to the simulation of Subsection 11.2.2. In the first stage we produce the sampling distribution of \(\hat P\) (stored in the sequence “P.hat") and in the second stage we compute the relative frequency in the simulation of the intervals that contain the actual value of \(p\) that was used in the simulation:

p <- 0.12
n <- 205
P.hat <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rbinom(n,1,p)
  P.hat[i] <- mean(X)
}
LCL <- P.hat - 1.96*sqrt(P.hat*(1-P.hat)/n)
UCL <- P.hat + 1.96*sqrt(P.hat*(1-P.hat)/n)
mean((p >= LCL) & (p <= UCL))

## [1] 0.95225

In this simulation we obtained that the actual confidence level is approximately 0.951, which is slightly above the nominal confidence level of 0.95.

The formula \(\bar X \pm 1.96 \cdot S/\sqrt{n}\) that is used for a confidence interval for the expectation and the formula \(\hat P \pm 1.96 \cdot \{\hat P (1-\hat P)/n\}^{1/2}\) for the probability both refer to a confidence intervals with confidence level of 95%. If one is interested in a different confidence level then the width of the confidence interval should be adjusted: a wider interval for higher confidence and a narrower interval for smaller confidence level.

Specifically, if we examine the derivation of the formulae for confidence intervals we may notice that the confidence level is used to select the number 1.96, which is the 0.975-percentile of the standard Normal distribution (1.96 =qnorm(0.975)). The selected number satisfies that the interval \([-1.96,1.96]\) contains 95% of the standard Normal distribution by leaving out 2.5% on both tails. For a different confidence level the number 1.96 should be replace by a different number.

For example, if one is interested in a 90% confidence level then one should use 1.645, which is the 0.95-percentile of the standard Normal distribution (qnorm(0.95)), leaving out 5% in both tails. The resulting confidence interval for an expectation is \(\bar X \pm 1.645 \cdot S/\sqrt{n}\) and the confidence interval for a probability is \(\hat P \pm 1.645 \cdot \{\hat P (1-\hat P)/n\}^{1/2}\text{.}\)