Introduction to Statistical Thinking

How close is the estimated value of the expectation — the average price — to the expected price?

There is no way of answering this question on the basis of the data we observed. Indeed, we think of the price of a random car as a random variable. The expectation we seek to estimate is the expectation of that random variable. However, the actual value of that expectation is unknown. Hence, not knowing what is the target value, how can we determine the distance between the computed average 13207.13 and that unknown value?

As a remedy for not being able to answer the question we would like to address we, instead, change the question. In the new formulation of the question we ignore the data at hand altogether. The new formulation considers the sample average as a statistic and the question is formulated in terms of the sampling distribution of that statistic. The question, in its new formulation is: How close is the sample average of the price, taken as a random variable, to the expected price?

Notice that in the new formulation of the question the observed average price \(\bar x = 13207.13\) has no special role. The question is formulated in terms of the sampling distribution of the sample average (\(\bar X\)). The observed average value is only one among many in the sampling distribution of the average.

The advantage of the new formulation of the question is that it can be addressed. We do have means for investigating the closeness of the estimator to the parameter and thereby producing meaningful answers. Specifically, consider the current case where the estimator is the sample average \(\bar X\text{.}\) This estimator attempts to estimate the expectation \(\Expec(X)\) of the measurement, which is the parameter. Assessing the closeness of the estimator to the parameter corresponds to the comparison between the distribution of the random variable, i.e. the estimator, and the value of the parameter.

For this comparison we may note that the expectation \(\Expec(X)\) is also the expectation of the sample average \(\bar X\text{.}\) Consequently, in this problem the assessment of the closeness of the estimator to the parameter is equivalent to the investigation of the spread of the distribution of the sample average about its expectation.

Consider an example of such investigation. Imagine that the expected price of a car is equal to $13,000. A question one may ask is how likely it is that the estimator’s guess at the value is within $1,000 of the actual value? In other words, what is the probability that sample average falls in the range \([12,000, 14,000]\text{?}\)

Let us investigate this question using simulations. Recall our assumption that the distribution of the price is Exponential. An expectation of 13,000 corresponds to a rate parameter of \(\lambda = 1/13,000\text{.}\) We simulate the sampling distribution of the estimator by the generation of a sample of 201 Exponential random variables with this rate. The sample average is computed for each sample and stored. The sampling distribution of the sample average is approximated via the production of a large number of sample averages:

lam <- 1/13000
X.bar <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rexp(201,lam)
  X.bar[i] <- mean(X)
}
mean(abs(X.bar - 1/lam) <= 1000)

## [1] 0.72405

In the last line of the code we compute the probability of being within $1,000 of the expected price. Recall that the expected price in the Exponential case is the reciprocal of the rate \(\lambda\text{.}\) In this simulation we obtained 0.7247 as an approximation of the probability.

In the case of the sample average we may also apply the Normal approximation in order to assess the probability under consideration. In particular, if \(\lambda = 1/13,000\) then the expectation of an Exponential observation is \(\Expec(X) = 1/\lambda = 13,000\) and the variance is \(\Var(X) = 1/\lambda^2 = (13,000)^2\text{.}\) The expectation of the sample average is equal to the expectation of the measurement, 13,000 in this example. The variance of the sample average is equal to the variance of the observation, divided by the sample size. In the current setting it is equal to \((13,000)^2/201\text{.}\) The standard deviation is equal to the square root of the variance.

The Normal approximation uses the Normal distribution in order to compute probabilities associated with the sample average. The Normal distribution that is used has the same expectation and standard deviation as the sample average:

mu <- 13000
sig <- 13000/sqrt(201)
pnorm(14000,mu,sig) - pnorm(12000,mu,sig)

## [1] 0.7245391

The probability of falling within the interval \([12000, 14000]\) is computed as the difference between the cumulative Normal probability at 14,000 and the cumulative Normal probability at 12,000.

These cumulative probabilities are computed with the function pnorm. Recall that this function computes the cumulative probability for the Normal distribution. If the first argument is 14,000 then the function produces the probability that a Normal random variable is less than or equal to 14,000. Likewise, if the first argument is 12,000 then the computed probability is the probability of being less than or equal to 12,000. The expectation of the Normal distribution enters in the second argument of the function and the standard deviation enters in the third argument.

The Normal approximation of falling in the interval \([12000, 14000]\text{,}\) computed as the difference between the two cumulative probabilities, produces 0.7245391 as the probability

²

As a matter of fact, the difference is the probability of falling in the half-open interval \((12000,14000]\text{.}\) However, for continuous distributions the probability of the end-points is zero and they do not contribute to the probability of the interval.

. Notice that the probability 0.7247 computed in the simulations is in agreement with the Normal approximation.

If we wish to assess the accuracy of the estimator at other values of the parameter, say \(\Expec(X) = 12,000\) (which corresponds to \(\lambda = 1/12,000\)) or \(\Expec(X) = 14,033\text{,}\) (which corresponds to \(\lambda = 1/14,033\)) all we need to do is change the expression lam <- 1/13000 to the new value and rerun the simulation.

Alternatively, we may use a Normal approximation with modified interval, expectation, and standard deviation. For example, consider the case where the expected price is equal to $12,000. In order to asses the probability that the sample average falls within $1,000 of the parameter we should compute the probability of the interval \([11,000, 13,000]\) and change the entries to the first argument of the function pnorm accordingly. The new expectation is 12,000 and the new standard deviation is \(12,000/\sqrt{201}\text{:}\)

mu <- 12000
sig <- 12000/sqrt(201)
pnorm(13000,mu,sig) - pnorm(11000,mu,sig)

## [1] 0.7625775

This time we get that the probability is, approximately, 0.763.

The fact that the computed value of the average 13,207.13 belongs to the interval \([12,000, 14,000]\) that was considered in the first analysis but does not belong to the interval \([11,000, 13,000]\) that was considered in the second analysis is irrelevant to the conclusions drawn from the analysis. In both cases the theoretical properties of the sample average as an estimator were considered and not its value at specific data.

In the simulation and in the Normal approximation we applied one method for assessing the closeness of the sample average to the expectation it estimates. This method involved the computation of the probability of being within $1,000 of the expected price. The higher this probability, the more accurate is the estimator.

An alternative method for assessing the accuracy of an estimator of the expectation may involve the use of an overall summary of the spread of the distribution. A standard method for quantifying the spread of a distribution about the expectation is the variance (or its square root, the standard deviation). Given an estimator of the expectation of a measurement, the sample average for example, we may evaluate the accuracy of the estimator by considering its variance. The smaller the variance the more accurate is the estimator.

Consider again the case where the sample average is used in order to estimate the expectation of a measurement. In such a situation the variance of the estimator, i.e. the variance of the sample average, is obtained as the ratio between the variance of the measurement \(\Var(X)\text{,}\) divided by the sample size \(n\text{:}\)

Notice that for larger sample sizes the estimator is more accurate. The lager the sample size \(n\) the smaller is the variance of the estimator, in which case the values of the estimator tend to be more concentrated about the expectation. Hence, one may make the estimator more accurate by increasing the sample size.

Another method for improving the accuracy of the average of measurements in estimating the expectation is the application of a more accurate measurement device. If the variance \(\Var(X)\) of the measurement device decreases so does the variance of the sample average of such measurements.

In the sequel, when we investigate the accuracy of estimators, we will generally use overall summaries of the spread of their distribution around the target value of the parameter.

Notice that the formulation of the accuracy of estimation that we use replaces the question: “How close is the given value of the estimator to the unknown value of the parameter?” by the question: “How close are the unknown (and random) values of the estimator to a given value of the parameter?” In the second formulation the question is completely academic and unrelated to actual measurement values. In this academic context we can consider different potential values of the parameter. Once the value of the parameter has been selected it can be treated as known in the context of the academic discussion. Clearly, this does not imply that we actually know what is the true value of the parameter.

The sample average is a natural estimator of the expectation of the measurement. However, one may propose other estimators. For example, when the distribution of the measurement is symmetric about the expectation then the median of the distribution is equal to the expectation. The sample median, which is a natural estimator of the measurement median, is an alternative estimator of the expectation in such case. Which of the two alternatives, the sample average or the sample median, should we prefer as an estimator of the expectation in the case of a symmetric distribution?

The straightforward answer to this question is to prefer the better one, the one which is more accurate. As part of the solved exercises you are asked to compare the sample average to the sample median as estimators of the expectation. Here we compare the sample average to yet another alternative estimator — the mid-range estimator — which is the average between the smallest and the largest observations.

In the comparison between estimators we do not evaluate them in the context of the observed data. Rather, we compare them as random variables. The comparison deals with the properties of the estimators in a given theoretical context. This theoretical context is motivated by the realities of the situation as we know them. But, still, the frame of reference is the theoretical model and not the collected data.

Hence, depending on the context, we may assume in the comparison that the observations emerge from some distribution. We may specify parameter values for this distribution and select the appropriate sample size. After setting the stage we can compare the accuracy of one estimator against that of the other. Assessment at other parameter values in the context of the given theoretical model, or of other theoretical models, may provide insight and enhance our understanding regarding the relative merits and weaknesses of each estimator.

Let us compare the sample average to the sample mid-range as estimators of the expectation in a situation that we design. Consider a Normal measurement \(X\) with expectation \(\Expec(X) = 3\) and variance that is equal to 2. Assume that the sample size is \(n = 100\text{.}\) Both estimators, due to the symmetry of the Normal distribution, are centered at the expectation. Hence, we may evaluate the accuracy of the two estimators using their variances. These variances are the measure of the spread of the distributions of each estimator about the target parameter value.

We produce the sampling distribution and compute the variances using a simulation. Recall that the distribution of the mid-range statistic was simulated in the previous chapter. In the computation of the mid-range statistic we used the function max that computes the maximum value of its input and the function min that computes the minimum value:

mu <- 3
sig <- sqrt(2)
X.bar <- rep(0,10^5)
mid.range <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rnorm(100,mu,sig)
  X.bar[i] <- mean(X)
  mid.range[i] <- (max(X)+min(X))/2
}
var(X.bar)
var(mid.range)

## [1] 0.02003947
## [1] 0.1870101

We get that the variance of the sample average

³

As a matter of fact, the variance of the sample average is exactly \(\Var(X)/100 = 0.02\text{.}\) Due to the inaccuracy of the simulation we got a slightly different variance.

is approximately equal to 0.02. The variance of the mid-range statistic is approximately equal to 0.185, more than 9 times as large. We see that the accuracy of the sample average is better in this case than the accuracy of the mid-range estimator. Evaluating the two estimators at other values of the parameter will produce the same relation. Hence, in the current example it seems as if the sample average is the better of the two.

Is the sample average necessarily the best estimator for the expectation? The next example will demonstrate that this need not always be the case.

Consider again a situation of observing a sample of size \(n=100\text{.}\) However, this time the measurement \(X\) is Uniform and not Normal. Say \(X \sim \mathrm{Uniform}(0.5,5.5)\) has the Uniform distribution over the interval \([0.5, 5.5]\text{.}\) The expectation of the measurement is equal to 3 like before, since \(\Expec(X) = (0.5+5.5)/2 = 3\text{.}\) The variance on an observation is \(\Var(X) = (5.5 - 0.5)^2/12 = 2.083333\text{,}\) not much different from the variance that was used in the Normal case. The Uniform distribution, like the Normal distribution, is a symmetric distribution about the center of the distribution. Hence, using the mid-range statistic as an estimator of the expectation makes sense

⁴

Observe that the middle range of the \(\mathrm{Uniform}(a,b)\) distribution, the middle point between the maximum value of the distribution \(b\) and the minimal value \(a\text{,}\) is \((a+b)/2\text{,}\) which is equal to the expectation of the distribution

.

We re-run the simulations, using the function runif for the simulation of a sample from the Uniform distribution and the parameters of the Uniform distribution instead of the function rnorm that was used before:

a <- 0.5
b <- 5.5
X.bar <- rep(0,10^5)
mid.range <- rep(0,10^5)
for(i in 1:10^5) {
  X <- runif(100,a,b)
  X.bar[i] <- mean(X)
  mid.range[i] <- (max(X)+min(X))/2
}
var(X.bar)
var(mid.range)

## [1] 0.02087122
## [1] 0.001209319

Again, we get that the variance of the sample average is approximately equal to 0.02, which is close to the theoretical value

⁵

Actually, the exact value of the variance of the sample average is \(\Var(X)/100 = 0.02083333\text{.}\) The results of the simulation are consistent with this theoretical computation.

. The variance of mid-range statistic is approximately equal to 0.0012.

Observe that in the current comparison between the sample average and the mid-range estimator we get that the latter is a clear winner. Examination of other values of \(a\) and \(b\) for the Uniform distribution will produce the same relation between the two competitors. Hence, we may conclude that for the case of the Uniform distribution the sample average is an inferior estimator.

The last example may serve as yet another reminder that life is never simple. A method that is good in one situation may not be as good in a different situation.

Still, the estimator of choice of the expectation is the sample average. Indeed, in some cases we may find that other methods may produce more accurate estimates. However, in most settings the sample average beats its competitors. The sample average also possesses other useful benefits. Its sampling distribution is always centered at the expectation it is trying to estimate. Its variance has a simple form, i.e. it is equal to the variance of the measurement divided by the sample size. Moreover, its sampling distribution can be approximated by the Normal distribution. Henceforth, due to these properties, we will use the sample average whenever estimation of the expectation is required.