Introduction to Statistical Thinking

The spread of the measurement about its expected value may be measured by the variance or by the standard deviation, which is the square root of the variance. The standard estimator for the variance of the measurement is the sample variance and the square root of the sample variance is the default estimator of the standard deviation.

The computation of the sample variance from the data is discussed in Chapter 3. Recall that the sample variance is computed via the formula:

where \(\bar x\) is the sample average and \(n\) is the sample size. The term \(x_i-\bar x\) is the deviation from the sample average of the \(i\)th observation and \(\sum_{i=1}^n (x_i - \bar x)^2\) is the sum of the squares of deviations. It is pointed out in Chapter 3 that the reason for dividing the sum of squares by \((n-1)\text{,}\) rather than \(n\text{,}\) stems from considerations of statistical inference. A promise was made that these reasonings will be discussed in due course. Now we want to deliver on this promise.

Let us compare between two competing estimators for the variance, both considered as random variables. One is the estimator \(S^2\text{,}\) which is equal to the formula for the sample variance applied to a random sample:

The computation of this statistic can be carried out with the function var.

The second estimator is the one obtained when the sum of squares is divided by the sample size (instead of the sample size minus 1):

Observe that the second estimator can be represented in the form:

Hence, the second estimator may be obtained by the multiplication of the first estimator \(S^2\) by the ratio \((n-1)/n\text{.}\) We seek to compare between \(S^2\) and \([(n-1)/n] S^2\) as estimators of the variance.

In order to make the comparison concrete, let us consider it in the context of a Normal measurement with expectation \(\mu = 5\) and variance \(\sigma^2 = 3\text{.}\) Let us assume that the sample is of size 20 (\(n=20\)).

Under these conditions we carry out a simulation. Each iteration of the simulation involves the generation of a sample of size \(n=20\) from the given Normal distribution. The sample variance \(S^2\) is computed from the sample with the application of the function var. The resulting estimate of the variance is stored in an object that is called X.var:

mu <- 5
std <- sqrt(3)
X.var <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rnorm(20,mu,std)
  X.var[i] <- var(X)
}

The content of the object X.var, at the end of the simulation, approximates the sampling distribution of the estimator \(S^2\text{.}\)

Our goal is to compare between the performance of the estimator of the variance \(S^2\) and that of the alternative estimator. In this alternative estimator the sum of squared deviations is divided by the sample size (\(n=20\)) and not by the sample size minus 1 (\(n-1 = 19\)). Consequently, the alternative estimator is obtained by multiplying \(S^2\) by the ratio \(19/20\text{.}\) The sampling distribution of the values of \(S^2\) is approximated by the content of the object X.var. It follows that the sampling distribution of the alternative estimator is approximated by the object (19/20)*X.var, in which each value of \(S^2\) is multiplied by the appropriate ratio. The comparison between the sampling distribution of \(S^2\) and the sampling distribution of the alternative estimator is obtained by comparing between X.var and (19/20)*X.var.

Let us start by the investigation of the expectation of the estimators. Recall that when we analyzed the sample average as an estimator of the expectation of a measurement we obtained that the expectation of the sampling distribution of the estimator is equal to the value of the parameter it is trying to estimate. One may wonder: What is the situation for the estimators of the variance? Is it or is it not the case that the expectation of their sampling distribution equals the value of the variance? In other words, is the distribution of either estimators of the variance centered at the value of the parameter they are trying to estimate?

Compute the expectations of the two estimators:

mean(X.var)
mean((19/20)*X.var)

## [1] 2.995744

## [1] 2.845957

Note that 3 is the value of the variance of the measurement that was used in the simulation. Observe that the expectation of \(S^2\) is essentially equal to 3, whereas the expectation of the alternative estimator is less than 3. Hence, at least in the example that we consider, the center of the distribution of \(S^2\) is located on the target value. On the other hand, the center of the sampling distribution of the alternative estimator is located off that target value.

As a matter of fact it can be shown mathematically that the expectation of the estimator \(S^2\) is always equal to the variance of the measurement. This holds true regardless of what is the actual value of the variance. On the other hand the expectation of the alternative estimator is always off the target value

¹

For the estimator \(S^2\) we get that \(\Expec(S^2) = \Var(X)\text{.}\) On the other hand, for the alternative estimator we get that \(\Expec([(n-1)/n]\cdot S^2) = [(n-1)/n]\Var(X) \not = \Var(X)\text{.}\) This statement holds true also in the cases where the distribution of the measurement is not Normal.

.

An estimator is called unbiased if its expectation is equal to the value of the parameter that it tries to estimate. We get that \(S^2\) is an unbiased estimator of the variance. Similarly, the sample average is an unbiased estimator of the expectation. Unlike these two estimators, the alternative estimator of the variance is a biased estimator.

The default is to use \(S^2\) as the estimator of the variance of the measurement and to use its square root as the estimator of the standard deviation of the measurement. A justification, which is frequently quoted to justify this selection, is the fact that \(S^2\) is an unbiased estimator of the variance

²

As part of your homework assignment you are required to investigate the properties of \(S\text{,}\) the square root of \(S^2\text{,}\) as an estimator of the standard deviation of the measurement. A conclusion of this investigation is that \(S\) is a biased estimator of the standard deviation.

.

In the previous section, when comparing two competing estimators of the expectation, or main concern was the quantification of the spread of the sampling distribution of either estimator about the target value of the parameter. We used that spread as a measure of the distance between the estimator and the value it tries to estimate. In the setting of the previous section both estimators were unbiased. Consequently, the variance of the estimators, which measures the spread of the distribution about its expectation, could be used in order to quantify the distance between the estimator and the parameter. (Since, for unbiased estimators, the parameter is equal to the expectation of the sampling distribution.)

In the current section one of the estimators (\(S^2\)) is unbiased, but the other (the alternative estimator) is not. In order to compare their accuracy in estimation we need to figure out a way to quantify the distance between a biased estimator and the value it tries to estimate.

Towards that end let us recall the definition of the variance. Given a random variable \(X\) with an expectation \(\Expec(X)\text{,}\) we consider the square of the deviations \((X - \Expec(X))^2\text{,}\) which measure the (squared) distance between each value of the random variable and the expectation. The variance is defined as the expectation of the squared distance: \(\Var(X) = \Expec[(X-\Expec(X))^2]\text{.}\) One may think of the variance as an overall measure of the distance between the random variable and the expectation.

Assume now that the goal is to assess the distance between an estimator and the parameter it tries to estimate. In order to keep the discussion on an abstract level let us use the Greek letter \(\theta\) (read: theta) to denote this parameter

³

The letter \(\theta\) is frequently used in the statistical literature to denote a parameter of the distribution. In the previous section we considered \(\theta = \Expec(X)\) and in this section we consider \(\theta=\Var(X)\text{.}\)

. The estimator is denoted by \(\hat \theta\) (read: theta hat). It is a statistic, a formula applied to the data. Hence, with respect to the sampling distribution, \(\hat \theta\) is a random variable

⁴

Observe that we diverge here slightly from our promise to use capital letters to denote random variables. However, denoting the parameter by \(\theta\) and denoting the estimator of the parameter by \(\hat \theta\) is standard in the statistical literature. As a matter of fact, we will use the “hat” notation, where a hat is placed over a Greek letter that represents the parameter, in other places in this book. The letter with the hat on top will represent the estimator and will always be considered as a random variable. For Latin letters we will still use capital letters, with or without a hat, to represent a random variable and small letter to represent evaluation of the random variable for given data.

. The issue is to measure the distance between the random variable \(\hat \theta\) and the parameter \(\theta\text{.}\)

Motivated by the method that led to the definition of the variance we consider the deviations between the estimator and the parameter. The square deviations \((\hat \theta - \theta)^2\) may be considered in the current context as a measure of the (squared) distance between the estimator and the parameter. When we take the expectation of these square deviations we get an overall measure of the distance between the estimator and the parameter. This overall distance is called the mean square error of the estimator and is denoted by MSE:

The mean square error of an estimator is tightly linked to the bias and the variance of the estimator. The bias of an estimator \(\hat \theta\) is the difference between the expectation of the estimator and the parameter it seeks to estimate:

In an unbiased estimator the expectation of the estimator and the estimated parameter coincide, i.e. the bias is equal to zero. For a biased estimator the bias is either negative, as is the case for the alternative estimator of the variance, or else it is positive.

The variance of the estimator, \(\mbox{Variance} = \Var(\hat \theta)\text{,}\) is a measure of the spread of the sampling distribution of the estimator about its expectation.

The link between the mean square error, the bias, and the variance is described by the formula:

Hence, the mean square error of an estimator is the sum of its variance, the (squared) distance between the estimator and its expectation, and the square of the bias, the square of the distance between the expectation and the parameter. The mean square error is influenced both by the spread of the distribution about the expected value (the variance) and by the distance between the expected value and the parameter (the bias). The larger either of them become the larger is the mean square error, namely the distance between the estimator and the parameter.

Let us compare between the mean square error of the estimator \(S^2\) and the mean square error of the alternative estimator \([19/20] S^2\text{.}\) Recall that we have computed their expectations and found out that the expectation of \(S^2\) is essentially equal to 3, the target value of the variance. The expectation of the alternative estimator turned out to be equal to 2.845630, which is less than the target value

⁵

It can be shown mathematically that \(\Expec([(n-1)/n] S^2) = [(n-1)/n] \Expec(S^2)\text{.}\) Consequently, the actual value of the expectation of the alternative estimator in the current setting is \([19/20]\cdot 3 = 2.85\) and the bias is \(-0.15\text{.}\) The results of the simulation are consistent with this fact.

. It turns out that the bias of \(S^2\) is zero (or essentially zero in the simulations) and the bias of the alternative estimator is \(2.845630 - 3 = -0.15437 \approx -0.15\text{.}\)

In order to compute the mean square errors of both estimators, let us compute their variances:

var(X.var)
var((19/20)*X.var)

## [1] 0.9484016

## [1] 0.8559324

Observe that the variance of \(S^2\) is essentially equal to 0.936 and the variance of the alternative estimator is essentially equal to 0.845.

The estimator \(S^2\) is unbiased. Consequently, the mean square error of \(S^2\) is equal to its variance. The bias of the alternative is -0.15. As a result we get that the mean square error of this estimator, which is the sum of the variance and the square of the bias, is essentially equal to

Observe that the mean square error of the estimator \(S^2\text{,}\) which is equal to 0.936, is larger than the mean square error of the alternative estimator.

Notice that even though the alternative estimator is biased it still has a smaller mean square error than the default estimator \(S^2\text{.}\) Indeed, it can be prove mathematically that when the measurement has a Normal distribution then the mean square error of the alternative estimator is always smaller than the mean square error of the sample variance \(S^2\text{.}\)

Still, although the alternative estimator is slightly more accurate than \(S^2\) in the estimation of the variance, the tradition is to use the latter. Obeying this tradition we will henceforth use \(S^2\) whenever estimation of the variance is required. Likewise, we will use \(S\text{,}\) the square root of the sample variance, to estimate the standard deviation.

In order to understand how is it that the biased estimator produced a smaller mean square error than the unbiased estimator let us consider the two components of the mean square error. The alternative estimator is biased but, on the other hand, it has a smaller variance. Both the bias and the variance contribute to the mean square error of an estimator. The price for reducing the bias in estimation is usually an increase in the variance and vice versa. The consequence of producing an unbiased estimator such as \(S^2\) is an inflated variance. A better estimator is an estimator that balances between the error that results from the bias and the error that results from the variance. Such is the alternative estimator.

We will use \(S^2\) in order to estimate the variance of a measurement. A context in which an estimate of the variance of a measurement is relevant is in the assessment of the variance of the sample mean. Recall that the variance of the sample mean is equal to \(\Var(X)/n\text{,}\) where \(\Var(X)\) is the variance of the measurement and \(n\) is the size of the sample. In the case where the variance of the measurement is not known one may estimate it from the sample using \(S^2\text{.}\) It follows that the estimator of the variance of the sample average is \(S^2/n\text{.}\) Similarly, \(S/\sqrt{n}\) can be used as an estimator of the standard deviation of the sample average.