The Sampling Distribution

Section 9.4 The Sampling Distribution

Subsection 9.4.1 Statistics

Statistical inferences, be it point estimation, confidence intervals, or testing hypothesis, are based on statistics computed from the data. Examples of statistics are the sample average and the sample standard deviation. These are important examples, but clearly not the only ones. Given numerical data, one may compute the smallest value, the largest value, the quartiles, and the median. All are examples of statistics. Statistics may also be associated with factors. The frequency of a given attribute among the observations is a statistic. (An example of such statistic is the frequency of diesel cars in the data frame.) As part of the discussion in the subsequent chapters we will consider these and other types of statistics.

🔗

Any statistic, when computed in the context of the data frame being analyzed, obtains a single numerical value. However, once a sampling distribution is being considered then one may view the same statistic as a random variable. A statistic is a function or a formula which is applied to the data frame. Consequently, when a random collection of data frames is the frame of reference then the application of the formula to each of the data frames produces a random collection of values, which is the sampling distribution of the statistic.

🔗

We distinguish in the text between the case where the statistic is computed in the context of the given data frame and the case where the computation is conducted in the context of the random sample. This distinguishing is emphasized by the use of small letters for the former and capital letters for the later. Consider, for example, the sample average. In the context of the observed data we denote the data values for a specific variable by $x_1, x_2, \ldots, x_n\text{.}$ The sample average computed for these values is denoted by

🔗

\begin{equation*} \bar x = \frac{x_1 + x_2 + \cdots + x_n}{n}\;. \end{equation*}

On the other hand, if the discussion of the sample average is conducted in the context of a random sample then the sample is a sequence $X_1, X_2, \ldots, X_n$ of random variables. The sample average is denoted in this context as

🔗

\begin{equation*} \bar X = \frac{X_1 + X_2 + \cdots + X_n}{n}\;. \end{equation*}

The same formula that was applied to the data values is applied now to the random components of the random sample. In the first context $\bar x$ is an observed non-random quantity. In the second context $\bar X$ is a random variable, an abstract mathematical concept.

🔗

A second example is the sample variance. When we compute the sample variance for the observed data we use the formula:

🔗

\begin{equation*} s^2 = \frac{\mbox{Sum of the squares of the deviations}}{\mbox{Number of values in the sample}-1}= \frac{\sum_{i=1}^n (x_i - \bar x)^2}{n-1}\;. \end{equation*}

However, when we discuss the sampling distribution of the sample variance we apply the same formula to the random sample:

🔗

\begin{equation*} S^2 = \frac{\mbox{Sum of the squares of the deviations}}{\mbox{Number of values in the sample}-1}= \frac{\sum_{i=1}^n (X_i - \bar X)^2}{n-1}\;. \end{equation*}

Again, $S^2$ is a random variable whereas $s^2$ is a non-random quantity: The evaluation of the random variable at the specific sample that is being observed.

🔗

Subsection 9.4.2 The Sampling Distribution

The sampling distribution may emerge as random selection of samples from a particular population. In such a case, the sampling distribution of the sample, and hence of the statistic, is linked to the distribution of values of the variable in the population.

🔗

Alternatively, one may assign theoretical distribution to the measurement associated with the variable. In this other case the sampling distribution of the statistic is linked to the theoretical model.

🔗

Consider, for example, the variable “price” that describes the prices of the 205 car types (with 4 prices missing) in the data frame “cars”. In order to define a sampling distribution one may imagine a larger population of car types, perhaps all the car types that were sold during the 80’s in the United States, or some other frame of reference, with the car types that are included in the data frame considered as a random sample from that larger population. The observed sample corresponds to car types that where sold in 1985. Had one chosen to consider car types from a different year then one may expect to obtain other evaluations of the price variable. The reference population, in this case, is the distribution of the prices of the car types that were sold during the 80’s and the sampling distribution is associated with a random selection of a particular year within this period and the consideration of prices of car types sold in that year. The data for 1985 is what we have at hand. But in the sampling distribution we take into account the possibility that we could have obtained data for 1987, for example, rather than the data we did get.

🔗

An alternative approach for addressing sampling distribution is to consider a theoretical model. Referring again to the variable “price” one may propose an Exponential model for the distribution of the prices of cars. This model implies that car types in the lower spectrum of the price range are more frequent than cars with a higher price tag. With this model in mind, one may propose the sampling distribution to be composed of 205 unrelated copies from the Exponential distribution (or 201 if we do not want to include the missing values). The rate $\lambda$ of the associated Exponential distribution is treated as an unknown parameter. One of the roles of statistical inference is to estimate the value of this parameter with the aid of the data at hand.

🔗

Sampling distribution is relevant also for factor variables. Consider the variable “fuel.type” as an example. In the given data frame the frequency of diesel cars is 20. However, had one considered another year during the 80’s one may have obtained a different frequency, resulting in a sampling distribution. This type of sampling distribution refers to all cars types that were sold in the United States during the 80’s as the frame of reference.

🔗

Alternatively, one may propose a theoretical model for the sampling distribution. Imagine there is a probability $p$ that a car runs on diesel (and probability $1-p$ that it runs on gas). Hence, when one selects 205 car types at random then one obtains that the distribution of the frequency of car types that run on diesel has the $\mathrm{Binomial}(205,p)$ distribution. This is the sampling distribution of the frequency statistic. Again, the value of $p$ is unknown and one of our tasks is to estimate it from the data we observe.

🔗

In the context of statistical inference the use of theoretical models for the sampling distribution is the standard approach. There are situation, such as the application surveys to a specific target population, where the consideration of the entire population as the frame of reference is more natural. But, in most other applications the consideration of theoretical models is the method of choice. In this part of the book, where we consider statistical inference, we will always use the theoretical approach for modeling the sampling distribution.

🔗

Subsection 9.4.3 Theoretical Distributions of Observations

In the first part of the book we introduced several theoretical models that may describe the distribution of an observation. Let us take the opportunity and review the list of models:

🔗

Binomial 🔗: The Binomial distribution is used in settings that involve counting the number of occurrences of a particular outcome. The parameters that determine the distribution are $n\text{,}$ the number of observations, and $p\text{,}$ the probability of obtaining the particular outcome in each observation. The expression “$\mathrm{Binomial}(n,p)$” is used to mark the Binomial distribution. The sample space for this distribution is formed by the integer values $\{0, 1, 2, \ldots, n\}\text{.}$ The expectation of the distribution is $np$ and the variance is $np(1-p)\text{.}$ The functions “dbinom”, “pbinom”, and “qbinom” may be used in order to compute the probability, the cumulative probability, and the percentiles, respectively, for the Binomial distribution. The function “rbinom” can be used in order to simulate a random sample from this distribution.
🔗
Poisson 🔗: The Poisson distribution is also used in settings that involve counting. This distribution approximates the Binomial distribution when the number of examinations $n$ is large but the probability $p$ of the particular outcome is small. The parameter that determines the distribution is the expectation $\lambda\text{.}$ The expression “$\mathrm{Poisson}(\lambda)$” is used to mark the Poisson distribution. The sample space for this distribution is the entire collection of natural numbers $\{0, 1, 2, \ldots\}\text{.}$ The expectation of the distribution is $\lambda$ and the variance is also $\lambda\text{.}$ The functions “dpois”, “ppois”, and “qpois” may be used in order to compute the probability, the cumulative probability, and the percentiles, respectively, for the Poisson distribution. The function “rpois” can be used in order to simulate a random sample from this distribution.
🔗
Uniform 🔗: The Uniform distribution is used in order to model measurements that may have values in a given interval, with all values in this interval equally likely to occur. The parameters that determine the distribution are $a$ and $b\text{,}$ the two end points of the interval. The expression “$\mathrm{Uniform}(a,b)$” is used to identify the Uniform distribution. The sample space for this distribution is the interval $[a,b]\text{.}$ The expectation of the distribution is $(a+b)/2$ and the variance is $(b-a)^2/12\text{.}$ The functions “dunif”, “punif”, and “qunif” may be used in order to compute the density, the cumulative probability, and the percentiles for the Uniform distribution. The function “runif” can be used in order to simulate a random sample from this distribution.
🔗
Exponential 🔗: The Exponential distribution is frequently used to model times between events. It can also be used in other cases where the outcome of the measurement is a positive number and where a smaller value is more likely than a larger value. The parameter that determines the distribution is the rate $\lambda\text{.}$ The expression “$\mathrm{Exponential}(\lambda)$” is used to identify the Exponential distribution. The sample space for this distribution is the collection of positive numbers. The expectation of the distribution is $1/\lambda$ and the variance is $1/\lambda^2\text{.}$ The functions “dexp”, “pexp”, and “qexp” may be used in order to compute the density, the cumulative probability, and the percentiles, respectively, for the Exponential distribution. The function “rexp” can be used in order to simulate a random sample from this distribution.
🔗
Normal 🔗: The Normal distribution frequently serves as a generic model for the distribution of a measurement. Typically, it also emerges as an approximation of the sampling distribution of statistics. The parameters that determine the distribution are the expectation $\mu$ and the variance $\sigma^2\text{.}$ The expression “$\mathrm{Normal}(\mu,\sigma^2)$” is used to mark the Normal distribution. The sample space for this distribution is the collection of all numbers, negative or positive. The expectation of the distribution is $\mu$ and the variance is $\sigma^2\text{.}$ The functions “dnorm”, “pnorm”, and “qnorm” may be used in order to compute the density, the cumulative probability, and the percentiles for the Normal distribution. The function “rnorm” can be used in order to simulate a random sample from this distribution.
🔗

🔗

Subsection 9.4.4 Sampling Distribution of Statistics

Theoretical models describe the distribution of a measurement as a function of a parameter, or a small number of parameters. For example, in the Binomial case the distribution is determined by the number of trials $n$ and by the probability of success in each trial $p\text{.}$ In the Poisson case the distribution is a function of the expectation $\lambda\text{.}$ For the Uniform distribution we may use the end-points of the interval, $a$ and $b\text{,}$ as the parameters. In the Exponential case, the rate $\lambda$ is a natural parameter for specifying the distribution and in Normal case the expectation $\mu$ and the variance $\sigma^2$ my be used for that role.

🔗

The general formulation of statistical inference problems involves the identification of a theoretical model for the distribution of the measurements. This theoretical model is a function of a parameter whose value is unknown. The goal is to produce statements that refer to this unknown parameter. These statements are based on a sample of observations from the given distribution.

🔗

For example, one may try to guess the value of the parameter (point estimation), one may propose an interval which contains the value of the parameter with some subscribed probability (confidence interval) or one may test the hypothesis that the parameter obtains a specific value (hypothesis testing).

🔗

The vehicles for conducting the statistical inferences are statistics that are computed as a function of the measurements. In the case of point estimation these statistics are called estimators. In the case where the construction of an interval that contains the value of the parameter is the goal then the statistics are called confidence interval. In the case of testing hypothesis these statistics are called test statistics.

🔗

In all cases of inference, The relevant statistic possesses a distribution that it inherits from the sampling distribution of the observations. This distribution is the sampling distribution of the statistic. The properties of the statistic as a tool for inference are assessed in terms of its sampling distribution. The sampling distribution of a statistic is a function of the sample size and of the parameters that determine the distribution of the measurements, but otherwise may be of complex structure.

🔗

In order to assess the performance of the statistics as agents of inference one should be able to determine their sampling distribution. We will apply two approaches for this determination. One approach is to use a Normal approximation. This approach relies on the Central Limit Theorem. The other approach is to simulate the distribution. This other approach relies on the functions available in R for the simulation of a random sample from a given distribution.

🔗

Subsection 9.4.5 The Normal Approximation

In general, the sampling distribution of a statistic is not the same as the sampling distribution of the measurements from which it is computed. For example, if the measurements are from the Uniform distributed then the distribution of a function of the measurements will, in most cases, not possess the Uniform distribution. Nonetheless, in many cases one may still identify, at least approximately, what the sampling distribution of the statistic is.

🔗

The most important scenario where the limit distribution of the statistic has a known shape is when the statistic is the sample average or a function of the sample average. In such a case the Central Limit Theorem may be applied in order to show that, at least for a sample size not too small, the distribution of the statistic is approximately Normal.

🔗

In the case where the Normal approximation may be applied then a probabilistic statement associated with the sampling distribution of the statistic can be substituted by the same statement formulated for the Normal distribution. For example, the probability that the statistic falls inside a given interval may be approximated by the probability that a Normal random variable with the same expectation and the same variance (or standard deviation) as the statistic falls inside the given interval.

🔗

For the special case of the sample average one may use the fact that the expectation of the average of a sample of measurements is equal to the expectation of a single measurement and the fact that the variance of the average is the variance of a single measurement, divided by the sample size. Consequently, the probability that the sample average falls within a given interval may be approximate by the probability of the same interval according to the Normal distribution. The expectation that is used for the Normal distribution is the expectation of the measurement. The standard deviation is the standard deviation of the measurement, divided by the square root of the number of observations.

🔗

The Normal approximation of the distribution of a statistic is valid for cases other than the sample average or functions thereof. For example, it can be shown (under some conditions) that the Normal approximation applies to the sample median, even though the sample median is not a function of the sample average.

🔗

On the other hand, one need not always assume that the distribution of a statistic is necessarily Normal. In many cases it is not, even for a large sample size. For example, the minimal value of a sample that is generated from the Exponential distribution can be shown to follow the Exponential distribution with an appropriate rate,

If the rate of an Exponential measurement is $\lambda$ then the rate of the minimum of $n$ such measurements is $n\lambda\text{.}$

regardless of the sample size.

🔗

Subsection 9.4.6 Simulations

In most problems of statistical inference that will be discussed in this book we will be using the Normal approximation for the sampling distribution of the statistic. However, every now and then we may want to check the validity of this approximation in order to reassure ourselves of its appropriateness. Computerized simulations can be carried out for that checking. The simulations are equivalent to those used in the first part of the book.

🔗

A model for the distribution of the observations is assumed each time a simulation is carried out. The simulation itself involves the generation of random samples from that model for the given sample size and for a given value of the parameter. The statistic is evaluated and stored for each generated sample. Thereby, via the generation of many samples, an approximation of the sampling distribution of the statistic is produced. A probabilistic statement inferred from the Normal approximation can be compared to the results of the simulation. Substantial disagreement between the Normal approximation and the outcome of the simulations is an evidence that the Normal approximation may not be valid in the specific setting.

🔗

As an illustration, assume the statistic is the average price of a car. It is assumed that the price of a car follows an Exponential distribution with some unknown rate parameter $\lambda\text{.}$ We consider the sampling distribution of the average of 201 Exponential random variables. (Recall that in our sample there are 4 missing values among the 205 observations.) The expectation of the average is $1/\lambda\text{,}$ which is the expectation of a single Exponential random variable. The variance of a single observation is $1/\lambda^2\text{.}$ Consequently, the standard deviation of the average is $\sqrt{(1/\lambda^2)/201} = (1/\lambda)/\sqrt{201} = (1/\lambda)/14.17745 = 0.0705/\lambda\text{.}$

🔗

In the first part of the book we found out that for $\mathrm{Normal}(\mu,\sigma^2)\text{,}$ the Normal distribution with expectation $\mu$ and variance $\sigma^2\text{,}$ the central region that contains 95% of the distribution takes the form $\mu \pm 1.96\, \sigma$ (namely, the interval $[\mu-1.96\,\sigma,\mu + 1.96\, \sigma]$). Thereby, according to the Normal approximation for the sampling distribution of the average price we state that the region $1/\lambda \pm 1.96 \cdot 0.0705/\lambda$ should contain 95% of the distribution.

🔗

We may use simulations in order to validate this approximation for selected values of the rate parameter $\lambda\text{.}$ Hence, for example, we may choose $\lambda = 1/12,000$ (which corresponds to an expected price of $12,000 for a car) and validate the approximation for that parameter value.

🔗

The simulation itself is carried out by the generation of a sample of size $n=201$ from the $\mathrm{Exponential}(1/1200)$ distribution using the function “rexp” for generating Exponential samples.

The expression for generating a sample is “rexp(201,1/12000)”

The function for computing the average (mean) is applied to each sample and the result stored. We repeat this process a large number of times (100,000 is the typical number we use) in order to produce an approximation of the sampling distribution of the sample average. Finally, we check the relative frequency of cases where the simulated average is within the given range.

In the case where the simulated averages are stored in the sequence “X.bar” then we may use the expression “mean(abs(X.bar - 12000) <= 1.96*0.0705*12000)” in order to compute the relative frequency.

This relative frequency is an approximation of the required probability and may be compared to the target value of 0.95.

🔗

Let us run the proposed simulation for the sample size of $n=201$ and for a rate parameter equal to $\lambda = 1/12000\text{.}$ Observe that the expectation of the sample average is equal to $12,000$ and the standard deviation is $0.0705\times 12000\text{.}$ Hence:

🔗

X.bar <- rep(0,10^5)
for(i in 1:10^5) {
  X <- rexp(201,1/12000)
  X.bar[i] <- mean(X)
}
mean(abs(X.bar-12000) <= 1.96*0.0705*12000)

## [1] 0.94978

Observe that the simulation produces 0.9496 as the probability of the interval. This result is close enough to the target probability of 0.95, proposing that the Normal approximation is adequate in this example.

🔗

The simulation demonstrates the appropriateness of the Normal approximation for the specific value of the parameter that was used. In order to gain more confidence in the approximation we may want to consider other values as well. However, simulations in this book are used only for demonstration. Hence, in most cases where we conduct a simulation experiment, we conduct it only for a single evaluation of the parameters. We leave it to the curiosity of the reader to expand the simulations and try other evaluations of the parameters.

🔗

Simulations may also be used in order to compute probabilities in cases where the Normal approximation does not hold. As an illustration, consider the mid-range statistic. This statistic is computed as the average between the largest and the smallest values in the sample. This statistic is discussed in the next chapter.

🔗

Consider the case where we obtain 100 observations. Let the distribution of each observation be Uniform. Suppose we are interested as before in the central range that contains 95% of the distribution of the mid-range statistic. The Normal approximation does not apply in this case. Yet, if we specify the parameters of the Uniform distribution then we may use simulations in order to compute the range.

🔗

As a specific example let the distribution of an observation be $\mathrm{Uniform}(3,7)\text{.}$ In the simulation we generate a sample of size $n=100$ from this distribution

⁴

With the expression “runif(100,3,7)”.

and compute the mid-range for the sample.

🔗

For the computation of the statistic we need to obtain the minimal and the maximal values of the sample. The minimal value of a sequence is compute with the function “min”. The input to this function is a sequence and the output is the minimal value of the sequence. Similarly, the maximal value is computed with the function “max”. Again, the input to the function is a sequence and the output is the maximal value in the sequence. The statistic itself is obtained by adding the two extreme values to each other and dividing the sum by two.

⁵

If the sample is stored in an object by the name “X” then one may compute the mid-range statistic with the expression “(max(X)+min(X))/2”.

🔗

We produce, just as before, a large number of samples and compute the value of the statistic to each sample. The distribution of the simulated values of the statistic serves as an approximation of the sampling distribution of the statistic. The central range that contains 95% of the sampling distribution may be approximated with the aid of this simulated distribution.

🔗

Specifically, we approximate the central range by the identification of the 0.025-percentile and the 0.975-percentile of the simulated distribution. Between these two values are 95% of the simulated values of the statistic. The percentiles of a sequence of simulated values of the statistic can be identified with the aid of the function “quantile” that was presented in the first part of the book. The first argument to the function is a sequence of values and the second argument is a number $p$ between 0 and 1. The output of the function is the $p$-percentile of the sequence.

⁶

The $p$-percentile of a sequence is a number with the property that the proportion of entries with values smaller than that number is $p$ and the proportion of entries with values larger than the number is $1-p\text{.}$

The $p$-percentile of the simulated sequence serves as an approximation of the $p$-percentile of the sampling distribution of the statistic.

🔗

The second argument to the function “quantile” may be a sequence of values between 0 and 1. If so, the percentile for each value in the second argument is computed.

⁷

If the simulated values of the statistic are stored in a sequence by the name “mid.range” then the 0.025-percentile and the 0.975-percentile of the sequence can be computed with the expression “quantile(mid.range,c(0.025,0.975))”.

🔗

Let us carry out the simulation that produces an approximation of the central region that contains 95% of the sampling distribution of the mid-range statistic for the Uniform distribution:

🔗

mid.range <- rep(0,10^5)
for(i in 1:10^5) {
  X <- runif(100,3,7)
  mid.range[i] <- (max(X)+min(X))/2
}
quantile(mid.range,c(0.025,0.975))

##     2.5%    97.5% 
## 4.941019 5.058398

Observe that (approximately) 95% of the sampling distribution of the statistic are in the range $[4.941680, 5.059004]\text{.}$

🔗

Simulations can be used in order to compute the expectation, the standard deviation or any other numerical summary of the sampling distribution of a statistic. All one needs to do is compute the required summary for the simulated sequence of statistic values and hence obtain an approximation of the required summary. For example, we my use the sequence “mid.range” in order to obtain the expectation and the standard deviation of the mid-range statistic of a sample of 100 observations from the $\mathrm{Uniform}(3,7)$ distribution:

🔗

mean(mid.range)
sd(mid.range)

## [1] 4.999871
## [1] 0.02772162

The expectation of the statistic is obtained by the application of the function “mean” to the sequence. Observe that it is practically equal to 5. The standard deviation is obtained by the application of the function “sd”. Its value is approximately equal to 0.028.

🔗

Prev Top Next