Introduction to Statistical Thinking

In the previous two section we considered the estimation of the expectation and the variance of a measurement. The proposed estimators, the sample average for the expectation and the sample variance for the variance, are not tied to any specific model for the distribution of the measurement. They may be applied to data whether or not a theoretical model for the distribution of the measurement is assumed.

In the cases where a theoretical model for the measurement is assumed one may be interested in the estimation of the specific parameters associated with this model. In the first part of the book we introduced the Binomial, the Poisson, the Uniform, the Exponential, and the Normal models for the distribution of measurements. In this section we consider the estimation of the parameters that determine each of these theoretical distributions based on a sample generated from the same distribution. In some cases the estimators coincide with the estimators considered in the previous sections. In other cases the estimators are different.

Start with the Binomial distribution. We will be interested in the special case \(X \sim \mathrm{Binomial}(1,p)\text{.}\) This case involves the outcome of a single trial. The trial has two possible outcomes, one of them is designated as “success” and the other as “failure”. The parameter \(p\) is the probability of the success. The \(\mathrm{Binomial}(1,p)\) distribution is also called the Bernoulli distribution. Our concern is the estimation of the parameter \(p\) based on a sample of observations from this Bernoulli distribution.

This estimation problem emerges in many settings that involve the assessment of the probability of an event based on a sample of \(n\) observations. In each observation the event either occurs or not. A natural estimator of the probability of the event is its relative frequency in the sample. Let us show that this estimator can be represented as an average of a Bernoulli sample and the sample average is used for the estimation of a Bernoulli expectation.

Consider an event, one may code a measurement \(X\text{,}\) associated with an observation, by 1 if the event occurs and by 0 if it does not. Given a sample of size \(n\text{,}\) one thereby produces \(n\) observations with values 0 or 1. An observation has the value 1 if the event occurs for that observation or, else, the value is 0.

Notice that \(\Expec(X) = 1 \cdot p = p\text{.}\) Consequently, the probability of the event is equal to the expectation of the Bernoulli measurement

¹

The expectation of \(X\sim\mathrm{Binomial}(n,p)\) is \(\Expec(X)=np\text{.}\) In the Bernoulli case \(n=1\text{.}\) Therefore, \(\Expec(X) = 1\cdot p = p\text{.}\)

. It turns out that the parameter one seeks to estimate is the expectation of a Bernoulli measurement. The estimation is based on a sample of size \(n\) of Bernoulli observations.

In Section 10.3 it was proposed to use the sample average as an estimate of the expectation. The sample average is the sum of the observations, divided by the number of observation. In the specific case of a sample of Bernoulli observations, the sum of observation is the sum of zeros and one. The zeros do not contribute to the sum. Hence, the sum is equal to the number of times that 1 occurs, namely the frequency of the occurrences of the event. When we divide by the sample size we get the relative frequency of the occurrences. The conclusion is that the sample average of the Bernoulli observations and the relative frequency of occurrences of the event in the sample are the same. Consequently, the sample relative frequency of the event is also a sample average that estimates the expectation of the Bernoulli measurement.

We seek to estimate \(p\text{,}\) the probability of the event. The estimator is the relative frequency of the event in the sample. We denote this estimator by \(\hat P\text{.}\) This estimator is a sample average of Bernoulli observations that is used in order to estimate the expectation of the Bernoulli distribution. From the discussion in Section 10.3 one may conclude that this estimator is an unbiased estimator of \(p\) (namely, \(\Expec(\hat P) = p\)) and that its variance is equal to:

where the variance of the measurement is obtained from the formula for the variance of a \(\mathrm{Binomial}(1,p)\) distribution

²

The variance of \(X\sim\mathrm{Binomial}(n,p)\) is \(\Var(X)=np(1-p)\text{.}\) In the Bernoulli case \(n=1\text{.}\) Therefore, \(\Var(X) = 1\cdot p(1-p) = p(1-p)\text{.}\)

.

The second example of an integer valued random variable that was considered in the first part of the book is the \(\mathrm{Poisson}(\lambda)\) distribution. Recall that \(\lambda\) is the expectation of a Poisson measurement. Hence, one may use the sample average of Poisson observations in order to estimate this parameter.

The first example of a continuous distribution that was discussed in the first part of the book is the \(\mathrm{Uniform}(a,b)\) distribution. This distribution is parameterized by \(a\) and \(b\text{,}\) the end-points of the interval over which the distribution is defined. A natural estimator of \(a\) is the smallest value observed and a natural estimator of \(b\) is the largest value. One may use the function min for the computation of the former estimate from the sample and use the function max for the computation of the later. Both estimators are slightly biased but have a relatively small mean square error.

Next considered the \(X \sim \mathrm{Exponential}(\lambda)\) random variable. This distribution was applied in this chapter to model the distribution of the prices of cars. The distribution is characterized by the rate parameter \(\lambda\text{.}\) In order to estimate the rate one may notice the relation between it and the expectation of the measurement:

The rate is equal to the reciprocal of the expectation. The expectation can be estimated by the sample average. Hence a natural proposal is to use the reciprocal of the sample average as an estimator of the rate:

The final example that we mention is the \(\mathrm{Normal}(\mu,\sigma^2)\) case. The parameter \(\mu\) is the expectation of the measurement and may be estimated by the sample average \(\bar X\text{.}\) The parameter \(\sigma^2\) is the variance of a measurement, and can be estimated using the sample variance \(S^2\text{.}\)