Introduction to Statistical Thinking

The variable "price" in the file "cars.csv" contains data on the prices of different types of cars that were sold in the United States during 1985. The average price of a car back then — the average of the variable "price" — was $13,207. One may be interested in the question: Do Americans pay today for cars a different price than what they used to pay in the 80’s? Has the price of cars changed significantly since 1985?

The average price of a car in the United States in 2009 was $27,958

¹

Source: "http://wiki.answers.com/Q/Average_price_of_a_car_in_2009".

. Clearly, this figure is higher than $13,207. However, in order to produce a fair answer to the question we have to take into account that, due to inflation, the prices of all products went up during these years. A more meaningful comparison will involve the current prices of cars in terms of 1985 Dollars. Indeed, if we take into account inflation then we get that, on the average, the cost of today’s cars corresponds to an average price of $13,662 in 1985 values

²

Source: "http://www.westegg.com/inflation/". The interpretation of adjusting prices to inflation is that our comparison will correspond to changes in the price of cars, relative to other items that enter into the computation of the Consumer Price Index.

. This price is still higher than the prices in the 1985 but not as much. The question we are asking is: "Is the difference between $13,207 and $13,662 significant or is it not so?".

In order to give a statistical answer to this question we carry out a statistical test. The specific test is conducted with the aid of the function "t.test". Later we will discuss in more details some of the arguments that may be used in this function. Currently, we simply apply it to the data stored in the variable "price" to test that the expected price is different than the $13,662, the average price of a car in 2009, adjusted for inflation:

cars <- read.csv("_data/cars.csv")
t.test(cars$price,mu=13662)

## 
##  One Sample t-test
## 
## data:  cars$price
## t = -0.81148, df = 200, p-value = 0.4181
## alternative hypothesis: true mean is not equal to 13662
## 95 percent confidence interval:
##  12101.80 14312.46
## sample estimates:
## mean of x 
##  13207.13

The data in the file "cars.csv" is read into a data frame that is given the name "cars". Afterwards, the data on prices of car types in 1985 is entered as the first argument to the function "t.test". The other argument is the expected value that we want to test, the current average price of cars, given in terms of 1985 Dollar value. The output of the function is reported under the title: "One Sample t-test".

Let us read the report from the bottom up. The bottom part of the report describes the confidence interval and the point estimate of the expected price of a car in 1985, based on the given data. Indeed, the last line reports the sample average of the price, which is equal to 13,207.13. This number, the average of the 201 non-missing values of the variable "price", serves as the estimate of the expected price of a car in 1985. The 95% confidence interval of the expectation, the interval [12101.80, 14312.46], is presented on the 4th line from the bottom. This is the confidence interval for the expectation that was computed in Subsection 11.2.2

³

As a matter of fact, the confidence interval computed in Subsection 11.2.2 is [12108.47, 14305.79], which is not identical to the confidence that appears in the report. The reason for the discrepancy is that we used the 0.975-percentile of the Normal distribution, 1.96, whereas the confidence interval computed here uses the 0.975-percentile of the \(t\)-distribution on 201-1=200 degrees of freedom. The latter is equal to 1.971896. Nonetheless, for all practical purposes, the two confidence intervals are the same.

.

The information relevant to conducting the statistical test itself is given in the upper part of the report. Specifically, it is reported that the data in "cars$price" is used in order to carry out the test. Based on this data a test statistic is computed and obtains the value of "t = -0.8115". This statistic is associated with the \(t\)-distribution with "df = 200" degrees of freedom. The last quantity that is being reported is denoted the \(p\)-value and it obtains the value "p-value = 0.4181". The test may be carried out with the aid of the value of the \(t\) statistic or, more directly, using the \(p\)-value. Currently we will use the \(p\)-value.

The test itself examines the hypothesis that the expected price of a car in 1985 was equal to $13,662, the average price of a car in 2009, given in 1985 values. This hypothesis is called the null hypothesis. The alternative hypothesis is that the expected price of a car in 1985 was not equal to that figure. The specification of the alternative hypothesis is reported on the third line of the output of the function "t.test".

One may decide between the two hypothesis on the basis of the size of the \(p\)-value. The rule of thumb is to reject the null hypothesis, and thus accept the alternative hypothesis, if the \(p\)-value is less than 0.05. In the current example the \(p\)-value is equal 0.4181 and is larger than 0.05. Consequently, we may conclude that the expected price of a car in 1985 was not significantly different than the current price of a car.

In the rest of this section we give a more rigorous explanation of the theory and practice of statistical hypothesis testing.

The initial step in statistical inference in general, and in statistical hypothesis testing in particular, is the formulation of the statistical model and the identification of the parameter/s that should be investigated. In the current situation the statistical model may correspond to the assumption that the data in the variable "price" are an instance of a random sample (of size \(n=201\)). The parameter that we want to investigate is the expectation of the measurement that produced the sample. The variance of the measurement is also relevant for the investigation.

After the statistical model has been set, one may split the process of testing a statistical hypothesis into three steps: (i) formulation of the hypotheses, (ii) specification of the test, and (iii) reaching the final conclusion. The first two steps are carried out on the basis of the probabilistic characteristics of the statistical model and in the context of the sampling distribution. In principal, the first two steps may be conducted in the planning stage prior to the collection of the observations. Only the third step involves the actual data. In the example that was considered in the previous subsection the third step was applied to the data in the variable "price" using the function "t.test".

(i) Formulating the hypotheses: A statistical model involves a parameter that is the target of the investigation. In principle, this parameter may obtain any value within a range of possible values. The formulation of the hypothesis corresponds to splitting the range of values into two sub-collections: a sub-collection that emerges in response to the presence of the phenomena and a sub-collection that emerges in response to the situation when the phenomena is absent. The sub-collection of parameter values where the phenomena is absent is called the null hypothesis and is marked as "\(H_0\)". The other sub-collection, the one reflecting the presence of the phenomena, is denoted the alternative hypothesis and is marked "\(H_1\)".

For example, consider the price of cars. Assume that the phenomena one wishes to investigate is the change in the relative price of a car in the 80’s as compared to prices today. The parameter of interest is the expected price of cars back then, which we denote by \(\Expec(X)\text{.}\) The formulation of the statement that the expected price of cars has changed is "\(\Expec(X) \not = 13,662\)". This statement corresponds to the presence of a phenomena, to a change, and is customarily defined as the alternative hypothesis. On the other hand, the situation "\(\Expec(X) = 13,662\)" corresponds to not having any change in the price of cars. Hence, this situation corresponds to the absence of the phenomena and is denoted the null hypothesis. In summary, in order to investigate the change in the relative price of cars we my consider the null hypothesis "\(H_0:\Expec(X) = 13,662\)" and test it against the alternative hypothesis "\(H_1: \Expec(X)\not = 13,662\)".

A variation in the formulation of the phenomena can change the definition of the null and alternative hypotheses. For example, if the intention is to investigate the rise in the price of cars then the phenomena will correspond to the expected price in 1985 being less than $13,662. Accordingly, the alternative hypothesis should be defined as \(H_1: \Expec(X) < 13,662\text{,}\) with the null hypothesis defined as \(H_0: \Expec(X) \geq 13,662\text{.}\) Observe that in this case an expected price larger than $13,662 relates to the phenomena of rising (relative) prices not taking place.

On the other hand, if one would wants to investigate a decrease in the price then one should define the alternative hypothesis to be \(H_1: \Expec(X) > 13,662\text{,}\) with the null hypothesis being \(H_0: \Expec(X) \leq 13,662\text{.}\)

The type of alternative that was considered in the example, \(H_1: \Expec(X) \not = 13,622\) is called a two-sided alternative. The other two types of alternative hypotheses that were considered thereafter, \(H_1: \Expec(X) < 13,662\) and \(H_1: \Expec(X) > 13,662\text{,}\) are both called one-sided alternatives.

In summary, the formulation of the hypothesis is a reflection of the phenomena one wishes to examine. The setting associated with the presence of the phenomena is denoted the alternative hypothesis and the complimentary setting, the setting where the phenomena is absent, is denoted the null hypothesis.

(ii) Specifying the test: The second step in hypothesis testing involves the selection of the decision rule, i.e. the statistical test, to be used in order to decide between the two hypotheses. The decision rule is composed of a statistic and a subset of values of the statistic that correspond to the rejection of the null hypothesis. The statistic is called the test statistic and the subset of values is called the rejection region. The decision is to reject the null hypothesis (and consequently choose the alternative hypothesis) if the test statistic falls in the rejection region. Otherwise, if the test statistic does not fall in the rejection region then the null hypothesis is selected.

Return to the example in which we test between \(H_0:\Expec(X) = 13,662\) and \(H_1:\Expec(X) \not= 13,662\text{.}\) One may compute the statistic:

where \(\bar X\) is the sample average (of the variable "price"), \(S\) is the sample standard deviation, and \(n\) is the sample size (\(n = 201\) in the current example).

The sample average \(\bar X\) is an estimator of the expected price of the car. In principle, the statistic \(T\) measures the discrepancy between the estimated value of the expectation (\(\bar X\)) and the expected value under the null hypothesis (\(\Expec(X) = 13,662\)). This discrepancy is measured in units of the (estimated) standard deviation of the sample average

⁴

If the variance of the measurement \(\Var(X)\) was known one could have use \(Z = (\bar X - 13,662)/\sqrt{\Var{X}/n}\) as a test statistic. This statistic corresponds to the discrepancy of the sample average from the null expectation in units of its standard deviation, i.e. the \(z\)-value of the sample average. Since the variance of the observation is unknown, we use an estimator of the variance (\(S^2\)) instead.

.

If the null hypothesis \(H_0:\Expec(X) = 13,662\) is true then the sampling distribution of the sample average \(\bar X\) should be concentrated about the value 13,662. Values of the sample average much larger or much smaller than this value may serve as evidence against the null hypothesis.

In reflection, if the null hypothesis holds true then the values of the sampling distribution of the statistic \(T\) should tend to be in the vicinity of 0. Values with a relative small absolute value are consistent with the null hypothesis. On the other hand, extremely positive or extremely negative values of the statistic indicate that the null hypothesis is probably false.

It is natural to set a value \(c\) and to reject the null hypothesis whenever the absolute value of the statistic \(T\) is larger than \(c\text{.}\) The resulting rejection region is of the form \(\{|T| > c\}\text{.}\) The rule of thumb, again, is to take threshold \(c\) to be equal the 0.975-percentile of the \(t\)-distribution on \(n-1\) degrees of freedom, where \(n\) is the sample size. In the current example, the sample size is \(n=201\) and the percentile of the \(t\)-distribution is qt(0.975,200) = 1.971896. Consequently, the subset \(\{|T| > 1.971896\}\) is the rejection region of the test.

A change in the hypotheses that are being tested may lead to a change in the test statistic and/or the rejection region. For example, for testing \(H_0: \Expec(X) \geq 13,662\) versus \(H_1: \Expec(X) < 13,662\) one may still use the same test statistic \(T\) as before. However, only very negative values of the statistic are inconsistent with the null hypothesis. It turns out that the rejection region in this case is of the form \(\{T < -1.652508\}\text{,}\) where qt(0.05,200) = -1.652508 is the 0.05-percentile of the \(t\)-distribution on 200 degrees of freedom. On the other hand, the rejection region for testing between \(H_0: \Expec(X) \leq 13,662\) and \(H_1: \Expec(X) > 13,662\) is \(\{T > 1.652508\}\text{.}\) In this case, qt(0.95,200) = 1.652508 is the 0.95-percentile of the \(t\)-distribution on 200 degrees of freedom.

Selecting the test statistic and deciding what rejection region to use specifies the statistical test and completes the second step.

(iii) Reaching a conclusion: After the stage is set, all that is left is to apply the test to the observed data. This is done by computing the observed value of the test statistic and checking whether or not the observed value belongs to the rejection region. If it does belong to the rejection region then the decision is to reject the null hypothesis. Otherwise, if the statistic does not belong to the rejection region, then the decision is to accept the null hypothesis.

Return to the example of testing the price of car types. The observed value of the \(T\) statistic is part of the output of the application of the function "t.test" to the data. The value is "t = -0.8115". As an exercise, let us recompute directly from the data the value of the \(T\) statistic:

x.bar <- mean(cars$price,na.rm=TRUE)
x.bar
s <- sd(cars$price,na.rm=TRUE)

## [1] 13207.13

The observed value of the sample average is \(\bar x = 13207.13\) and the observed value of the sample standard deviation is \(s = 7947.066\text{.}\) The sample size (due to having 4 missing values) is \(n=201\text{.}\) The formula for the computation of the test statistic in this example is \(t = [\bar x - 13,662]/[s/\sqrt{n}]\text{.}\) Plugging in this formula the sample size and the computed values of the sample average and standard deviation produces:

(x.bar - 13662)/(s/sqrt(201))

## [1] -0.8114824

This value, after rounding up, is equal to the value "t = -0.8115" that is reported in the output of the function "t.test".

The critical threshold for the absolute value of the \(T\) statistic on \(201-1 = 200\) degrees of freedom is qt(0.975,200) = 1.971896. Since the absolute observed value (\(|t| = 0.8114824\)) is less then the threshold we get that the value of the statistic does not belong to the rejection region (which is composed of absolute values larger than the threshold). Consequently, we accept the null hypothesis. This null hypothesis declares that the expected price of a car was equal to the current expected price (after adjusting for the change in Consumer Price Index)

⁵

Previously, we carried out the same test using the \(p\)-value. The computed \(p\)-value in this example is 0.4181. The null hypothesis was accepted since this value is larger than 0.05. As a matter of fact, the test that uses the \(T\) statistic as a test statistic and reject the null hypothesis for absolute values larger than qt(0.975,n-1) is equivalent to the test that uses the \(p\)-value and rejects the null hypothesis for \(p\)-values less than 0.05. Below we discuss the computation of the \(p\)-value.

.

The \(T\) statistic was proposed for testing a change in the price of a car. This statistic measures the discrepancy between the sample average price of a car and the expected value of the sample average, where the expectation is computed under the null hypothesis. The structure of the rejection region of the test is \(\{|T| > c\}\text{,}\) where \(c\) is an appropriate threshold. In the current example the value of the threshold \(c\) was set to be equal to qt(0.975,200) = 1.971896. In general, the specification of the threshold \(c\) depends on the error probabilities that are associated with the test. In this section we describe these error probabilities.

The process of making decisions may involve errors. In the case of hypothesis testing one may specify two types of error. On the one hand, the case may be that the null hypothesis is correct (in the example, \(\Expec(X) = 13,662\)). However, the data is such that the null hypothesis is rejected (here, \(|T| > 1.971896\)). This error is called a Type I error.

A different type of error occurs when the alternative hypothesis holds (\(\Expec(X) \not= 13,662\)) but the null hypothesis is not rejected (\(|T| \leq 1.971896\)). This other type of error is called Type II error. A summary of the types of errors can be found in Table Table 12.2.1:

In statistical testing of hypothesis the two types of error are not treated symmetrically. Rather, making a Type I error is considered more severe than making a Type II error. Consequently, the test’s decision rule is designed so as to assure an acceptable probability of making a Type I error. Reducing the probability of a Type II error is desirable, but is of secondary importance.

Indeed, in the example that deals with the price of car types the threshold was set as high as qt(0.975,200) = 1.971896 in order to reject the null hypothesis. It is not sufficient that the sample average is not equal to 13,662 (corresponding to a threshold of 0), but it has to be significantly different from the expectation under the null hypothesis, the distance between the sample average and the null expectation should be relatively large, in order to exclude \(H_0\) as an option.

The significance level of the evidence for rejecting the null hypothesis is based on the probability of the Type I error. The probabilities associated with the different types of error are presented in Table Table 12.2.2:

Observe that the probability of a Type I error is called the significance level. The significance level is set at some pre-specified level such as 5% or 1%, with 5% being the most widely used level. In particular, setting the threshold in the example to be equal to qt(0.975,200) = 1.971896 produces a test with a 5% significance level.

This lack of symmetry between the two hypothesis proposes another interpretation of the difference between the hypothesis. According to this interpretation the null hypothesis is the one in which the cost of making an error is greater. Thus, when one separates the collection of parameter values into two subsets then the subset that is associated with a more severe error is designated as the null hypothesis and the other subset becomes the alternative.

For example, a new drug must pass a sequence of clinical trials before it is approved for distribution. In these trials one may want to test whether the new drug produces beneficial effect in comparison to the current treatment. Naturally, the null hypothesis in this case would be that the new drug is no better than the current treatment and the alternative hypothesis would be that it is better. Only if the clinical trials demonstrates a significant beneficiary effect of the new drug would it be released for marketing.

In scientific research, in general, the currently accepted theory, the conservative explanation, is designated as the null hypothesis. A claim for novelty in the form of an alternative explanation requires strong evidence in order for it to be accepted and be favored over the traditional explanation. Hence, the novel explanation is designated as the alternative hypothesis. It replaces the current theory only if the empirical data clearly supports its. The test statistic is a summary of the empirical data. The rejection region corresponds to values that are unlikely to be observed according to the current theory. Obtaining a value in the rejection region is an indication that the current theory is probably not adequate and should be replaced by an explanation that is more consistent with the empirical evidence.

The second type of error probability in Table 12.2.2 is the probability of a Type II error. Instead of dealing directly with this probability the tradition is to consider the complementary probability that corresponds to the probability of not making a Type II error. This complementary probability is called the statistical power:

The statistical power is the probability of rejecting the null hypothesis when the state of nature is the alternative hypothesis. (In comparison, the significance level is the probability of rejecting the null hypothesis when the state of nature is the null hypothesis.) When comparing two decision rules for testing hypothesis, both having the same significance level, the one that possesses a higher statistical power should be favored.

The \(p\)-value is another test statistic. It is associated with a specific test statistic and a structure of the rejection region. The \(p\)-value is equal to the significance level of the test in which the observed value of the statistic serves as the threshold. In the current example, where the \(T\) is the underlying test statistic and the structure of the rejection region is of the form \(\{|T| > c\}\) then the \(p\)-value is equal to the probability of rejecting the null hypothesis in the case where the threshold \(c\) is equal to the observed absolute value of the \(T\) statistic. In other words:

where \(t=-0.8114824\) is the observed value of the \(T\) statistic and the computation of the probability is conducted under the null hypothesis.

Specifically, under the null hypothesis \(H_0: \Expec(X) = 13,662\) we get that the distribution of the statistic \(T = [\bar X - 13,662]/[S/\sqrt{n}]\) is the \(t\)-distribution on \(n-1 = 200\) degrees of freedom. The probability of the event \(\{|T| > 0.8114824\}\) corresponds to the sum of the probabilities of both tails of the distribution. By the symmetry of the \(t\)-distribution this equals twice the probability of the upper tail:

When we compute this probability in R we get:

2*(1-pt(0.8114824,200))

## [1] 0.4180534

This probability is equal, after rounding up, to the probability "p-value = 0.4181" that is reported in the output of the function "t.test".

The \(p\)-value is a function of the data. In the particular data set the computed value of the \(T\) statistic was -0.8114824. For a different data set the evaluation of the statistic would have produced a different value. As a result, the threshold that would have been used in the computation would have been different, thereby changing the numerical value of the \(p\)-value. Being a function of the data, we conclude that the \(p\)-value is a statistic.

The \(p\)-value is used as a test statistic by comparing its value to the pre-defined significance level. If the significance level is 1% then the null hypothesis is rejected for \(p\)-values less that 0.01. Likewise, if the significance level is set at the 5% level then the null hypothesis is rejected for \(p\)-values less than 0.05.

The statistical test that is based directly on the \(T\) statistic and the statistical test that is based on the \(p\)-value are equivalent to each other. The one rejects the null hypothesis if, and only if, the other does so. The advantage of using the \(p\)-value as the test statistic is that no further probabilistic computations are required. The \(p\)-value is compared directly to the significance level we seek. For the test that examines the \(T\) statistic we still need to identify the threshold associated with the given significance level.

In the next 2 sections we extend the discussion of the \(t\)-test and give further examples to the use of the function "t.test". We also deal with tests on probabilities of events and introduce the function "prop.test" for conducting such tests.