1.
What is wrong about saying, "The defendant is innocent." based on the US system of criminal trials?
Section 9.5 Interpreting hypothesis tests
Interpreting the results of hypothesis tests is one of the more challenging aspects of this method for statistical inference. In this section, we will focus on ways to help with deciphering the process and address some common misconceptions.
Subsection 9.5.1 Two possible outcomes
In Section 9.3, we mentioned that given a pre-specified significance level \(\alpha\) there are two possible outcomes of a hypothesis test:
-
If the \(p\)-value is less than \(\alpha\text{,}\) then we reject the null hypothesis \(H_0\) in favor of \(H_A\text{.}\)
-
If the \(p\)-value is greater than or equal to \(\alpha\text{,}\) we fail to reject the null hypothesis \(H_0\text{.}\)
Unfortunately, the latter result is often misinterpreted as "accepting the null hypothesis \(H_0\text{.}\)" While at first glance it may seem that the statements "failing to reject \(H_0\)" and "accepting \(H_0\)" are equivalent, there actually is a subtle difference. Saying that we "accept the null hypothesis \(H_0\)" is equivalent to stating that "we think the null hypothesis \(H_0\) is true." However, saying that we "fail to reject the null hypothesis \(H_0\)" is saying something else: "While \(H_0\) might still be false, we do not have enough evidence to say so." In other words, there is an absence of enough proof. However, the absence of proof is not proof of absence.
To further shed light on this distinction, let’s use the United States criminal justice system as an analogy. A criminal trial in the United States is a similar situation to hypothesis tests whereby a choice between two contradictory claims must be made about a defendant who is on trial:
-
The defendant is truly either "innocent" or "guilty."
-
The defendant is presumed "innocent until proven guilty."
-
The defendant is found guilty only if there is strong evidence that the defendant is guilty. The phrase "beyond a reasonable doubt" is often used as a guideline for determining a cutoff for when enough evidence exists to find the defendant guilty.
-
The defendant is found to be either "not guilty" or "guilty" in the ultimate verdict.
In other words, not guilty verdicts are not suggesting the defendant is innocent, but instead that "while the defendant may still actually be guilty, there was not enough evidence to prove this fact." Now let’s make the connection with hypothesis tests:
-
Either the null hypothesis \(H_0\) or the alternative hypothesis \(H_A\) is true.
-
Hypothesis tests are conducted assuming the null hypothesis \(H_0\) is true.
-
We reject the null hypothesis \(H_0\) in favor of \(H_A\) only if the evidence found in the sample suggests that \(H_A\) is true. The significance level \(\alpha\) is used as a guideline to set the threshold on just how strong of evidence we require.
-
We ultimately decide to either "fail to reject \(H_0\)" or "reject \(H_0\text{.}\)"
So while gut instinct may suggest "failing to reject \(H_0\)" and "accepting \(H_0\)" are equivalent statements, they are not. "Accepting \(H_0\)" is equivalent to finding a defendant innocent. However, courts do not find defendants "innocent," but rather they find them "not guilty." Putting things differently, defense attorneys do not need to prove that their clients are innocent, rather they only need to prove that clients are not "guilty beyond a reasonable doubt."
So going back to our songs activity in Section 9.4, recall that our hypothesis test was \(H_0: p_{m} - p_{d} = 0\) versus \(H_A: p_{m} - p_{d} \gt 0\) and that we used a pre-specified significance level of \(\alpha = 0.1\text{.}\) Since the \(p\)-value was smaller than \(\alpha = 0.1\text{,}\) we rejected \(H_0\text{.}\) In other words, we found needed levels of evidence in this particular sample to say that \(H_0\) is false at the \(\alpha = 0.1\) significance level. We also state this conclusion using non-statistical language: we found enough evidence in this data to suggest that there was a difference in the popularity of our two genres of music.
Subsection 9.5.2 Types of errors
Unfortunately, there is some chance a jury or a judge can make an incorrect decision in a criminal trial by reaching the wrong verdict. For example, finding a truly innocent defendant "guilty." Or on the other hand, finding a truly guilty defendant "not guilty." This can often stem from the fact that prosecutors do not have access to all the relevant evidence, but instead are limited to whatever evidence the police can find.
The same holds for hypothesis tests. We can make incorrect decisions about a population parameter because we only have a sample of data from the population and thus sampling variation can lead us to incorrect conclusions.
There are two possible erroneous conclusions in a criminal trial: either (1) a truly innocent person is found guilty or (2) a truly guilty person is found not guilty. Similarly, there are two possible errors in a hypothesis test: either (1) rejecting \(H_0\) when in fact \(H_0\) is true, called a Type I error or (2) failing to reject \(H_0\) when in fact \(H_0\) is false, called a Type II error. Another term used for "Type I error" is "false positive," while another term for "Type II error" is "false negative."
This risk of error is the price researchers pay for basing inference on a sample instead of performing a census on the entire population. But as we have seen in our numerous examples and activities so far, censuses are often very expensive and other times impossible, and thus researchers have no choice but to use a sample. Thus in any hypothesis test based on a sample, we have no choice but to tolerate some chance that a Type I error will be made and some chance that a Type II error will occur.
To help understand Type I and Type II errors, we apply these terms to our criminal justice analogy in Table 9.5.1. Thus, a Type I error corresponds to incorrectly putting a truly innocent person in jail, whereas a Type II error corresponds to letting a truly guilty person go free.
| Verdict | Truly not guilty | Truly guilty |
|---|---|---|
| Not guilty verdict | Correct | Type II error |
| Guilty verdict | Type I error | Correct |
Let’s show the corresponding table in Table 9.5.2 for hypothesis tests.
| Decision | \(H_0\) true | \(H_A\) true |
|---|---|---|
| Fail to reject \(H_0\) | Correct | Type II error |
| Reject \(H_0\) | Type I error | Correct |
Subsection 9.5.3 How do we choose \(\alpha\text{?}\)
If we are using a sample to make inferences about a population, we are operating under uncertainty and run the risk of making statistical errors. These are not errors in calculations or in the procedure used, but errors in the sense that the sample used may lead us to construct a confidence interval that does not contain the true value of the population parameter, for example. In the case of hypothesis testing, there are two well-defined errors: a Type I and a Type II error:
-
A Type I Error is rejecting the null hypothesis when it is true. The probability of a Type I Error occurring is \(\alpha\text{,}\) the significance level, which we defined in Subsection 9.1.1 and in Section 9.3.
-
A Type II Error is failing to reject the null hypothesis when it is false. The probability of a Type II Error is denoted by \(\beta\text{.}\) The value of \(1-\beta\) is known as the power of the test.
Ideally, we would like to minimize the errors, and we would like \(\alpha = 0\) and \(\beta = 0\text{.}\) However, this is not possible as there will always be the possibility of committing one of these errors when making a decision based on sample data. Furthermore, these two error probabilities are inversely related. As the probability of a Type I error goes down, the probability of a Type II error goes up.
When constructing a hypothesis test, we have control of the probability of committing a Type I Error because we can decide what is the significance level \(\alpha\) we want to use. Once \(\alpha\) has been pre-specified, we try to minimize \(\beta\text{,}\) the fraction of incorrect non-rejections of the null hypothesis.
So for example if we used \(\alpha = 0.01\text{,}\) we would be using a hypothesis testing procedure that in the long run would incorrectly reject the null hypothesis \(H_0\) one percent of the time. This is analogous to setting the confidence level of a confidence interval.
So what value should you use for \(\alpha\text{?}\) While different fields of study have adopted different conventions, although \(\alpha = 0.05\) is perhaps the most popular threshold, there is nothing special about this or any other number. Please review Subsection 9.1.1 and our discussion about \(\alpha\) and our tolerance for uncertainty. In addition, observe that choosing a relatively small value of \(\alpha\) reduces our chances of rejecting the null hypothesis, and also of committing a Type I Error; but increases the probability of committing a Type II Error.
On the other hand, choosing a relatively large value of \(\alpha\) increases the chances of failing to reject the null hypothesis, and also of committing a Type I Error; but reduces the probability of committing a Type II Error. Depending on the problem at hand, we may be willing to have a larger significance level in certain scenarios and a smaller significance level in others.
Exercises Exercises
2.
What is the purpose of hypothesis testing?
3.
What are some flaws with hypothesis testing? How could we alleviate them?
4.
Consider two \(\alpha\) significance levels of 0.1 and 0.01. Of the two, which would lead to a higher chance of committing a Type I Error?
