Statistical Inference via Data Science A ModernDive into R and the Tidyverse (Second Edition)

Much like the terminology, notation, and definitions relating to sampling you saw in Section 7.2, there are a lot of terminology, notation, and definitions related to hypothesis testing as well. Some of this was introduced in Section 9.1. Learning these may seem like a very daunting task at first. However, with practice, practice, and more practice, anyone can master them.

First, a hypothesis is a statement about the value of an unknown population parameter. In our genre popularity activity, our population parameter of interest is the difference in population proportions \(p_{m} - p_{d}\text{.}\) Hypothesis tests can involve any of the population parameters in the five inference scenarios we will cover in this book and also more advanced types we will not cover here.

Second, a hypothesis test consists of a test between two competing hypotheses: (1) a null hypothesis \(H_0\) (pronounced "H-naught") versus (2) an alternative hypothesis \(H_A\) (also denoted \(H_1\)).

When working with the comparison of two populations parameters, typically, the null hypothesis is a claim that there is "no effect" or "no difference of interest." In many cases, the null hypothesis represents the status quo. Furthermore, the alternative hypothesis is the claim the experimenter or researcher wants to establish or find evidence to support. It is viewed as a "challenger" hypothesis to the null hypothesis \(H_0\text{.}\) In our genre popularity activity, an appropriate hypothesis test would be:

Note some of the choices we have made. First, we set the null hypothesis \(H_0\) to be that there is no difference in popularity rate and the "challenger" alternative hypothesis \(H_A\) to be that there is a difference in favor of metal. As discussed earlier, the null hypothesis is set to reflect a situation of "no change." As we discussed earlier, in this case, \(H_0\) corresponds to there being no difference in popularity. Furthermore, we set \(H_A\) to be that metal is popular at a higher rate, a subjective choice reflecting a prior suspicion we have that this is the case. As discussed earlier this is a one-sided test . It can be left- or right-sided, and this becomes clear once we express it in terms of proportions. If someone else however does not share such suspicions and only wants to investigate that there is a difference, whether higher or lower, they would construct a two-sided test .

We can re-express the formulation of our hypothesis test using the mathematical notation for our population parameter of interest, the difference in population proportions \(p_{m} - p_{d}\text{:}\)

Observe how the alternative hypothesis \(H_A\) is written \(p_{m} - p_{d} \gt 0\text{.}\) Since we have chosen this particular formulation, the one-sided test becomes right-sided because we are looking for a difference that is greater than zero as evidence to reject the null hypothesis. Had we opted for a two-sided alternative, we would have set \(p_{m} - p_{d} \neq 0\text{.}\) We work here with the right-sided test and will present an example of a two-sided test in Section 9.6.

Third, a test statistic is a point estimate/sample statistic formula used for hypothesis testing. Note that a sample statistic is merely a summary statistic based on a sample of observations. Recall we saw in Chapter 3 that a summary statistic takes in many values and returns only one. Here, the samples would be the \(n_m = 1000\) metal songs and the \(n_d = 1000\) deep house songs. Hence, the point estimate of interest is the difference in sample proportions \(\widehat{p}_{m} - \widehat{p}_{d}\text{.}\)

Fourth, the observed test statistic is the value of the test statistic that we observed in real life. In our case, we computed this value using the data saved in the spotify_metal_deephouse data frame. It was the observed difference \(\widehat{p}_{m} - \widehat{p}_{d}\) in favor of metal songs.

Fifth, the null distribution is the sampling distribution of the test statistic assuming the null hypothesis \(H_0\) is true. Let’s unpack these ideas slowly. The key to understanding the null distribution is that the null hypothesis \(H_0\) is assumed to be true. We are not saying that \(H_0\) is true at this point, we are only assuming it to be true for hypothesis-testing purposes. In our case, this corresponds to our hypothesized universe of no difference in popularity rates. Assuming the null hypothesis \(H_0\text{,}\) also stated as "Under \(H_0\text{,}\)" how does the test statistic vary due to sampling variation? In our case, how will the difference in sample proportions \(\widehat{p}_{m} - \widehat{p}_{d}\) vary due to sampling under \(H_0\text{?}\) Recall from Subsection 7.3.4 that distributions displaying how point estimates vary due to sampling variation are called sampling distributions. The only additional thing to keep in mind about null distributions is that they are sampling distributions assuming the null hypothesis \(H_0\) is true.

Sixth, the \(p\)-value is the probability of obtaining a test statistic just as extreme as or more extreme than the observed test statistic assuming the null hypothesis \(H_0\) is true. You can think of the \(p\)-value as a quantification of "surprise": assuming \(H_0\) is true, how surprised are we with what we observed? Or in our case, in our hypothesized universe of no difference in genre popularity, how surprised are we that we observed higher popularity rates from our collected samples if no difference in genre popularity exists? Very surprised? Somewhat surprised?

The \(p\)-value quantifies this probability, or what proportion had a more "extreme" result? Here, extreme is defined in terms of the alternative hypothesis \(H_A\) that metal popularity is at a higher rate than deep house. In other words, how often was the popularity of metal even more pronounced than the observed difference in sample proportions \(0.563-0529=0.034=3.4%\text{?}\)

Seventh and lastly, in many hypothesis testing procedures, it is commonly recommended to set the significance level of the test beforehand. It is denoted by \(\alpha\text{.}\) Please review our discussion of \(\alpha\) in Subsection 9.1.1 when we discussed the theory-based approach. For now, it is sufficient to recall that if the \(p\)-value is less than or equal to \(\alpha\text{,}\) we reject the null hypothesis \(H_0\text{.}\)

Alternatively, if the \(p\)-value is greater than \(\alpha\text{,}\) we would "fail to reject \(H_0\text{.}\)" Note the latter statement is not quite the same as saying we "accept \(H_0\text{.}\)" This distinction is rather subtle and not immediately obvious. So we will revisit it later in Section 9.5.

While different fields tend to use different values of \(\alpha\text{,}\) some commonly used values for \(\alpha\) are 0.1, 0.01, and 0.05; with 0.05 being the choice people often make without putting much thought into it. We will talk more about \(\alpha\) significance levels in Section 9.5, but first let’s fully conduct a hypothesis test corresponding to our genre popularity activity using the infer package.