Statistical Inference via Data Science A ModernDive into R and the Tidyverse (Second Edition)

The bootstrap is one of many resampling methods. Chernick and LaBudde [13] noted that the bootstrap’s roots trace back to the development of similar techniques like permutations and the jackknife. The bootstrap was initially conceived as an approximation to another resampling method called the jackknife but quickly gained recognition for its broader applicability and efficiency. Since then, it has been shown in multiple contexts that the bootstrap performs at least as well as traditional methods in estimating standard errors, constructing confidence intervals, performing hypothesis testing, and many other statistical techniques.

Furthermore, since the 1980s (but even more in the last two decades), the use of simulations to compare advanced bootstrap methods against other techniques, such as cross-validation, further established its superiority in specific contexts, particularly when dealing with small sample sizes. The use of the bootstrap and bootstrap-related methods has become a cornerstone in modern statistical and data science practices.

In this section, we introduce additional details about the bootstrap, and explain the advantages and limitations of using the bootstrap.

We want to compare how the bootstrap performs when building confidence intervals with respect to the theory-based approach discussed in Section 8.1. The formal comparison requires mathematical concepts beyond the scope of this book, and it is not pursued here. Instead, we provide just enough elements to help you with the intuition of how this comparison is made and why bootstrap-related methods can be as strong as or even stronger than theory-based or alternative methods.

Given that we can never make \(n\) infinity in real-life applications, we would like to produce 95% confidence intervals that make the difference between the approximate coverage probability and the true coverage probability as small as possible when we increase the sample size of our sample. Imagine a sequence of sample sizes \(n_1, n_2, n_3, \dots\) that gets bigger and bigger and bigger. The rate at which the corresponding consecutive differences in confidence intervals’ coverage probability decreases is called the rate of convergence of the difference between approximate and true coverage probabilities.

In the case of a 95% confidence interval for \(\mu\) using the theory-based approach, the rate of convergence for the difference is about \(1/\sqrt{n}\text{.}\) This means that if we increase \(n\) from 100 to 400, the difference between the approximate and true coverage goes down from a factor of \(1/\sqrt{100} = 0.1\) to a factor of \(1/\sqrt{400} = 0.05\text{.}\) Thus, increasing \(n\) four times leads to a decrease of the difference of about two times. In the statistical literature, a method that has this rate of convergence is called first-order correct [12] or first-order accurate [13].

The bootstrap percentile method discussed in Subsection 8.2.5, in the case of a 95% confidence interval for \(\mu\text{,}\) is also first-order accurate. Thus, confidence intervals calculated using the theory-based and the bootstrap percentile method are comparable. This is consistent with the results we obtained earlier in this chapter.

In general, if you have two different methods that produce similar 95% confidence intervals and we need to choose one of them, we would choose the one that has a faster rate of convergence. As we discuss in the next subsection, other bootstrap methods have, in certain contexts, faster rates of convergence.

Why is it suitable to learn and use bootstrap methods for confidence intervals? The most important reason is that bootstrap methods, in particular advanced bootstrap methods, can deal with many limitations of the theory-based approach. Let’s discuss three of these limitations.

First, a 95% confidence interval for \(\mu\) is appropriate if the population distribution is not too extreme and the sample size is large enough for the distribution of \(\overline{X}\) to be approximately normal. But there are situations where these conditions are not satisfied, for example, if the population distribution is heavily skewed to the right, as in the case of income or wealth; or the distribution is constructed from only two values (1 or 0) but the chances of getting zero are much greater than the ones of getting one (for example, chance of getting zero is 0.999 and chance of getting 1 is 0.001) as it is the case in lottery outcomes or the presence of some disease in a population. When these situations are present, the confidence intervals would be inaccurate because the sample mean \(\overline{X}\) is biased when \(n\) is not too large. This means that if you were to take a large number of random samples and construct the sample mean of these samples, their average will be clearly different than \(\mu\text{,}\) breaking down the theory we developed in Section 8.1. This problem may be fixed if the sample size is large, but, depending on how extreme the population distribution is, the sample size may need to be extremely large; perhaps in the order of thousands, or tens of thousands, or even more. Getting samples of those sizes may not be doable in real-life situations.

Second, in this chapter, we have studied confidence intervals for the population mean \(\mu\text{,}\) because it is a fundamental quantity and it is the foundation for other cases. However, building confidence intervals for other parameters using the theory-based approach (for example, for the median, the first quartile, the standard deviation, etc.) becomes more complicated or even unfeasible.

Third, when working with estimators more complicated than \(\overline{X}\text{,}\) it is often not possible to derive the standard error estimator with a formula as clean as \(\sigma/\sqrt{n}\text{.}\) Sometimes, there is no formula for the standard error and alternative methods have to be used to estimate it. This can create an additional source of bias. When bias is present, the confidence intervals created using the theory-based approach in Subsection 8.1.4 could be suspect, even completely useless.

The bootstrap percentile method is not affected directly by the second and third limitations. It can be implemented for any other parameter beyond the population mean, as long as all the information needed can be extracted from each bootstrap sample. On the other hand, the first limitation listed above can also affect the accuracy of this method.

Fortunately, since the inception of the bootstrap, many improvements have been made to the percentile method. Bootstrap methods have been proposed that address the presence of bias either by the limitations discussed above or the bias created when obtaining estimators or incorporating these methods. In addition, in certain contexts, these methods also improve the rate of convergence of the difference between the approximate and true coverage probability. Some of these methods are the percentile-\(t\) and the Bias Correction and Acceleration bootstrap method (BCa). In terms of rates of convergence, these methods are second-order accurate; that is, they have a rate of convergence of about \(1/n\text{.}\) Another method called the double bootstrap (or more generally, the iterated bootstrap) can even be a third-order accurate.

We have not included these methods directly here because the theory that justifies them goes beyond the scope of this book and, when dealing with confidence intervals for \(\mu\) from populations with distributions that are not extreme, there are not real gains in using them over the theory-based approach or the percentile method. (We encourage you to check out one implementation of bias-corrected confidence intervals in the infer package by setting type = "bias-corrected" in the get_confidence_interval() function.)

To summarize, when working with skewed distributions, small sample sizes, estimators of parameters other than \(\mu\) (such as the median), or the estimation of the standard error when there are no formulas to obtain them, many advanced bootstrap methods would be preferred over the theory-based approach. In the Appendices of the online version of the book, we plan to explore some of these advanced methods and present simulations that show when these methods are preferred over the percentile method or the theory-based approach.