Section 14.10 Computation and Analysis
This book focuses on computational methods like resampling and permutation. These methods have several advantages over analysis:
-
They are easier to explain and understand. For example, one of the most difficult topics in an introductory statistics class is hypothesis testing. Many students don’t really understand what p-values are. I think the approach we took in Chapter 9—simulating the null hypothesis and computing test statistics—makes the fundamental idea clearer.
-
They are robust and versatile. Analytic methods are often based on assumptions that don’t hold in practice. Computational methods require fewer assumptions, and can be adapted and extended more easily.
-
They are debuggable. Analytic methods are often like a black box: you plug in numbers and they spit out results. But it’s easy to make subtle errors, hard to be confident that the results are right, and hard to diagnose the problem if they are not. Computational methods lend themselves to incremental development and testing, which fosters confidence in the results.
But there are drawbacks:
-
Computational methods can be slow.
-
Randomized methods like resampling don’t produce the same results every time, which makes it harder to check that they are correct.
Taking into account these pros and cons, I recommend the following process:
-
Use computational methods during exploration. If you find a satisfactory answer and the run time is acceptable, you can stop.
-
If run time is not acceptable, look for opportunities to optimize. Using analytic methods is one of several methods of optimization.
-
If replacing a computational method with an analytic method is appropriate, use the computational method as a basis of comparison, providing mutual validation between the computational and analytic results.
For many practical problems, the run time of computational methods is not a problem, and we don’t have to go past the first step.
