Skip to main content

Section 9.2 Model Selection

The best model is not always the most complicated. Sometimes including variables that are not evidently important can actually reduce the accuracy of predictions. In this section, we discuss model selection strategies, which will help us eliminate variables from the model that are found to be less important. It’s common (and hip, at least in the statistical world) to refer to models that have undergone such variable pruning as parsimonious.
In practice, the model that includes all available explanatory variables is often referred to as the full model. The full model may not be the best model, and if it isn’t, we want to identify a smaller model that is preferable.

Subsection 9.2.1 Identifying variables in the model that may not be helpful

Adjusted \(R^2\) describes the strength of a model fit, and it is a useful tool for evaluating which predictors are adding value to the model, where adding value means they are (likely) improving the accuracy in predicting future outcomes.
Let’s consider two models, which are shown in Figure 9.2.1 and Figure 9.2.2. The first table summarizes the full model since it includes all predictors, while the second does not include the issued variable.
Estimate Std. Error t value Pr(\(>\)|t|)
(Intercept) 1.9251 0.2102 9.16 \(\lt\)0.0001
income_ver (source_only) 0.9750 0.0991 9.83 \(\lt\)0.0001
income_ver (verified) 2.5374 0.1172 21.65 \(\lt\)0.0001
debt_to_income 0.0211 0.0029 7.18 \(\lt\)0.0001
credit_util 4.8959 0.1619 30.24 \(\lt\)0.0001
bankruptcy 0.3864 0.1324 2.92 0.0035
term 0.1537 0.0039 38.96 \(\lt\)0.0001
issued (Jan2018) 0.0276 0.1081 0.26 0.7981
issued (Mar2018) -0.0397 0.1065 -0.37 0.7093
credit_checks 0.2282 0.0182 12.51 \(\lt\)0.0001
\(R_{adj}^2 = 0.25843\) \(df=9990\)
Figure 9.2.1. The fit for the full regression model, including the adjusted \(R^2\text{.}\)
Estimate Std. Error t value Pr(\(>\)|t|)
(Intercept) 1.9213 0.1982 9.69 \(\lt\)0.0001
income_ver (source_only) 0.9740 0.0991 9.83 \(\lt\)0.0001
income_ver (verified) 2.5355 0.1172 21.64 \(\lt\)0.0001
debt_to_income 0.0211 0.0029 7.19 \(\lt\)0.0001
credit_util 4.8958 0.1619 30.25 \(\lt\)0.0001
bankruptcy 0.3869 0.1324 2.92 0.0035
term 0.1537 0.0039 38.97 \(\lt\)0.0001
credit_checks 0.2283 0.0182 12.51 \(\lt\)0.0001
\(R_{adj}^2 = 0.25854\) \(df=9992\)
Figure 9.2.2. The fit for the regression model after dropping the issued variable.

Example 9.2.3. Which of the two models is better?

Which of the two models is better?
Solution.
We compare the adjusted \(R^2\) of each model to determine which to choose. Since the first model has an \(R^2_{adj}\) smaller than the \(R^2_{adj}\) of the second model, we prefer the second model to the first.
Will the model without issued be better than the model with issued? We cannot know for sure, but based on the adjusted \(R^2\text{,}\) this is our best assessment.

Subsection 9.2.2 Two model selection strategies

Two common strategies for adding or removing variables in a multiple regression model are called backward elimination and forward selection. These techniques are often referred to as stepwise model selection strategies, because they add or delete one variable at a time as they “step” through the candidate predictors.
Backward elimination starts with the model that includes all potential predictor variables. Variables are eliminated one-at-a-time from the model until we cannot improve the adjusted \(R^2\text{.}\) The strategy within each elimination step is to eliminate the variable that leads to the largest improvement in adjusted \(R^2\text{.}\)

Example 9.2.4. Backward elimination with the loans data.

Results corresponding to the full model for the loans data are shown in Figure 9.2.1. How should we proceed under the backward elimination strategy?
Solution.
Our baseline adjusted \(R^2\) from the full model is \(R^2_{adj} = 0.25843\text{,}\) and we need to determine whether dropping a predictor will improve the adjusted \(R^2\text{.}\) To check, we fit models that each drop a different predictor, and we record the adjusted \(R^2\text{:}\)
Exclude... income_ver debt_to_income credit_util bankruptcy
\(R^2_{adj} = 0.22380\) \(R^2_{adj} = 0.25468\) \(R^2_{adj} = 0.19063\) \(R^2_{adj} = 0.25787\)
term issued credit_checks
\(R^2_{adj} = 0.14581\) \(R^2_{adj} = 0.25854\) \(R^2_{adj} = 0.24689\)
The model without issued has the highest adjusted \(R^2\) of 0.25854, higher than the adjusted \(R^2\) for the full model. Because eliminating issued leads to a model with a higher adjusted \(R^2\text{,}\) we drop issued from the model.
Since we eliminated a predictor from the model in the first step, we see whether we should eliminate any additional predictors. Our baseline adjusted \(R^2\) is now \(R^2_{adj} = 0.25854\text{.}\) We now fit new models, which consider eliminating each of the remaining predictors in addition to issued:
Exclude issued and... income_ver debt_to_income credit_util
\(R^2_{adj} = 0.22395\) \(R^2_{adj} = 0.25479\) \(R^2_{adj} = 0.19074\)
bankruptcy term credit_checks
\(R^2_{adj} = 0.25798\) \(R^2_{adj} = 0.14592\) \(R^2_{adj} = 0.24701\)
None of these models lead to an improvement in adjusted \(R^2\text{,}\) so we do not eliminate any of the remaining predictors. That is, after backward elimination, we are left with the model that keeps all predictors except issued, which we can summarize using the coefficients from Figure 9.2.2:
\begin{align*} \widehat{\text{rate}} \amp = 1.921 + 0.974 \times \mathbb{1}_{\text{income\_ver = source\_only}} + 2.535 \times \mathbb{1}_{\text{income\_ver = verified}}\\ \amp \qquad + 0.021 \times \text{debt\_to\_income} + 4.896 \times \text{credit\_util} + 0.387 \times \text{bankruptcy}\\ \amp \qquad + 0.154 \times \text{term} + 0.228 \times \text{credit\_checks} \end{align*}
The forward selection strategy is the reverse of the backward elimination technique. Instead of eliminating variables one-at-a-time, we add variables one-at-a-time until we cannot find any variables that improve the model (as measured by adjusted \(R^2\)).

Example 9.2.5. Forward selection with the loans data.

Construct a model for the loans data set using the forward selection strategy.
Solution.
We start with the model that includes no variables. Then we fit each of the possible models with just one variable. That is, we fit the model including just income_ver, then the model including just debt_to_income, then a model with just credit_util, and so on. Then we examine the adjusted \(R^2\) for each of these models:
Add... income_ver debt_to_income credit_util bankruptcy
\(R^2_{adj} = 0.05926\) \(R^2_{adj} = 0.01946\) \(R^2_{adj} = 0.06452\) \(R^2_{adj} = 0.00222\)
term issued credit_checks
\(R^2_{adj} = 0.12855\) \(R^2_{adj} = 0.00018\) \(R^2_{adj} = 0.01711\)
In this first step, we compare the adjusted \(R^2\) against a baseline model that has no predictors. The no-predictors model always has \(R_{adj}^2 = 0\text{.}\) The model with one predictor that has the largest adjusted \(R^2\) is the model with the term predictor, and because this adjusted \(R^2\) is larger than the adjusted \(R^2\) from the model with no predictors (\(R_{adj}^2 = 0\)), we will add this variable to our model.
We repeat the process again, this time considering 2-predictor models where one of the predictors is term and with a new baseline of \(R^2_{adj} = 0.12855\text{:}\)
Add term and... income_ver debt_to_income credit_util
\(R^2_{adj} = 0.16851\) \(R^2_{adj} = 0.14368\) \(R^2_{adj} = 0.20046\)
bankruptcy issued credit_checks
\(R^2_{adj} = 0.13070\) \(R^2_{adj} = 0.12840\) \(R^2_{adj} = 0.14294\)
The best second predictor, credit_util, has a higher adjusted \(R^2\) (0.20046) than the baseline (0.12855), so we also add credit_util to the model.
Since we have again added a variable to the model, we continue and see whether it would be beneficial to add a third variable:
Add term, credit_util, and... income_ver debt_to_income
\(R^2_{adj} = 0.24183\) \(R^2_{adj} = 0.20810\)
bankruptcy issued credit_checks
\(R^2_{adj} = 0.20169\) \(R^2_{adj} = 0.20031\) \(R^2_{adj} = 0.21629\)
The model adding income_ver improved adjusted \(R^2\) (0.24183 from 0.20046), so we add income_ver to the model.
We continue on in this way, next adding debt_to_income, then credit_checks, and bankruptcy. At this point, we come again to the issued variable: adding this variable leads to \(R_{adj}^2 = 0.25843\text{,}\) while keeping all the other variables but excluding issued leads to a higher \(R_{adj}^2 = 0.25854\text{.}\) This means we do not add issued. In this example, we have arrived at the same model that we identified from backward elimination.

Model selection strategies.

Backward elimination begins with the model having the largest number of predictors and eliminates variables one-by-one until we are satisfied that all remaining variables are important to the model. Forward selection starts with no variables included in the model, then it adds in variables according to their importance until no other important variables are found.
Backward elimination and forward selection sometimes arrive at different final models. If trying both techniques and this happens, it’s common to choose the model with the larger \(R_{adj}^2\text{.}\)

Subsection 9.2.3 The p-value approach, an alternative to adjusted \(R^2\)

The p-value may be used as an alternative to \(R_{adj}^2\) for model selection:
Backward elimination with the p-value approach
In backward elimination, we would identify the predictor corresponding to the largest p-value. If the p-value is above the significance level, usually \(\alpha = 0.05\text{,}\) then we would drop that variable, refit the model, and repeat the process. If the largest p-value is less than \(\alpha = 0.05\text{,}\) then we would not eliminate any predictors and the current model would be our best-fitting model.
Forward selection with the p-value approach
In forward selection with p-values, we reverse the process. We begin with a model that has no predictors, then we fit a model for each possible predictor, identifying the model where the corresponding predictor’s p-value is smallest. If that p-value is smaller than \(\alpha = 0.05\text{,}\) we add it to the model and repeat the process, considering whether to add more variables one-at-a-time. When none of the remaining predictors can be added to the model and have a p-value less than 0.05, then we stop adding variables and the current model would be our best-fitting model.

Checkpoint 9.2.6.

Examine Figure 9.2.2, which considers the model including all variables except the variable for the month the loan was issued. If we were using the p-value approach with backward elimination and we were considering this model, which of these variables would be up for elimination? Would we drop that variable, or would we keep it in the model?
Solution.
The bankruptcy predictor is up for elimination since it has the largest p-value. However, since that p-value is smaller than 0.05, we would still keep it in the model.
While the adjusted \(R^2\) and p-value approaches are similar, they sometimes lead to different models, with the \(R_{adj}^2\) approach tending to include more predictors in the final model.

Adjusted \(R^2\) vs p-value approach.

When the sole goal is to improve prediction accuracy, use \(R_{adj}^2\text{.}\) This is commonly the case in machine learning applications.
When we care about understanding which variables are statistically significant predictors of the response, or if there is interest in producing a simpler model at the potential cost of a little prediction accuracy, then the p-value approach is preferred.
Regardless of whether you use \(R_{adj}^2\) or the p-value approach, or if you use the backward elimination or forward selection strategy, our job is not done after variable selection. We must still verify the model conditions are reasonable.

Exercises 9.2.4 Exercises

1. Baby weights, Part IV.

Exercise 9.1.4.3 considers a model that predicts a newborn’s weight using several predictors (gestation length, parity, age of mother, height of mother, weight of mother, smoking status of mother). The table below shows the adjusted R-squared for the full model as well as adjusted R-squared values for all models we evaluate in the first step of the backward elimination process.
Model Adjusted \(R^2\)
1 Full model 0.2541
2 No gestation 0.1031
3 No parity 0.2492
4 No age 0.2547
5 No height 0.2311
6 No weight 0.2536
7 No smoking status 0.2072
Which, if any, variable should be removed from the model first?

2. Absenteeism, Part II.

Exercise 9.1.4.4 considers a model that predicts the number of days absent using three predictors: ethnic background (eth), gender (sex), and learner status (lrn). The table below shows the adjusted R-squared for the model as well as adjusted R-squared values for all models we evaluate in the first step of the backward elimination process.
Model Adjusted \(R^2\)
1 Full model 0.0701
2 No ethnicity -0.0033
3 No sex 0.0676
4 No learner status 0.0723
Which, if any, variable should be removed from the model first?

3. Baby weights, Part V.

Exercise 9.1.4.3 provides regression output for the full model (including all explanatory variables available in the data set) for predicting birth weight of babies. In this exercise we consider a forward-selection algorithm and add variables to the model one-at-a-time. The table below shows the p-value and adjusted \(R^2\) of each model where we include only the corresponding predictor. Based on this table, which variable should be added to the model first?
variable gestation parity age height weight smoke
p-value \(2.2 \times 10^{-16}\) 0.1052 0.2375 \(2.97 \times 10^{-12}\) \(8.2 \times 10^{-8}\) \(2.2 \times 10^{-16}\)
\(R_{adj}^2\) 0.1657 0.0013 0.0003 0.0386 0.0229 0.0569

4. Absenteeism, Part III.

Exercise 9.1.4.4 provides regression output for the full model, including all explanatory variables available in the data set, for predicting the number of days absent from school. In this exercise we consider a forward-selection algorithm and add variables to the model one-at-a-time. The table below shows the p-value and adjusted \(R^2\) of each model where we include only the corresponding predictor. Based on this table, which variable should be added to the model first?
variable ethnicity sex learner status
p-value 0.0007 0.3142 0.5870
\(R_{adj}^2\) 0.0714 0.0001 0

5. Movie lovers, Part I.

Suppose a social scientist is interested in studying what makes audiences love or hate a movie. She collects a random sample of movies (genre, length, cast, director, budget, etc.) as well as a measure of the success of the movie (score on a film review aggregator website). If as part of her research she is interested in finding out which variables are significant predictors of movie success, what type of model selection method should she use?

6. Movie lovers, Part II.

Suppose an online media streaming company is interested in building a movie recommendation system. The website maintains data on the movies in their database (genre, length, cast, director, budget, etc.) and additionally collects data from their subscribers (demographic information, previously watched movies, how they rated previously watched movies, etc.). The recommendation system will be deemed successful if subscribers actually watch, and rate highly, the movies recommended to them. Should the company use the adjusted \(R^2\) or the p-value approach in selecting variables for their recommendation system?
PrevTopNext