Section 3.6 Chapter 3 Appendix: Multiple Regression
1
This section is adapted from Rudy Guerra and David M. Lane. "Introduction to Multiple Regression." Online Statistics Education: A Multimedia Course of Study. https://onlinestatbook.com/2/regression/multiple_regression.html
In simple linear regression, a dependent variable is predicted from one independent variable. In multiple regression, the dependent variable is predicted by two or more variables. For example, in the SAT case study, you might want to predict a student’s university grade point average on the basis of their High-School GPA (HSGPA) and their total SAT score (verbal + math). The basic idea is to find a linear combination of HSGPA and SAT that best predicts University GPA (UGPA). That is, the problem is to find the values of \(\beta_1\) and \(\beta_2\) in the equation shown below that give the best predictions of UGPA. As in the case of simple linear regression, we define the best predictions as the predictions that minimize the squared errors of prediction.
2
A linear combination of variables is a way of creating a new variable by combining other variables. A linear combination is one in which each variable is multiplied by a coefficient and the products are summed. For example, if \(Y = 3X_1 + 2X_2 + .5X_3\) then Y is a linear combination of \(X_1\text{,}\) \(X_2\text{,}\) and \(X_3\text{.}\)
\begin{equation*}
\widehat{UGPA} = \alpha + \beta_1 HSGPA + \beta_2 SAT
\end{equation*}
where \(\widehat{UGPA}\) is the predicted value of University GPA and \(\alpha\) is a constant. For these data, the best prediction equation is shown below:
\begin{equation*}
\widehat{UGPA} = 0.540 + 0.541 \times HSGPA + 0.008 \times SAT
\end{equation*}
In other words, to compute the prediction of a student’s University GPA, you add up (a) 0.540, (b) their High-School GPA multiplied by 0.541, and (c) their SAT multiplied by 0.008. Table 3.6.1 shows the data and predictions for the first five students in the dataset.
| \(HSGPA\) | \(SAT\) | \(\widehat{UGPA}\) |
|---|---|---|
| 3.45 | 1232 | 3.38 |
| 2.78 | 1070 | 2.89 |
| 2.52 | 1086 | 2.76 |
| 3.67 | 1287 | 3.55 |
| 3.24 | 1130 | 3.19 |
The multiple correlation (R) is equal to the correlation between the predicted scores and the actual scores. In this example, it is the correlation between \(\widehat{UGPA}\) and \(UGPA\text{,}\) which turns out to be 0.79. That is, \(R = 0.79\text{.}\) Note that R will never be negative since if there are negative correlations between the predictor variables and the criterion, the regression coefficients will be negative so that the correlation between the predicted and actual scores will be positive.
Subsection 3.6.1 Interpretation of Regression Coefficients
A regression coefficient in multiple regression is the slope of the linear relationship between the criterion variable and the part of a predictor variable that is independent of all other predictor variables. In this example, the regression coefficient for HSGPA can be computed by first predicting HSGPA from SAT and saving the errors of prediction (the differences between \(HSGPA\) and \(\widehat{HSGPA}\)). These errors of prediction are called “residuals” since they are what is left over in HSGPA after the predictions from SAT are subtracted, and represent the part of HSGPA that is independent of SAT. These residuals are referred to as HSGPA.SAT, which means they are the residuals in HSGPA after having been predicted by SAT. The correlation between HSGPA.SAT and SAT is necessarily 0.
The final step in computing the regression coefficient is to find the slope of the relationship between these residuals and UGPA. This slope is the regression coefficient for HSGPA. The following equation is used to predict HSGPA from SAT:
\begin{equation*}
\widehat{HSGPA} = -1.314 + 0.0036 \times SAT
\end{equation*}
The residuals are then computed as:
\begin{equation*}
HSGPA.SAT = HSGPA - \widehat{HSGPA}
\end{equation*}
The linear regression equation for the prediction of UGPA by the residuals is
\begin{equation*}
\widehat{UGPA} = 3.173 + 0.541 \times HSGPA.SAT
\end{equation*}
Notice that the slope (0.541) is the same value given previously for the estimate of \(\beta_1\) in the multiple regression equation.
This means that the regression coefficient for HSGPA is the slope of the relationship between the dependent variable and the part of HSGPA that is independent of (uncorrelated with) the other independent variables. It represents the change in the dependent variable associated with a change of one in the independent variable when all other independent variables are held constant. Since the regression coefficient for HSGPA is 0.54, this means that, holding SAT constant, a change of one in HSGPA is associated with a change of 0.54 in \(\widehat{UGPA}\text{.}\) If two students had the same SAT and differed in HSGPA by 2, then you would predict they would differ in UGPA by \((2)(0.54) = 1.08\text{.}\) Similarly, if they differed by 0.5, then you would predict they would differ by \((0.50)(0.54) = 0.27\text{.}\)
The slope of the relationship between the dependent variable and the part of an independent variable that is unique from (independent of) other independent variables is its partial slope. Thus, the regression coefficient of 0.541 for HSGPA and the regression coefficient of 0.008 for SAT are partial slopes. Each partial slope represents the relationship between the independent variable and the dependent variable holding constant all of the other independent variables.
It is difficult to compare the coefficients for different variables directly because they are measured on different scales. A difference of 1 in HSGPA is a fairly large difference, whereas a difference of 1 on the SAT is negligible. Therefore, it can be advantageous to transform the variables so that they are on the same scale. The most straightforward approach is to standardize the variables (see Section 2.4.1) so that they each have a standard deviation of 1. A regression coefficient for standardized variables is called a “standardized coefficient” or “beta coefficient.” For these data, the standardized coefficients are 0.625 and 0.198. These values represent the change in the dependent variable (in standard deviations) associated with a change of one standard deviation on an independent variable (holding constant the value(s) on the other independent variable(s)). Clearly, a change of one standard deviation on HSGPA is associated with a larger difference than a change of one standard deviation of SAT. In practical terms, this means that if you know a student’s HSGPA, knowing the student’s SAT does not aid the prediction of UGPA much. However, if you do not know the student’s HSGPA, his or her SAT can aid in the prediction since the standardized coefficient in the simple regression predicting UGPA from SAT is 0.68. For comparison purposes, the standardized coefficient in the simple regression predicting UGPA from HSGPA is 0.78. As is typically the case, the partial slopes are smaller than the slopes in simple regression.
