Chapter 14: Linear Regression

Section A.13 Chapter 14: Linear Regression

Subsection A.13.1 Exercise 14.1

The line marked in *red* is increasing and at $x=0$ it seems to obtain the value $y=0\text{.}$ An increasing line is associated with a linear equation with a positive slope coefficient ($b > 0$). The only equation with that property is Equation 3, for which $b=1\text{.}$ Notice that the intercept of this equation is $a = 0\text{,}$ which agrees with the fact that the line passes through the origin $(x,y) = (0,0)\text{.}$ If the $x$-axis and the $y$-axis were on the same scale then one would expect the line to be tilted in the 45 degrees angle. However, here the axes are not on the same scale, so the tilting is different.
🔗

🔗
The $x$-value of the line marked with a *red triangle* is $x=-1\text{.}$ The $y$-value is below 5. The observation that has an $x$-value of -1 is Observation 6. The $y$-value of this observation is $y=4.3\text{.}$ Notice that there is another observation with the same $y$-value, Observation 3. However, the $x$-value of that observation is $x=1.6\text{.}$ Hence it is the point that is on the same level as the marked point, but it is placed to the right of it.
🔗

🔗

Subsection A.13.2 Exercise 14.2

The intercept is equal to $a = -3.60$ and the slope is equal to $b = 2.13\text{.}$ Notice that the lope is the coefficient that multiplies the explanatory variable and the intercept is the coefficient that does not multiply the explanatory variable.
🔗

🔗
The value of the explanatory variable for the 3rd observation is $x_3 = 4.2\text{.}$ When we use this value in the regression formula we obtain that:
🔗

\begin{equation*} \Expec(Y_3) = 2.13 \cdot x_3 - 3.60 = 2.13 \cdot 4.2 - 3.60 = 5.346\;. \end{equation*}

In words, the expectation of the response of the 3rd observation is equal to $5.346$

🔗

🔗

Subsection A.13.3 Exercise 14.3

After saving the file “aids.csv" in the working directory of R we read it’s content into a data frame by the name “aids". We then produce a summary of the fit of the linear regression model of “deaths" as a response and “diagnosed" as the explanatory variable:

🔗

aids <- read.csv("_data/aids.csv")
fit.deaths <- lm(deaths~diagnosed,data=aids)
summary(fit.deaths)

## 
## Call:
## lm(formula = deaths ~ diagnosed, data = aids)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -7988.7  -680.9    23.9  1731.3  7760.7 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   88.7161  1370.7191   0.065    0.949    
## diagnosed      0.6073     0.0312  19.468 1.81e-14 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3589 on 20 degrees of freedom
## Multiple R-squared:  0.9499, Adjusted R-squared:  0.9474 
## F-statistic:   379 on 1 and 20 DF,  p-value: 1.805e-14

The estimated value of the slope 0.6073. The computed $p$-value associated with this slope is $1.81 \times 10^{-14}\text{,}$ which is much smaller than the 5% threshold. Consequently, the slope is statistically significant.

🔗

For confidence intervals apply the function “confint" to the fitted model:

🔗

confint(fit.deaths)

##                     2.5 %      97.5 %
## (Intercept) -2770.5538947 2947.986092
## diagnosed       0.5422759    0.672427

We get that the confidence interval for the slope is $[0.5422759 , 0.672427]\text{.}$

🔗

A scatter plot of the two variables is produced by the application of the function “plot" to the formula that involves these two variables. The regression line is added to the plot using the function “abline" with the fitted model as an input:
🔗
```
plot(deaths~diagnosed,data=aids)
abline(fit.deaths)
```
Observe that the points are nicely placed very to a line that characterizes the linear trend of the regression.
🔗

🔗

We fit a linear regression model of “diagnosed" as a response and “year" as the explanatory variable and save the fit in the object “fit.diagnosed". A summary of the model is produced by the application of the function “summary" to the fit:

🔗

fit.diagnosed <- lm(diagnosed~year,data=aids)
summary(fit.diagnosed)

## 
## Call:
## lm(formula = diagnosed ~ year, data = aids)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -28364 -18321  -3909  14964  41199 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)  
## (Intercept) -3448225.0  1535037.3  -2.246   0.0361 *
## year            1749.8      770.8   2.270   0.0344 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 22940 on 20 degrees of freedom
## Multiple R-squared:  0.2049, Adjusted R-squared:  0.1651 
## F-statistic: 5.153 on 1 and 20 DF,  p-value: 0.03441

The estimated value of the slope 1749.8. The computed $p$-value associated with this slope is $0.0344\text{,}$ which is less than the 0.05. Consequently, one may declare the slope to be statistically significant. Confidence intervals are produced using the function “confint":

🔗

confint(fit.diagnosed)

##                    2.5 %      97.5 %
## (Intercept) -6650256.665 -246193.429
## year             141.936    3357.618

We get that the 95% confidence interval for the slope is $[141.9360, 3357.618]\text{.}$

🔗

A scatter plot of the two variables is produced by the application of the function “plot" and the regression line is added with function “abline":
🔗
```
plot(diagnosed~year,data=aids)
abline(fit.diagnosed)
```
Observe that the points do not follow a linear trend. It seems that the number of diagnosed cases increased in an exponential rate during the first years after Aids was discovered. The trend changed in the mid 90’s with a big drop in the number of diagnosed Aids patients. This drop may be associated with the administration of therapies such as AZT to HIV infected subjects that reduced the number of such subjects that developed Aids. In the late 90’s there seams to be yet again a change in the trend and the possible increase in numbers. The line of linear regression misses all these changes and is a poor representation of the historical development.
🔗

The take home message from this exercise is to not use models blindly. A good advise is to plot the data. An examination of the plot provides a warning that the linear model is probably not a good model for the given problem.
🔗

🔗

🔗

Subsection A.13.4 Exercise 14.4

We read the data in the table into R. The variable “year" is the explanatory variable and the variable “percent" is the response. The scatter plot is produced using the function “plot" and the regression line, fitted to the data with the function “lm", is added to the plot using the function “abline":
🔗
```
year <- c(1945,1950,1960,1970,1980,1986,1993)
percent <- c(35.5,31.5,31.4,27.3,21.9,17.5,15.8)
plot(percent~year)
abline(lm(percent~year))
```
Observe that a linear trend is a reasonable description of the reduction in the percentage of workers that belong to labor unions in the post World War II period.
🔗

🔗
We compute the averages, standard deviations and the covariance:
🔗
```
mean.x <- mean(year)
mean.y <- mean(percent)
sd.x <- sd(year)
sd.y <- sd(percent)
cov.x.y <- cov(year,percent)
mean.x
mean.y
sd.x
sd.y
cov.x.y
```
```
## [1] 1969.143
## [1] 25.84286
## [1] 18.27957
## [1] 7.574927
## [1] -135.6738
```
The average of the variable “year" is $1969.143$ and the standard deviation is $18.27957\text{.}$ The average of the variable “percent" is $25.84286$ and the standard deviation is $7.574927\text{.}$ The covariance between the two variables is $-135.6738\text{.}$
🔗

🔗
The slope of the regression line is the ratio between the covariance and the variance of the explanatory variable. The intercept is the solution of the equation that states that the value of regression line at the average of the explanatory variable is the average of the response:
🔗
```
b <- cov.x.y/sd.x^2
a <- mean.y - b*mean.x
a
b
```
```
## [1] 825.3845
## [1] -0.4060353
```
We get that the intercept is equal to $825.3845$ and the slope is equal to $-0.4060353\text{.}$
🔗

In order to validate these figures, let us apply the function “lm" to the data:
🔗
```
lm(percent~year)
```
```
## 
## Call:
## lm(formula = percent ~ year)
## 
## Coefficients:
## (Intercept)         year  
##     825.384       -0.406
```
Indeed, we get the same numbers that we got from the manual computation.
🔗

🔗

🔗

Subsection A.13.5 Exercise 14.5

The estimate for the expected value for the $i$th observation is obtained by the evaluation of the expression $a + b\cdot x_i\text{,}$ where $a$ and $b$ are the coefficients of the fitted regression model and $x_i$ is the value of the explanatory variable for the $i$th observation. In our case $i=4$ and $x_4 = 6.7\text{:}$
🔗

\begin{equation*} a + b\cdot x_4 = 2.5-1.13\cdot x_4 = 2.5-1.13\cdot 6.7 = -5.071\;. \end{equation*}

Therefore, the estimate expectation of the response is $-5.071\text{.}$

🔗

🔗
The residual from the regression line is the difference between the observed value of the response and the estimated expectation of the response. For the 4th observation we have that the observed value of the response is $y_4 = 0.17\text{.}$ The estimated expectation was computed in the previous question. Therefore, the residual from the regression line for the 4th observation is:
🔗

\begin{equation*} y_4 - (a + b\cdot x_4) = 0.17 - (-5.071) = 5.241\;. \end{equation*}

🔗

🔗

🔗

Subsection A.13.6 Exercise 14.6

We create the response and then fit a model and apply the summarizing function to the model:
🔗
```
dif.mpg <- cars$highway.mpg - cars$city.mpg
summary(lm(dif.mpg ~ cars$curb.weight))
```
```
## 
## Call:
## lm(formula = dif.mpg ~ cars$curb.weight)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -5.4344 -0.7755  0.1633  0.8844  6.3035 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       8.1653491  0.5460856  14.953  < 2e-16 ***
## cars$curb.weight -0.0010306  0.0002094  -4.921 1.77e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.557 on 203 degrees of freedom
## Multiple R-squared:  0.1066, Adjusted R-squared:  0.1022 
## F-statistic: 24.22 on 1 and 203 DF,  p-value: 1.775e-06
```
The $p$-value associated with the slope, $1.77\times 10^{-6}\text{,}$ is much smaller than the 5% threshold proposing a significant (negative) trend. The value of R-squared, the fraction of the variability of the response that is explained by a regression model, is $0.1066\text{.}$
🔗

The standard deviation is the square root of the variance. It follows that the fraction of the standard deviation of the response that is explained by the regression is $\sqrt{0.1066}= 0.3265\text{.}$
🔗

Following our own advice, we plot the data and the regression model:
🔗
```
plot(dif.mpg ~ cars$curb.weight)
abline(lm(dif.mpg ~ cars$curb.weight))
```
One may observe that although there seems to be an overall downward trend, there is still allot of variability about the line of regression.
🔗

🔗

We now fit and summarize the regression model with the size of engine as the explanatory variable:

🔗

summary(lm(dif.mpg ~ cars$engine.size))

## 
## Call:
## lm(formula = dif.mpg ~ cars$engine.size)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -5.7083 -0.7083  0.1889  1.1235  6.1792 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       6.717304   0.359385  18.691  < 2e-16 ***
## cars$engine.size -0.009342   0.002691  -3.471 0.000633 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.601 on 203 degrees of freedom
## Multiple R-squared:  0.05603,    Adjusted R-squared:  0.05138 
## F-statistic: 12.05 on 1 and 203 DF,  p-value: 0.0006329

The regression slope is negative. The $p$-value is $0.000633\text{,}$ which is statistically significant. The value of R-squared is $0.05603\text{.}$ Consequently, the fraction of the standard deviation of the response that is explained by the current regression model is $\sqrt{0.05603}= 0.2367\text{.}$

🔗

Produce the scatter plot with the line of regression:

🔗

plot(dif.mpg ~ cars$engine.size)
abline(lm(dif.mpg ~ cars$engine.size))

Again, there is variability about the line of regression.

🔗

Of the two models, the model that uses the curb weigh as the explanatory variable explains a larger portion of the variability in the response. Hence, unless other criteria tells us otherwise, we will prefer this model over the model that uses the size of engine as an explanatory variable.
🔗

🔗

🔗

Prev Top Next