Section B.3 Residuals Revisited: R Squared
In the output above we saw there was something called the residuals. The residuals are the differences between the observed values of Y for each case minus the predicted or expected value of Y; in other words, the distances between each point in the dataset and the regression line (see the visual example below).
You see that we have our line, which is our predicted values, and then we have the black dots which are our actually observed values. The distance between them is essentially the amount by which we were wrong, and all these distances between observed and predicted values are our residuals. Least square estimation essentially aims to reduce the average of the squares of all these distances: that’s how it draws the line.
Why do we have residuals? Well, think about it. The fact that the line is not a perfect representation of the cloud of points makes sense, doesn’t it? You cannot predict perfectly what the value of Y is for every observation just by looking ONLY at their level of resource deprivation! This line only uses information regarding resource deprivation. This means that there’s bound to be some difference between our predicted level of homicide given our knowledge of deprivation (regression line) and the actual level of homicide (actual location of the points in the scatterplot). There are other things that matter that are not being taken into account by our model to predict the values of Y. There are other things that surely matter in terms of understanding homicide. And then, of course, we have measurement error and other forms of noise.
We can re-write our equation like this if we want to represent each value of Y (rather than the predicted value of Y) then: \(y = \beta_0 + \beta_1 x + \text{residuals}\text{.}\)
The residuals capture how much variation is unexplained, how much we still have to learn if we want to understand variation in Y. A good model tries to maximise explained variation and reduce the magnitude of the residuals.
We can use information from the residuals to produce a measure of effect size, of how good our model is in predicting variation in our dependent variables. Remember our game where we try to guess homicide (Y)? If we did not have any information about X, our best bet for Y would be the mean of Y. The regression line aims to improve that prediction. By knowing the values of X, we can build a regression line that that aims to get us closer to the actual values of Y (look at the Figure below).
The distance between the mean (our best guess without any other piece of information) and the observed value of Y is what we call the total variation. The residual is the difference between our predicted value of Y and the observed value of Y. This is what we cannot explain (i.e., variation in Y that is unexplained). The difference between the mean value of Y and the expected value of Y (value given by our regression line) is how much better we are doing with our prediction by using information about X, how much closer the regression line gets us to the observed values. We can then contrast these two different sources of variation (explained and unexplained) to produce a single measure of how good our model is. The formula is as follows:
\begin{equation*}
R^2 = \dfrac{SSR}{SST} = \dfrac{\Sigma(\hat{y}_i - \bar{y})^2}{\Sigma(y_i - \bar{y})^2}
\end{equation*}
All this formula is doing is taking a ratio of the explained variation (squared differences between the regression line and the mean of Y for each observation) by the total variation (squared differences of the observed values of Y for each observation from the mean of Y). This gives us a measure of the percentage of variation in Y that is "explained" by X. If this sounds familiar, it is because it is a measure similar to eta squared (\(\eta^2\)) in ANOVA.
As then we can take this value as a measure of the strength of our model. If you look at the R output, you will see that the \(R^2\) for our model was .32. We can say that our model explains 32% of the variance in the homicide rate. When doing regression, you will often find that regression models with aggregate data such as county level data will give you better results than when dealing with individuals. It is much harder understanding individual variation than county level variation.
As an aside, and to continue emphasising your appreciation of the object oriented nature of R, when we run the
summary() function, we are simply generating a list object of the class summary.lm.
attributes(summary(fit_1))
## $names ## [1] "call" "terms" "residuals" ## [4] "coefficients" "aliased" "sigma" ## [7] "df" "r.squared" "adj.r.squared" ## [10] "fstatistic" "cov.unscaled" ## ## $class ## [1] "summary.lm"
This means that we can access its elements if so we wish. So, for example, to obtain just the R Squared (\(R^2\)), we could ask for:
summary(fit_1)$r.squared
## [1] 0.3224
Knowing how to interpret this is important. \(R^2\) ranges from 0 to 1. The greater it is, the more powerful our model is and the more explaining we are doing: the better we are able to account for variation in our outcome Y with our input. In other words, the stronger the relationship is between Y and X. As with all the other measures of effect size interpretation is a matter of judgment. You are advised to see what other researchers report in relation to the particular outcome that you may be exploring.
[273] suggest that in criminal justice you rarely see values for \(R^2\) greater than .40. Thus, if your \(R^2\) is larger than .40, you can assume you have a powerful model. When, on the other hand, \(R^2\) is lower than .15 or .2 the model is likely to be viewed as relatively weak. Our observed \(R^2\) here is rather poor. There is considerable room for improvement if we want to develop a better model to explain fear of violent crime. In any case, many people would argue that \(R^2\) is a bit overrated. You need to be aware of what it measures and the context in which you are using it.
