Minimizing MSE

Section 10.3 Minimizing MSE

Earlier I said that the least squares fit is the straight line that minimizes the mean squared error (MSE). We won’t prove that, but we can test it by adding small random values to the intercept and slope, and checking whether the MSE gets worse.

🔗

Listing 10.3.1. Python Code

🔗

intercept = result.intercept + np.random.normal(0, 1)
slope = result.slope + np.random.normal(0, 1)

To run the test, we need to make an object with intercept and slope attributes -- we’ll use the SimpleNamespace object provided by the types module.

🔗

Listing 10.3.2. Python Code

🔗

$ from types import SimpleNamespace

fake_result = SimpleNamespace(intercept=intercept, slope=slope)
fake_result
namespace(intercept=np.float64(-2533.193389187175),
          slope=np.float64(32.15393529728294))

We can pass this object to compute_residuals and use the residuals to compute the MSE.

🔗

Listing 10.3.3. Python Code

🔗

fake_residuals = compute_residuals(fake_result, flipper_length, body_mass)
fake_mse = np.mean(fake_residuals**2)

If we compare the result to the MSE of the least squares line, it is always worse.

🔗

Listing 10.3.4. Python Code

🔗

$ mse, fake_mse, fake_mse > mse
(np.float64(163098.85902884745), np.float64(179019.20812685465), np.True_)

Minimizing MSE is nice, but it’s not the only definition of "best". One alternative is to minimize the absolute values of the errors. Another is to minimize the shortest distance from each point to the fitted line, which is called the "total error". In some contexts, guessing too high might be better (or worse) than guessing too low. In that case you might want to compute a cost function for each residual, and minimize total cost.

🔗

But the least squares fit is much more widely used than these alternatives, primarily because it is efficient to compute. The following function shows how.

🔗

Listing 10.3.5. Python Code

🔗

def least_squares(xs, ys):
    xbar = np.mean(xs)
    ybar = np.mean(ys)

    xdev = xs - xbar
    ydev = ys - ybar

    slope = np.sum(xdev * ydev) / np.sum(xdev**2)
    intercept = ybar - slope * xbar

    return intercept, slope

To test this function, we’ll use flipper length and body mass again.

🔗

Listing 10.3.6. Python Code

🔗

$ intercept, slope = least_squares(flipper_length, body_mass)
intercept, slope
(np.float64(-2535.8368022002524), np.float64(32.831689751150094))

And we can confirm that we get the same results we got from linregress.

🔗

Listing 10.3.7. Python Code

🔗

$ np.allclose([intercept, slope], [result.intercept, result.slope])
True

Minimizing MSE made sense when computational efficiency was more important than choosing the method most appropriate to the problem at hand. But that’s no longer the case, so it is worth considering whether squared residuals are the right thing to minimize.

🔗

Prev Top Next