Applications: Model and infer

Chapter 27 Applications: Model and infer

Section 27.1 Case study: Mario Kart

In this case study, we consider Ebay auctions of a video game called Mario Kart for the Nintendo Wii. The outcome variable of interest is the total price of an auction, which is the highest bid plus the shipping cost. We will try to determine how total price is related to each characteristic in an auction while simultaneously controlling for other variables. For instance, all other characteristics held constant, are longer auctions associated with higher or lower prices? And, on average, how much more do buyers tend to pay for additional Wii wheels (plastic steering wheels that attach to the Wii controller) in auctions? Multiple regression will help us answer these and other questions.

🔗

Note 27.1. Data.

The mariokart data can be found in the openintro R package.

🔗

The mariokart dataset includes results from 141 auctions. Four observations from this dataset are shown in Table 27.2, and descriptions for each variable are shown in Table 27.3. Notice that the condition and stock photo variables are indicator variables, similar to bankruptcy in the loans dataset from Chapter 25.

🔗

Table 27.2. Top four rows of the mariokart dataset.

🔗

price	cond_new	stock_photo	duration	wheels
51.55	new	yes	3	1
37.04	used	yes	7	1
45.50	new	no	3	1
44.00	new	yes	3	1

Table 27.3. Variables and their descriptions for the mariokart dataset.

🔗

Variable	Description
`price`	Final auction price plus shipping costs, in US dollars.
`cond_new`	Indicator variable for if the game is new (1) or used (0).
`stock_photo`	Indicator variable for if the auction’s main photo is a stock photo.
`duration`	The length of the auction, in days, taking values from 1 to 10.
`wheels`	The number of Wii wheels included with the auction. A Wii wheel is an optional steering wheel accessory that holds the Wii controller.

Subsection 27.1.1 Mathematical approach to linear models

In Table 27.4 we fit a mathematical linear regression model with the game’s condition as a predictor of auction price.

🔗

\begin{equation*} E[\texttt{price}] = \beta_0 + \beta_1\times \texttt{cond_new} \end{equation*}

Results of the model are summarized below:

🔗

Table 27.4. Summary of a linear model for predicting price based on cond_new.

🔗

term	estimate	std.error	statistic	p.value
(Intercept)	42.9	0.81	52.67	<0.0001
cond_new	10.9	1.26	8.66	<0.0001

Checkpoint 27.5. Guided Practice.

Write down the equation for the model, note whether the slope is statistically different from zero, and interpret the coefficient.

🔗

Solution.

The equation for the line may be written as $\widehat{\texttt{price}} = 47.15 + 10.90\times \texttt{cond_new}\text{.}$ Examining the regression output in Table 27.4 we can see that the p-value for cond_new is very close to zero, indicating there is strong evidence that the coefficient is different from zero when using this one-variable model. The variable cond_new is a two-level categorical variable that takes value 1 when the game is new and value 0 when the game is used. This means the 10.90 model coefficient predicts a price of an extra $10.90 for those games that are new versus those that are used.

🔗

Sometimes there are underlying structures or relationships between predictor variables. For instance, new games sold on Ebay tend to come with more Wii wheels, which may have led to higher prices for those auctions. We would like to fit a model that includes all potentially important variables simultaneously, which would help us evaluate the relationship between a predictor variable and the outcome while controlling for the potential influence of other variables.

🔗

We want to construct a model that accounts for not only the game condition but simultaneously accounts for three other variables:

🔗

\begin{align*} E[\texttt{price}] = \beta_0 \amp+ \beta_1 \times \texttt{cond_new} + \beta_2\times \texttt{stock_photo}\\ \amp+ \beta_3 \times \texttt{duration} + \beta_4 \times \texttt{wheels} \end{align*}

Table 27.6 summarizes the full model. Using the output, we identify the point estimates of each coefficient and the corresponding impact (measured with information on the standard error used to compute the p-value).

🔗

Table 27.6. Summary of a linear model for predicting price based on cond_new, stock_photo, duration, and wheels.

🔗

term	estimate	std.error	statistic	p.value
(Intercept)	36.21	1.51	23.92	<0.0001
cond_new	5.13	1.05	4.88	<0.0001
stock_photo	1.08	1.06	1.02	0.3085
duration	-0.03	0.19	-0.14	0.8882
wheels	7.29	0.55	13.13	<0.0001

Checkpoint 27.7. Guided Practice.

Write out the model’s equation using the point estimates from Table 27.6. How many predictors are there in the model? How many coefficients are estimated?

🔗

Solution.

$\widehat{\texttt{price}} = 36.21 + 5.13 \times \texttt{cond_new} + 1.08 \times \texttt{stock_photo} - 0.03 \times \texttt{duration} + 7.29 \times \texttt{wheels},$ with 4 predictors but 5 coefficients (including the intercept).

🔗

Checkpoint 27.8. Guided Practice.

What does $\beta_4,$ the coefficient of variable $x_4$ (Wii wheels), represent? What is the point estimate of $\beta_4?$

🔗

Solution.

In the population of all auctions, it is the average difference in auction price for each additional Wii wheel included when holding the other variables constant. The point estimate is $b_4 = 7.29\text{.}$

🔗

Checkpoint 27.9. Guided Practice.

Compute the residual of the first observation in Table 27.2 using the equation identified in Table 27.6.

🔗

Solution.

$e_i = y_i - \hat{y_i} = 51.55 - 49.62 = 1.93\text{.}$

🔗

Example 27.10.

In Table 27.4, we estimated a coefficient for cond_new in of $b_1 = 10.90$ with a standard error of $SE_{b_1} = 1.26$ when using simple linear regression. Why might there be a difference between that estimate and the one in the multiple regression setting?

🔗

Solution.

If we examined the data carefully, we would see that there is multicollinearity among some predictors. For instance, when we estimated the connection of the outcome price and predictor cond_new using simple linear regression, we were unable to control for other variables like the number of Wii wheels included in the auction. That model was biased by the confounding variable wheels. When we use both variables, this particular underlying and unintentional bias is reduced or eliminated (though bias from other confounding variables may still remain).

🔗

Subsection 27.1.2 Computational approach to linear models

Previously, using a mathematical model, we investigated the coefficients associated with cond_new when predicting price in a linear model.

🔗

Listing 27.11. R code for Figure 27.12

🔗

set.seed(47)
mariokart |>
  specify(price ~ cond_new) |>
  hypothesize(null = "independence") |>
  generate(reps = 1000, type = "permute") |>
  calculate(stat = "slope") |>
  ggplot(aes(x = stat)) +
  geom_histogram(binwidth = 0.5, fill = IMSCOL["green", "full"]) +
  labs(
    title = "Slopes from models fit to 1,000 permutations of the data",
    x = "Slope of model fit to permuted data",
    y = "Count"
    ) +
  geom_vline(xintercept = 10.9, color = IMSCOL["red","full"], size = 1) +
  theme(plot.title.position = "plot")

Figure 27.12. Estimated slopes from linear models (`price` regressed on `cond_new`) built on 1,000 randomized datasets. Each dataset was permuted under the null hypothesis.
🔗

Example 27.13.

In Figure 27.12, the red line (the observed slope) is far from the bulk of the histogram. Explain why the randomly permuted datasets produce slopes that are quite different from the observed slope.

🔗

Solution.

The null hypothesis is that, in the population, there is no linear relationship between the price and the cond_new of the Mario Kart games. When the data are randomly permuted, prices are randomly assigned to a condition (new or used), so that the null hypothesis is forced to be true, i.e., permutation is done under the assumption that no relationship between the two variables exists. In the actual study, the new Mario Kart games do actually cost more (on average) than the used games! So the slope describing the actual observed relationship is not one that is likely to have happened in a randomly dataset permuted under the assumption that the null hypothesis is true.

🔗

Checkpoint 27.14. Guided Practice.

Using the histogram in Figure 27.12, find the p-value and conclude the hypothesis test in the context of the problem.

🔗

Solution.

The observed slope is 10.9 which is nowhere near the range of values for the permuted slopes (roughly -5 to +5). Because the observed slope is not a plausible value under the null distribution, the p-value is essentially zero. We reject the null hypothesis and claim that there is a relationship between whether the game is new (or not) and the average predicted price of the game.

🔗

Checkpoint 27.15. Guided Practice.

Is the conclusion based on the histogram of randomized slopes consistent with the conclusion obtained using the mathematical model? Explain.

🔗

Solution.

The p-value in Table 27.4 is also essentially zero, so the null hypothesis is also rejected when the mathematical model approach is taken. Often, the mathematical and computational approaches to inference will give quite similar answers.

🔗

Although knowing there is a relationship between the condition of the game and its price, we might be more interested in the difference in price, here given by the slope of the linear regression line. That is, $\beta_1$ represents the population value for the difference in price between new Mario Kart games and used games.

🔗

Listing 27.16. R code for Figure 27.17

🔗

set.seed(47)
mariokart |>
  specify(price ~ cond_new) |>
  generate(reps = 1000, type = "bootstrap") |>
  calculate(stat = "slope") |>
  ggplot(aes(x = stat)) +
  geom_histogram(binwidth = 0.5, fill = IMSCOL["green", "full"]) +
  labs(
    title = "Slopes from models fit to 1,000 bootstrap samples of the data",
    x = "Slope of model fit to bootstrapped data",
    y = "Count"
    ) +
  theme(plot.title.position = "plot")

Figure 27.17. Estimated slopes from linear models (`price` regressed on `cond_new`) built on 1,000 bootstrapped datasets. Each bootstrap sample was taken from the original Mario Kart auction data.
🔗

Example 27.18.

Figure 27.17 displays the slope estimates taken from bootstrap samples of the original data. Using the histogram, estimate the standard error of the slope. Is your estimate similar to the value of the standard error of the slope provided in the output of the mathematical linear model?

🔗

Solution.

The slopes seem to vary from approximately 8 to 14. Using the empirical rule, we know that if a variable has a bell-shaped distribution, most of the observations will be with 2 standard errors of the center. Therefore, a rough approximation of the standard error is 1.5. The standard error given in Table 27.4 is 1.26 which is not too different from the value computed using the bootstrap approach.

🔗

Checkpoint 27.19. Guided Practice.

Use Figure 27.17 to create a 90% standard error bootstrap confidence interval for the true slope. Interpret the interval in context.

🔗

Solution.

Using the bootstrap SE method, we know the normal percentile is $z^\star = 1.645\text{,}$ which gives a CI of $b_1 \pm 1.645 \cdot SE \rightarrow 10.9 \pm 1.645 \cdot 1.5 \rightarrow (8.43, 13.37).$ For games that are new, the average price is higher by between $8.43 and $13.37 than games that are used, with 90% confidence.

🔗

Checkpoint 27.20. Guided Practice.

Use Figure 27.17 to create a 90% bootstrap percentile confidence interval for the true slope. Interpret the interval in context.

🔗

Solution.

Because there were 1,000 bootstrap resamples, we look for the cutoffs which provide 50 bootstrap slopes on the left, 900 in the middle, and 50 on the right. Looking at the bootstrap histogram, the rough 90% confidence interval is $9 to $13.10. For games that are new, the average price is higher by between $9.00 and $13.10 than games that are used, with 90% confidence.

🔗

Subsection 27.1.3 Cross-validation

In Chapter 8, models were compared using $R^2_{adj}.$ In Chapter 25, however, a computational approach was introduced to compare models by removing chunks of data one at a time and assessing how well the variables predicted the observations that had been held out.

🔗

Figure 27.22 was created by cross-validating models with the same variables as in Table 27.4 and Table 27.6. We applied 3-fold cross-validation, so 1/3 of the data was removed while 2/3 of the observations were used to build each model (first on cond_new only and then on cond_new, stock_photo, duration, and wheels). Note that each time 1/3 of the data is removed, the resulting model will produce slightly different model coefficients.

🔗

The points in Figure 27.22 represent the prediction (x-axis) and residual (y-axis) for each observation run through the cross-validated model. In other words, the model is built (using the other 2/3) without the observation (which is in the 1/3) being used. The residuals give us a sense for how well the model will do at predicting observations which were not a part of the original dataset, e.g., future studies.

🔗

Listing 27.21. R code for Figure 27.22

🔗

set.seed(4)
mario_CV1 <- train(price ~ cond_new,
  data = mariokart,
  na.action = na.omit,
  method = "lm",
  trControl = trainControl(
    method = "cv", number = 3,
    savePredictions = TRUE
  )
)

mario_CV2 <- train(price ~ cond_new + stock_photo + duration + wheels,
  data = mariokart,
  na.action = na.omit,
  method = "lm",
  trControl = trainControl(
    method = "cv", number = 3,
    savePredictions = TRUE
  )
)

SSE_CV1 <- mario_CV1$pred |>
  mutate(resid = obs - pred) |>
  select(resid) |>
  pull() %>%
  `^`(2) |>
  sum() |>
  round(2)

SSE_CV2 <- mario_CV2$pred |>
  mutate(resid = obs - pred) |>
  select(resid) |>
  pull() %>%
  `^`(2) |>
  sum() |>
  round(2)

mario_CV1$pred |>
  mutate(resid = obs - pred) |>
  ggplot() +
  geom_point(aes(x = pred, y = resid, color = Resample, shape = Resample)) +
  geom_rug(aes(y = resid)) +
  geom_hline(yintercept = 0) +
  ylim(c(-15, 25)) +
  xlim(c(20, 65)) +
  labs(
    color = "CV Fold", shape = "CV Fold",
    x = "Predicted", y = "Prediction error (residual)"
  ) +
  annotate(geom = "text", x = 55, y = 25, label = paste("CV SSE =", SSE_CV1)) +
  labs(title = "Model with one predictor") +
  theme(
    legend.position = c(0.2, 0.7),
    legend.background = element_rect(color = "gray", fill = "white")
  ) +
  scale_color_openintro("three")

mario_CV2$pred |>
  mutate(resid = obs - pred) |>
  ggplot() +
  geom_point(aes(x = pred, y = resid, color = Resample, shape = Resample)) +
  geom_rug(aes(y = resid)) +
  geom_hline(yintercept = 0) +
  ylim(c(-15, 25)) +
  xlim(c(20, 65)) +
  labs(
    color = "CV Fold", shape = "CV Fold",
    x = "Predicted", y = "Prediction error (residual)"
  ) +
  annotate(geom = "text", x = 55, y = 25, label = paste("CV SSE =", SSE_CV2)) +
  labs(title = "Model with four predictors") +
  scale_color_openintro("three")

Checkpoint 27.23. Guided Practice.

In Figure 27.22(b), note the point at roughly predicted = 50 and prediction error = 10. Estimate the observed and predicted value for that observation.

🔗

Solution.

The predicted value is roughly $\widehat{\texttt{price}} = \$50.$ The observed value is roughly $\texttt{price}_i = \$60$ (using $e_i = y_i - \hat{y}_i).$

🔗

Checkpoint 27.24. Guided Practice.

In Figure 27.22(b), for the same point at roughly predicted = 50 and prediction error = 10, describe which cross-validation fold(s) were used to build its prediction model.

🔗

Solution.

The point appears to be in fold 2, so folds 1 and 3 were used to build the prediction model.

🔗

Checkpoint 27.25. Guided Practice.

By noting the spread of the cross-validated prediction errors (on the y-axis) in Figure 27.22, which model should be chosen for a final report on these data?

🔗

Solution.

The cross-validated residuals on cond_new vary roughly from -15 to 15, while the cross-validated residuals on the four predictor model vary less, roughly from -10 to 10. Given the smaller residuals from the four predictor model, it seems as though the larger model is better.

🔗

Checkpoint 27.26. Guided Practice.

Using the summary statistic cross-validation sum of squared errors (CV SSE), which model should be chosen for a final report on these data?

🔗

Solution.

The CV SSE is smaller (by a factor of almost two!) for the model with four predictors. Using a single valued criterion (CV SSE) allows us to make a decision to choose the model with four predictors.

🔗

Section 27.2 Interactive R tutorials

Navigate the concepts you’ve learned in this part in R using the following self-paced tutorials. All you need is your browser to get started!

🔗

Tutorial 6: Inferential modeling

🔗

You can also access the full list of tutorials supporting this book at https://openintrostat.github.io/ims-tutorials.

🔗

Section 27.3 R labs

Further apply the concepts you’ve learned in this part in R with computational labs that walk you through a data analysis case study.

🔗

Multiple linear regression - Grading the professor

🔗

You can also access the full list of labs supporting this book at https://www.openintro.org/go?id=ims-r-labs.

🔗

Prev Top Next