The formula for the logistic regression is as follows:.
\begin{equation*}
\log\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 \cdot \text{balance} + \beta_2 \cdot \text{income} + \beta_3 \cdot \text{student}
\end{equation*}
Section 9.7 Build Logistic Regression Model
When building a logistic regression model, it is going to look very similar to the linear regression model we built starting in Section 8.7. The major difference from a coding perspective between a linear regression and a logistic regression model is that instead of using
lm(), you need to use glm().
Before we create our model, let’s take a step back. Logistic regression, just like linear regression, is a tool to help us predict an outcome. Specifically, we are predicting the probability of an outcome.
where (p) is the probability of default. The left-hand side is called the logit, or log-odds. Notice that the right-hand side looks just like a straight-line regression.
In Linear Regression, your line goes on forever in both directions. But in probability, you can’t be less than 0% or more than 100%. If you used a straight line to predict defaults, the math might eventually say someone has a -20% chance of defaulting, which is impossible.
To fix this, we use the Sigmoid Function. This “squashes” the straight line into an S-curve that stays strictly between 0 and 1.
ggplot(cc_data, aes(x = balance, y = as.numeric(default) - 1)) +
geom_point(alpha = 0.2) +
geom_smooth(method = "glm", method.args = list(family = "binomial"), se = FALSE, color = "red") +
labs(y = "Probability of Default", x = "Balance")
In the figure above, notice how the red line transitions from a low probability to a high probability as the balance increases. Unlike a straight linear line, this S-curve naturally ’levels off’ at 0 and 1, ensuring our predictions remain realistic probabilities.
Now, let’s get to business and create our logistic regression model.
# We are using binomial as family since default is yes/no
train_model <- glm(default ~ student + balance + income,
family = "binomial", data = training_data)
summary(train_model)
Call:
glm(formula = default ~ student + balance + income, family = "binomial",
data = training_data)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.090e+01 5.931e-01 -18.372 < 2e-16 ***
studentYes -8.245e-01 2.835e-01 -2.908 0.00364 **
balance 5.796e-03 2.805e-04 20.661 < 2e-16 ***
income 2.768e-06 9.736e-06 0.284 0.77614
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 2043.8 on 6999 degrees of freedom
Residual deviance: 1098.2 on 6996 degrees of freedom
AIC: 1106.2
Number of Fisher Scoring iterations: 8
After we create our model and call
summary(), we get an output that should look similar to the outputs we’ve seen before in this chapter. Taking a look at our wall of words, it is incredibly important to note that the numbers here represent the log-odds
Model interpretation:
-
Student (negative coefficient, not significant): Students less likely to default.
-
Balance (positive, significant): Higher balance = higher odds of default.
-
Income (tiny effect, not significant).
-
“Fisher Scoring iterations: 8” means R needed 8 rounds to find best fit.
One thing to note is that unlike a linear regression model, logistic regression models take a few tries to create the best model (in this case, 8).
While the outputs look the same, there is a major difference in the numbers that are output in a logistic regression vs a linear regression. Remember earlier when we calculated the probability of someone defaulting? It was mentioned that probabilities will always be a number between 0 and 1. But, when we are creating a line in a regression model, we can get any numbers between -∞ and ∞. How does logistic regression, which uses probabilities, make a line then?
Log-odds!
We need to get a number that actually has a range, and this is exactly where log-odds come in. Because with regression, we need to get a line (and to build a line, we need a formula). Instead of modeling probability, we model the log of the odds. Here is the step by step of how logistic regression works:
-
Start with a probability.
-
Convert to odds.
-
Takes the log of the odds
-
Using those log odds, fits a regression line
-
Converts it back with a logistic function
# Convert log-odds → odds ratios
exp(coef(train_model))
# Example: students have 0.43× the odds (≈57% lower odds) of defaulting
(Intercept) studentYes balance income 1.850225e-05 4.384379e-01 1.005813e+00 1.000003e+00
A very important thing to note is that odds and probability are not the same thing. With odds ratios, we are looking to see how much larger or smaller it is than 1.
-
OR = 1 → no change
-
OR > 1 → increase
-
OR < 1 → decrease
Breaking this down using our model:
-
If you are a student, the odds of you defaulting decrease, as the odds ratio is 0.438. Their odds are 56.2% lower than non-students (1 - 0.438). This is a significant difference.
-
For every dollar your balance increases, the odds of you defaulting increase by 0.58%. This seems small per dollar, but compounds rapidly across real-life balance amounts. For example, increasing the balance by $100 multiplies the odds of default by about 1.78, which is a 78% increase in odds.
-
The odds ratio for income is so small (OR = 1.000003), that a $1 increase in income has basically no influence on your default odds.
Subsection 9.7.1 McFadden’s Pseudo-R²
Unlike in linear regression, logistic regression does not have an R²: (for a review on R²: see Section 8.7). However, we can use the
pR2() function from the pscl library ([D.1.21]) to calculate a pseudo R² value.
# McFadden's pseudo-R2
library(pscl)
pR2(train_model)["McFadden"]
fitting null model for pseudo-r2
McFadden
0.4626515
As of right now, this is a moderately strong model!
Subsection 9.7.2 Variable Importance
With the
varImp() function inside the caret package ([D.1.3]), we can also see which variables are the most important in our regression model.
# Variable importance
library(caret)
varImp(train_model) |> arrange(desc(Overall))
# balance seems to be the most impactful in our model
Overall balance 20.6607703 studentYes 2.9080015 income 0.2843578
Subsection 9.7.3 Multicollinearity check
Just like in Section 7.8, we want to check to see if there is any interactions between some of our variables that may be impacting our regression model. Using the
vif() function from the car library ([D.1.2]), we can see the interactions between our variables in our model.
# Multicollinearity check
library(car)
vif(train_model)
# does not seem to be any multicollinearity
# anything less than 5 means no multicollinearity
student balance income 2.671007 1.072618 2.593366
As of right now, it is safe to say that there is no multicollinearity, as all values are less than 5.
