Evaluate Model

Section 9.9 Evaluate Model

Using real data (our test data) and a built logistic regression model, we now have predicted if a person will or will not default on their credit card. The next question is: how well did our model predict if someone defaulted on their credit card or not?

🔗

To do this, we run the command confusionMatrix. This comes from the caret package which we have already loaded.

🔗

conf_matrix <- confusionMatrix(test_data$pred_class, test_data$default, positive = "Yes")

conf_matrix

Confusion Matrix and Statistics

          Reference
Prediction   No  Yes
       No  2886   71
       Yes   14   29
                                          
               Accuracy : 0.9717          
                 95% CI : (0.9651, 0.9773)
    No Information Rate : 0.9667          
    P-Value [Acc > NIR] : 0.06743         
                                          
                  Kappa : 0.3934          
                                          
 Mcnemar's Test P-Value : 1.247e-09       
                                          
            Sensitivity : 0.290000        
            Specificity : 0.995172        
         Pos Pred Value : 0.674419        
         Neg Pred Value : 0.975989        
             Prevalence : 0.033333        
         Detection Rate : 0.009667        
   Detection Prevalence : 0.014333        
      Balanced Accuracy : 0.642586        
                                          
       'Positive' Class : Yes

Another wall of data. The first thing we see if a table of Reference vs Prediction. This tells us our true/false negatives/positives. For instance, there were 14 false positives. Most of our data is accurate: 2886 true negatives (most of our data is negative - people do not default).

🔗

Next, we get some performance metrics. Here is an initial breakdown of what we see:

🔗

Accuracy ≈ how often model is correct overall.
🔗
- The model is very accurate, at about 97%.
  🔗
  
  🔗
🔗
95% CI: Where the accuracy of our model most likely lies.
🔗

🔗
No Information Rate (NIR): If we just predicted “No” for each row, how accurate would we be.
🔗
- This is not much different from the actual accuracy.
  🔗
  
  🔗
🔗
P-value [Acc > NIR]: Is our model statistically different than if we just put “No” for each.
🔗
- The p-value is not statistically significant, meaning there is no difference between running our model or just predicting “No” for each person.
  🔗
  
  🔗
🔗
Sensitivity: how well it finds actual defaults.
🔗
- Only 29% of actual defaulters are detected.
  🔗
  
  🔗
🔗
Specificity: how well it identifies non-defaulters.
🔗
- It predicts non-defaulters at almost 100%.
  🔗
  
  🔗
🔗
High accuracy but low sensitivity means the model misses many rare defaults.
🔗

🔗
Balanced Accuracy is a better metric for imbalanced data.
🔗

🔗

🔗

Prev Top Next