## 11.5 Model Performance

• Say we decide that a value of 0.15 is our optimal cutoff value to predict depression using this model.
• We can use this probability to classify each row into groups.
• The assigned class values must match the data type and levels of the true value.
• It also has to be in the same order, so the 0 group needs to come first.
• I want this matrix to show up like the one in Wikipedia, so I’m leveraging the forcats package to reverse my factor level ordering.
• We can calculate a confusion matrix using the similarly named function from the caret package.
library(caret)

plot.mpp$pred.class2 <- ifelse(plot.mpp$pred.prob <0.15, 0,1)
plot.mpp$pred.class2 <- factor(plot.mpp$pred.class2, labels=c("Not Depressed", "Depressed")) %>%
forcats::fct_rev()

confusionMatrix(plot.mpp$pred.class2, forcats::fct_rev(plot.mpp$truth), positive="Depressed")
## Confusion Matrix and Statistics
##
##                Reference
## Prediction      Depressed Not Depressed
##   Depressed            40           121
##   Not Depressed        10           123
##
##                Accuracy : 0.5544
##                  95% CI : (0.4956, 0.6121)
##     No Information Rate : 0.8299
##     P-Value [Acc > NIR] : 1
##
##                   Kappa : 0.1615
##
##  Mcnemar's Test P-Value : <2e-16
##
##             Sensitivity : 0.8000
##             Specificity : 0.5041
##          Pos Pred Value : 0.2484
##          Neg Pred Value : 0.9248
##              Prevalence : 0.1701
##          Detection Rate : 0.1361
##    Detection Prevalence : 0.5476
##       Balanced Accuracy : 0.6520
##
##        'Positive' Class : Depressed
## 
• 123 people were correctly predicted to not be depressed (True Negative, $$n_{11}$$)
• 121 people were incorrectly predicted to be depressed (False Positive, $$n_{21}$$)
• 10 people were incorrectly predicted to not be depressed (False Negative, $$n_{12}$$)
• 40 people were correctly predicted to be depressed (True Positive, $$n_{22}$$)

Other terminology:

• Sensitivity/Recall/True positive rate: P(predicted positive | total positive) = 40/(10+40) = .8
• Specificity/true negative rate: P(predicted negative | total negative) = 123/(123+121) = .504
• Precision/positive predicted value: P(true positive | predicted positive) = 40/(121+40) = .2484
• Accuracy: (TP + TN)/ Total: (40 + 123)/(40+123+121+10) = .5544
• Balanced Accuracy: $$[(n_{11}/n_{.1}) + (n_{22}/n_{.2})]/2$$ - This is to adjust for class size imbalances (like in this example)
• F1 score: the harmonic mean of precision and recall. This ranges from 0 (bad) to 1 (good): $$2*\frac{precision*recall}{precision + recall}$$ = 2*(.2484*.8)/(.2484+.8) = .38