Cutoff Points

The Cutoff Points Analysis complements nicely the ROC Analysis of the previous section. The Cutoff Points Chart shows clearly the intersection of both the sensitivity (TPR) and specificity (TNR) lines and also the intersection of the FPR line with the FNR line. Seeing how these four lines change with model output is a great aid for choosing the Ideal Cutoff Point for your test values.

It’s true that the Ideal Cutoff Point varies from problem to problem, as one might be interested in minimizing or maximizing different things. Sometimes the goal is to minimize the number of false positives; other times the number false negatives. Other times, though, one might need to maximize the number of true positives or true negatives. With the help of the Cutoff Points Chart of GeneXproTools you can see clearly the best way to move your model threshold to achieve your goals.

Notwithstanding, there is a generic Optimal Cutoff Point. This Optimal Cutoff Point is given by the Youden index and you can see where it exactly lays in the Cutoff Points Chart. When you tick the Show OMT check box, GeneXproTools draws the Optimal Model Threshold in dark brown.

The Youden index J returns the maximum value of the expression (for inverted models it returns the minimum):

where SE_(t) and SP_(t) are, respectively, the sensitivity and specificity over all possible threshold values t of the model. Thus, the Optimal Model Threshold corresponds to the model output at the Optimal Cutoff Point.

In the Cutoff Points Table, GeneXproTools also shows all “SE + SP -1” values and highlights in light green the row with the Optimal Cutoff Point and corresponding Optimal Model Threshold. These parameters are also shown in the companion Cutoff Points Statistics Report.

 

The Optimal Model Threshold can be obviously used to infer a Confusion Matrix (in GeneXproTools it is called ROC Confusion Matrix) and, in the Cutoff Points Table, you have access to the Predicted Class, the Match, and Type values used to build this ROC Confusion Matrix (you can see the graphical representation of the ROC Confusion Matrix in the Confusion Matrix section).

The visualization of the ROC Confusion Matrix is a valuable tool and can in fact be used to determine the right number of buckets to achieve a good fit with the Logistic Regression Model. But GeneXproTools allows you to do more with the ROC Confusion Matrix and associated Optimal Model Threshold. By allowing the conversion of Logistic Regression runs to the Classification Framework, you can use this model, with its finely adapted Optimal Model Threshold, straightaway to make discrete classifications using the Classification Scoring Engine of GeneXproTools.

The Youden index is also used to evaluate a wide set of useful statistics at the Optimal Cutoff Point (OCP statistics for short). They include:

• TP (True Positives)
• TN (True Negatives)
• FP (False Positives)
• FN (False Negatives)
• TPR (True Positives Rate or Sensitivity)
• TNR (True Negatives Rate or Specificity)
• FPR (False Positives Rate, also known as 1-Specificity)
• FNR (False Negatives Rate)
• PPV (Positive Predictive Value)
• NPV (Negative Predictive Value)
• Classification Accuracy (Correct Classifications)
• Classification Error (Wrong Classifications)

How they are calculated is shown in the table below ("TC" represents the number of Total Cases):

It is worth pointing out that OCP statistics are quantile-independent and therefore are a good indicator of what could be achieved with a model in terms of logistic fit and accuracy.