ROC threshold blue line

damir_imamovic

Can anyone explain what the blue threshold line means and how to interpret it? Is there any way to disable it remove from chart in Rapid MIner?
Thanks in advance.

jacobcybulski

There is an easy way to interpret the threshold line. Both the ROC red line and a Threshold blue line share the same x-axis (FPR), however their y-axes are different, i.e. TPR for ROC and a threshold value for Threshold. If you want for example to find out what threshold generated a particular point in an ROC curve, just draw a vertical line crossing that ROC point, find where it intersect with the threshold line, and at that point draw the horizontal line to find the corresponding threshold value. Typically people want to find out the "optimum" threshold as the ROC point closest to left-upper corner (which is not quite correct) and you could find the threshold that way.

bernardo_pagnon

Hello,

the ROC curve is built by decreasing the thresholds, from 1 to 0. At each value, we have a confusion matrix which gives a point in the graph (FPR versus TPR). This the blue curve, the thresholds that were used to build the ROC. I am not sure if it cannot be disabled, but if you use the Compare ROC operator (even with one model) the blue curve does not appear. 

Regards,
Bernardo

damir_imamovic

@bernardo_pagnon thanks for your quick response. I wish i can delete blue line. Even if I use compare roc operator with one model there is still roc threshold blue line. I found that (The ROC (thresholds) curve just shows this confidence threshold (sometimes also called confidence cut).) Best solution would be to remove it. I'll make one ROC in excel.

amitd

@jacobcybulski, can you please clarify your comment - "Typically people want to find out the "optimum" threshold as the ROC point closest to left-upper corner (which is not quite correct) and you could find the threshold that way"? Why is this point not an optimum threshold? If not, what would be an optimum threshold (based on what criteria) and how do you find it?

jacobcybulski

@avd , you have asked the right question, which really addresses my qualification of the general belief that the optimum threshold can be found in the point closest to left-upper corner of ROC. The key of course is your observation that it all must depend on the selected criteria. The simplistic approach is to find the threshold, which maximises TPR and minimises FPR, which cannot be done unless you define some cost function to involve TPR and FPR, or some other metrics. There are many approaches to doing this, e.g. by looking at the ROC gradient (under the assumption of TPR and FPR having equal costs), looking at geometric mean or sensitivity and specificity, using Youden’s J index, or by tuning the threshold according to some other statistic, e.g. threshold vs kappa or F1, or relying on the precision-recall curve, etc.