Confidence or Prediction Intervals

npapan69

Dear All,
When reporting a performance metric (ie AUC) of a model that was trained on a single data set and tested on a hold-out set, what is the proper way to assess its variance? Calculating the confidence intervals of the AUC or the prediction intervals?
Many thanks
Nikos

varunm1

Hello @npapan69

This is a bit tricky as they both are centered around the same value, but a prediction interval is wider than the confidence interval. So in case of error reduction or a better accurate prediction you can go with prediction intervals.

npapan69

Thanks Varunm1 for your answer. Is the prediction interval provided by any of the rapidminer operators?

varunm1

Hello @npapan69

I am checking that, and I see no option in RM for that. I think it is a bit complicated to calculate.

@mschmitz or @IngoRM any comments on this.

Thanks