"[SOLVED] Using user set confidence for AUC performance, not working.."

mafern76

Hi!

What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.

What I did was:

1. Rename confidence(class) to confidence(1).
2. Generate attribute confidence(0) = 0.
3. Set roles, confidence(1) to confidence_1 and 0 to 0.
4. Rename prediction(multiclass) to prediction(singleclass).
5. Map classofinterest to 1 and default to 0.
6. Transform the polynominal prediction to binominal.

This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.

Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.

On the other hand, the prediction label works because I can get accuracy.

I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?

Thanks a lot!

mafern76

Oh well an apple fell upon my head and I realized the problem was I was assuming AUC would use confidence_1, when it could perfectly use confidence_0.

Solution: instead of (2) generate attribute confidence(0) = 0, generate it equal to 1- confidence_1...