Compare ROC Parameter

PlatyQ

Hello,

I have some questions but I am not anglophone and may be it will difficult to you to understand my questions and difficult to me to understand your answer.

Please if I am not understandable say it to me.

I try to make a learning machine to a stage in computer science and I want compare some models to this.

I have very few examples (20) but I have to try anyway.

My troubles are to adjust the parameters of "compare ROC", I think understand what is a ROC curve and how interpreted but may be I make a mistake cause I don't understand why they are two parameters :

• number_of_folds.
• split_ratio.

According to me if i put 4 in number_of_folds for each models Rapidminer compares 4 training machine with 15 examples to learn and evaluate them with the last 5 and after create for each model the ROC Curves. And it match with that I want (compare some models to my 20 examples).

But split ratio can not be disable so it can play a role in the roc curve but i don't understand which. May be if I put 1 in this parameter it work as I think?

 

Thank for your help

lionelderkrikor

Hi @PlatyQ,

 

Good question.

I asked myself the same question when I used this operator for the first time, but I never .......dared to ask the question.

Your post prompted me to look for an answer in the documentation : 

According to the documentation, Compare ROC behaves like a Split Validation or a Cross Validation operator : 

 - if number of folds = -1, then Compare ROC behaves like a Split Validation with the defined split ratio.

 - if number of folds is in Range ]1, infinite[, then Compare ROC behaves like a Cross Validation with the defined number of folds.       (and in this case, the defined split ratio is not taken into account).

 -  if number of folds = 0 or 1 , then RapidMiner raise an error .... and it's logic.

 

NB : That's what I understand from the documentation, but thanks to the experts to confirm or..... invalidate this explanation of the behaviour of Compare ROC operator.

 

I hope it helps,

 

Regards,

 

Lionel

PlatyQ

Hi @lionelderkrikor and thank for your response

I also search in documentation before ask the question but I read and read again and no see that "if number of folds is in Range ]1, infinite[ [...] (and in this case, the defined split ratio is not taken into account)". I think your right cause it seems more logic than use the two options in the same time but I regret than it is not well-defined I the documentation.

I am reassure by your response but I hope we can have a second opinion.

 

(And may be somebody know how to ask modifications in the documentation? If two people have doubts may be we are not the only ones.)

 

thanks