"Brute force feature selection"

ammargh

Shouldn't brute force feature selection return the best performance? The performance I had using features I have selected manually was better than the performance using features returned from brute force selection component.

Is this normal?

MariusHelf

Hi,

how big is the difference between your manual performance and the performance of the features found by brute force? Are you using a Cross Validation?

Please keep in mind that by default the X-Validation always uses random splits, and thus small performance changes can be produced by randomness. To enforce the same splits in all X-Validations in all iterations, and also in your manual evaluation, you can set the local random seed of all X-Validation operators to a constant. Then only the performance of the algorithms and the features is compared, and the factor "random" is eliminated.

Best regards,
Marius

ammargh

I see your point.
I will follow your advise
Thank you.

mafern76

It should be impossible for a manual selection to do better than a brute force selection, because the latter simply tries all possible combinations.

I agree with Marius, your results are biased because of randomness and I might add maybe due to high variation in your model algorithm.