Feature and parameter optimization - which comes first?

qwertz2

Dear all,

When I thought about feature and parameter selection, I was reminded of the well known hen and egg problem: Which one comes first?

The feature selection has a nested learner which requires parameters that are not known at the beginning. Based on its result, attributes are selected.

In the next step I do parameter optimization for the following learner. However, the result ist linked to the quality of attributes fed in.

Is there a golden rule of thumb how to approach this dilemma?


Best regards
Sachs

Telcontar120

Personally I tend to focus on feature selection first, since it is more concerned with dimensionality reduction and elimination of not-quite-necessary attributes.  Several feature selection approaches are unsupervised and do not even require an inner learner (PCA, remove correlated attributes, etc).  Even when you are using Forward Selection or Backward Elimination you could use an inner learner that does not require optimization, such as Naive Bayes.

Once I have a reduced set of attributes, then I tend to focus on the true modeling optimization.  But I would say feature engineering (creating new attributes from the existing raw attributes) may be even more important to getting the best model than either feature selection or optimization, so don't neglect that either!

 

qwertz2

 

Hi Brian,

 

Thank you for your comment.

 

So far I thought of using the same learner within the optimizer as in the modelling. The idea of taking a naive bayes instead sounds interesting. Though I have to admit that I cannot imagine how this should work. (I should have mentioned that I am working on series modelling.) Nevertheless I will give it a try and hopefully get surprised

 

Thank you as well for sharing your experience on the importance of feature engineering. This seems to be a point which is not in the focus of most discussions. So I am going to pay more attention to this and rework my process accordingly.

 

 

Kind regards

Sachs

IngoRM

+1 on the comments on feature engineering!  The impact of this is often much bigger than picking an uber-complex model or optimize its parameters to death in my opinion...

 

My personal approach is very iterative.  I start finding any model which is somewhat better than guessing.  As soon as I have this, I optimize its parameters a little bit.  Then I spend most time in feature engineering.  And in the end, I go back to model selection and parameter optimization.  If this is still not good enough, I iterate over this multiple times with the main focus on the the feature engineering part.

 

If you want to go really fancy, you can try some of the automatic genetic-programming-inspired operators like YAGGA2 ;-)

 

Cheers,

Ingo

qwertz2

Thanks for all your helpful comments :smileyhappy: