"[SOLVED] the Curse of Text High Dimensional nature"

siamak_want

Hi experts,

My question is about my new problem in text classification in a real-world project:

I have made a classification model based on a relatively huge labeled dataset. the model has about 50,000 attributes. Now I want to apply my model on the new unseen data. Here the problem turns out...
I have about 2000 attributes in my test data and about 500 of them do not exist in my model at all. I mean my model has not seen such attributes in the training time because these attributes have not exist in my train data set. So Is my model able to classify such a dynamic features accurately? Please explain if you have any idea about this challenge.

Thanks a lot.

MariusHelf

This is not the classical curse of high dimensionality, but nevertheless there's a solution: you have to connect the "wor" output of the training Process Documents operator to the Process Documents operator in the apply branch. Please have a look at this thread: http://rapid-i.com/rapidforum/index.php/topic,4802.0.html

Best, Marius

siamak_want

Thanks a lot Marius.
You were right.