The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
RemoveCorrelatedFeatures & unseen datasets
Contributor I
We are using RapidMiner for much of our Educational Data Mining Research and have a question for you:
We frequently use "RemoveCorrelatedFeatures" on our "training" datasets to generate models with a subset of (only) relevant attributes. However, we're running into some difficulties when trying to apply our model on our "unseen" datasets.
After we build a model with our "training" dataset with correlated features removed, we want to apply the models to "unseen" data from a different dataset. However, we must manually remove those attributes from the unseen dataset by hand to match what RemoveCorrelatedFeatures removed for attributes from the training set in order to be compatible with the model.
Is there a way to do this in RapidMiner more easily without having to do something outside of RapidMiner?
Thanks in advance!
We frequently use "RemoveCorrelatedFeatures" on our "training" datasets to generate models with a subset of (only) relevant attributes. However, we're running into some difficulties when trying to apply our model on our "unseen" datasets.
After we build a model with our "training" dataset with correlated features removed, we want to apply the models to "unseen" data from a different dataset. However, we must manually remove those attributes from the unseen dataset by hand to match what RemoveCorrelatedFeatures removed for attributes from the training set in order to be compatible with the model.
Is there a way to do this in RapidMiner more easily without having to do something outside of RapidMiner?
Thanks in advance!
0
Answers
yes, of course
You could extract weights from your resulting training data set with the operator "Data to Weights". This will result in a new attribute weights object with a weight of 1 for all remaining attributes. Then you can apply those weights on the testing data set with the operator "Select by Weights" which gets the test data together with the extracted weights. Make sure that the parameter "deselect unknown" is set to true (which is the default but anyway).
Cheers,
Ingo
You've made many people in our lab happy.