[SOLVED] Performance RAM Rapid Miner

marcopo

Hi,

at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:

criterion: accurancy

minimal size for split: 4

minimal leaf size: 2

minimal gain : 0,1

maximal depth 10

confidence 0,25

Cross Validation 5

sampling type : stratifield sampling

Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?

Best Regards and thank you

Marco

MariusHelf

Hi Marco,

the decision tree is one of the worst choices for data with many attributes. (It is perfectly fit to process data with a huge amount of examples though, with only few attributes).

For data with many attributes, but only a few examples as in your case, try a Linear SVM. Please note that the parameter C of the SVM must be optimized. You can use the Optimize Parameters (Grid) for that. For a choice of the search range for C, please have a look at this thread. Please let me know if you have any problems.

Concerning the usage of RAM, we are planning to make the maximum amount of memory available to RapidMiner configurable with the next release, which is planned to be available to the public on next monday.

Best regards,
Marius

marcopo

Thanks a lot for the replay.

It's funny I am doing at the moment the SVM with optimization ;-)
I am eager to use the new release.

Best Regards
Marco

MariusHelf

For SVM optimization and understanding you may be interested in this thread: http://rapid-i.com/rapidforum/index.php/topic,6194.0.html
I'll mark this topic as solved now.

Best regards,
Marius