The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Large Data Handeling

sangihisangihi Member Posts: 5 Contributor II
edited November 2018 in Help
Hi,

I'm trying to classify huge data (about 500K rows) using Decision Trees in RapidMiner.
Unfortunately, the GUI gives me not enough memory even for 50K data.

Since C4.5 is able to learn the whole data instance by instance, Is it possible to "stream" data so it does not require the whole data in memory?

Answers

  • marcin_blachnikmarcin_blachnik Member Posts: 61 Guru
    1) RM manage memory very efficiently so if you are able to store the dataset in the memory, you shouldn't have any problems
    2) If you have a problem then use database to store the data
    3) The main problem is that the decision tree (DT) build in RM is very crappy. My advice - don't use it, use J48 from weka plugin instead, or use other models kNN, Naive Bayes etc.
Sign In or Register to comment.