The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Extremely slow and running out of memory, but not using Java memory optimaly

IrisSIrisS Member Posts: 2 Learner III
edited November 2018 in Help

Hi,

 

During handling not thet extensive datasets datasets (6000 records with microtext), Rapidminer signals it runs out of memory and on others it's just extremely slow (have a knn-2 running for 22hrs now). I have 36Gb of ram, perform quite some preprocessing (including strong pruning while creating TF-IDF vector, not keeping any text either), and should have 8Gb available for Java. However, for some reason, Rapidminer only uses only 256Mb, while no other tasks are taking away more than 50Mb at any time. Is there a setting hidden somewhere, where this can be adapted? I'm guessing that is the bottleneck...

 

I'm using the latest version of RapidMiner Studio Educational in a Windows10 environment (64bit). Already went through the .bat file for MAX_JAVA_MEMORY as mentioned here in some other posts, but that string is simpely not in there. Did this move maybe?

Tagged:

Answers

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    The way we handle memory has changed from v5.3 to the current v7.5, so I don't know what versions you are comparing.  I don't know what your process looks like but if you generating n-grams, you could easily run out of memory depending on your set up. It's really hard to troubleshoot without looking at how your process is built along with where it chokes. 

Sign In or Register to comment.