Memory usage problem

Danyo83

Hi there,

I have already read and applied everything with reference to
http://rapid-i.com/content/view/17/211/lang,en/

But now the maximal memory is up to 8 GB but it uses only up to 2 GB RAM. How can I increase this? even during a long process it does not increase...
using the parallel extension does even make it worse... Any suggestions??

Many thanks in advance

Daniel

Nils_Woehler

Hi,

on what platform are you? How do you start RapidMiner (Script, exe, jar)?

Best,
Nils

Danyo83

Hi,

it is a linux system and I start the Gui over a command line with Putty and Xming.
I use this command to enhance the maximum available memoray usage:  "java -Xmx8000m -jar /usr/local/rapidminer/lib/rapidminer.jar"

When the Gui has startet and I proceed a process I see the system monitor and the max memory has raised up to the desired amount of RAM, but the total memory is not more the 2 GB RAM.

Since I have multiple cores I want to use the parallel extension and the parallel operators e.g. cross validation. But using this the process is even slower than using the simple cross validation.

Regards, 

Daniel

Nils_Woehler

Hi,

hm that is somehow weird. The way you start RapidMiner it should use all of the available memory and not only 2GB.
Do you have a 32-bit or a 64-bit java installed ("java -version")?

Currently it is not possible to do the cross validation in parallel. The parallel extension executes different branches of your process in parallel but is not able to parallelize single operators.

Best,
Nils

Danyo83

Hi,

I am sorry I was a bit confused. It is a AIX system on an IBM power machine (power 7). 16 cores with a total of 512 GB RAM. So I want to use at least 80 GB RAM and with the above command the max memory (system operator) goes up to 80 GB, but during the process, the total is about 2 GB RAM.
I think it is 64-bit Java.

Maybe I explained it wrong. I just use the parallel cross valdation operator and set the number of threads to 16. But it is always slower than using the simple cross validation operator.

Nils_Woehler

That is really strange. Does the normal X-Validation operator use more than 2GB?
If you are running on such a powerful machine and RapidMiner is able to acquire 80gb of memory, it should also be able to use it.
The beauty of Java is, that we don't need to implement memory handling because it is done by Java itself.
But in these rare cases it is hard to identifiy the problem because everything regarding memory handling is done under the hood....

Danyo83

Hey Nils,

the strange thing is that using the parallel Xvalidation operator is slower than the usual Xvalidation operator and NO, it does not use more memory using the simple one, even using only one single core would mean 16 GB RAM memory. I've got the impression, the longer a process is running, little by little it also uses more memory but my process which I run now only have a few testing exmaple to approve the process. So it hardly overcomes the usage of 2GB RAM. But after that I want to use the full data set and need the full power...
Right now the process hardly runs faster than on my 4 GB RAM Laptop :-(

Any suggestions?!

Daniel

Marco_Boeck

Hi,

the value displayed as "Max" is the value available to RapidMiner and it will be used if needed. The value displayed as "Total" is the current memory consumption. If that value is not going beyond 2GB, it usually simply indicates your process is not needing any more RAM. Note that you cannot speed up a process by giving it more RAM - it takes what it needs (if possible), but excess RAM is of no use whatsoever to the process.
If you setup a memory intensive process RapidMiner should automatically use the RAM up the maximum value displayed as "Max".
Also note that by using "Xmx8000m" (Maximum heap size) you're setting the maxium amount of RAM Java will use, but this value is NOT used right from the start. If you want Java to instantly allocate a set amount of RAM (which is not needed), you have to use the "Xms" (Initial heap size) parameter as well.

Regards,
Marco