"rapidminer process in java, memory problem"

sometech

Hello,
I have what seems to be a peculiar problem.
My test machine is running linux (centOS) and has 8Gb ram (I have also witnessed this on a windows 2003 server 64bit).
I am trying to get rapidminer to consistantly use all the ram I allow it to use (for this discussion, it has all 8Gb).
This seems to be specific to training a model.
When I run the rapidminer process directly i.e. pass an xml file to the scripts/rapidminer file then the process eventually uses almost all 8Gb.
I also have a java app that communicates with rapidminer through the api, when I try to do the same training through this java app the process never uses more than half (just over, about 58%) of the available memory and the training can take almost twice as long.

I'm wondering if there is some config parameter that has to be set in the java app to tell rapidminer it can use all the available memory, but i can't find any reference to it.
Any suggestions?

thanks

land

Hi,
rapidMiner uses all memory it needs. There is no global parameter telling it to save memory in change for a longer runtime. But there are several possibilities explaining the memory effect:
If you run java with both parameters -Xms and -Xmx, the memory will instantly be allocated and reserved, if it's used later or not. Perhabs your application does not do this and the training never used more than 58% of the total available memory.
But this does not explain the difference in the time. Two thoughts on this:
- There are several parameters on learners influencing the memory and time consumption. For SVMs for example the Kernel Cache Size is important, for memory AND time. Do you exactly run the same process? Or do you use the single operators, perhabs with different parameters?
- Does your application use the same JVM Version as rapid miner? Current rapidMiner versions are delivered with SUN Java 1.6, does your application perhabs use another JVM Implementation?


Greetings,
  Sebastian

sometech

Hello,

thanks land,
i am using jvm1.6.11 which seems to be the same as rapidminer.
The other point of ensuring that the parameters are the same is somewhat harder to define, since rapidminer does a number of things by default and I haven't yet determined if the java app uses the same defaults or if they need to be explictly set.
currently investigating this to ensure consistancy, was just hoping for something generic!
i'll post back once I can esnure that the two process are doing exactly the same thing

thanks