Remember/Recall - memory problem

Richy

Hi,

I'm using the remember/recall operators during a process on a dataset (80000 rows and 40 attributes) and at the end of the process, the memory is full when I open the final exampleSet. But when I close the view of the exampleSet, the memory is still full and I have to quit/open rapidminer to get it empty.
I don't understand because when I'm working on a proccess without remember/recall operators, I don't have this problem.

I tried to use the free memory operator at the end of the process after storing the final exampleSet, but it doesn't change anything.

Would you have any idea?


Regards,
Richard.

land

Hi,
do you get an OutOfMemoryException? Otherwise RapidMiner keeps data sets as long in memory as possible. So if there's memory left, he wont discard previous results of the process / data at the port.

Greetings,
  Sebastian

dragoljub

Hi Sebastian,

Does RM discard memory after each iteration of a process. It seems from the system monitor that memory usage builds up and then does not drop to initial levels, I'm not sure if its used or allocated memory. In either case, is there any way to manually free all allocated space before starting a process up. I find myself constantly restarting RM to make this happen.

Thanks,
-Gagi

IngoRM

Hi,

I am not sure but in this particular case I would assume that no manual memory freeing would be necessary. Did you try to activate the parameter "remove from store" of the operator "Recall"? Without removing it from the store where the "Remember" operator puts the object into, the memory can never be freed.

The operator you are searching for is called "Free Memory" by the way. But as I said, this operator will only free memory which is no longer needed which is not the case if the object remains in the store.

Cheers,
Ingo

land

Hi,
you don't need to worry about the memory consumption after a process has been executed. Everything, including the Stores is freed, but only if needed! So you can take a look at your previous results in the Result History as long as they fit into the memory. RapidMiner will automatically discard all these Results if a currently running process or new result will need them.
So it is completely normal that the memory usage will constantly grow with the time until it has reached a peak value and RapidMiner starts to discard previous results.

Greetings,
  Sebastian

Richy

Hi,

Sebastian, I don't have any OutOfMemoryException error. I understand how RapidMiner is working with memory (results are automatically discarded when RapidMiner doesn't need it anymore), but when I'm using the remember/recall operator, the memory is never freed.

As Ingo suggested, I also tried to activate the parameter "remove from store", but it doesn't change anything. Every time I launch the process, the memory used increases and is never freed. Even if I clear the history and I open an other process, the memory is still full and the processes are very slow.

My only solution is to close/open RapidMiner.


Regards,
Richard.

land

Hi,
if you don't get an out of memory exception there's not much one can do about this. If you get one by re executing the process again and again until the memory is really full, please tell me immediately. Until then I doubt this is a leak, since I have taken a look in to the code when you first wrote: The Remembered object store is cleaned after the process finished.

If RapidMiner becomes verrrry slow and the hard disk works, your operating system is probably swapping. Then the complete system went out of memory and tries to free RAM. It would be a good idea then to lower the maximal amount of memory RapidMiner should use. The forum contains many posts about this.
If you are on windows you should avoid starting many memory intensive programs after having started RapidMiner, since the .exe file uses 80% of the free memory.

Greetings,
 Sebastian

Richy

Thank you for the answers Sebastian.

I'll do more tests next week end and I'll keep you updated on things.


Regards,
Richard.

frankie

Seems to me that I have similar problems*. Has there been any fixes or ideas on how to deal with remembers/recalls' memory usage?
I cannot seem to solve my issues with the suggested "Free memory" objects (exactly where should they be inserted for best performance in my loop?) and I have the "remove from store" box checked in my Recalls. The memory just fills up / doesn't free up between the loop iterations.

*) http://rapid-i.com/rapidforum/index.php/topic,3106.0.html

land

Hi,
if you somehow create new Attributes inside your loop, use the Materialize Data Operator at the start of the loop. If not, please post a process showing your problem here.

Greetings,
Sebastian