Excessive memory usage: rapid-4-1-beta vs rapid-4-1

meb

I was wondering whether anyone else shared my experience with the latest community edition rapidminer-4.1?

For a while I was using rapidminer-4.1-beta. Then, I installed the update rapidminer-4.1 and memory usage seems excessive. My whole machine slows down and the hard drive light on solid. Task manager shows a lot of page faulting going on.

I'm using Windows XP (actually, XP is using me).

I'm considering going back to the beta version. That didn't give me problems like this.

steffen

Hello

I am also using Windows XP on a P4 with one cpu 3.06 GHz and 1 GB RAM. In my point of view, the memory usage of RapidMiner 4.1 is much better than in earlier versions. I create a lot of "iterating processes" , which stuck in earlier versions 'cause RM was not able to remove unused result objects from the memory properly (between two iteration steps). But now, the aforesaid processes run without problems.

Which kind of computer do you use ?

greetings

Steffen

TobiasMalbrecht

Hi,

I also have not the impression, that RapidMiner 4.1 is gravely consuming more memory than the previous beta version. Moreover that would be surprising since the work we currently do and have done for the last weeks is not only dedicated to the invention and implementation of new functionality for RapidMiner, but to a large extent also to the improvement of existing functions, operators, etc. due to further improve the performance and scalablity of RapidMiner. Nevertheless changes in the implementation might in some cases have side-effects such as higher memory consumption, which is unintentional in some cases or accepted in favour of other advantages (useability, functionality, ect.). However, we greatly appreciate users who report cases in which the usage of particular operators or processes and who reflect the need to either communicate some changes more clearly or point at things that can be potentially be improved.

To cut a long story short, if you experience significant losses of performance (i.e. also increased memory consumption) in particular processes or operators, please let us know. Unfortunately, a general or global statement such as yours is otherwise relatively hard to retrace.

Regards,
Tobias

IngoRM

Hello,

we made some tests on a set of 143 different data mining processes and the version 4.1 is using 22% less memory than before on average over all processes and there was no process which used more memory. Actually, a lot of work of the last months was put in the optimization with respect to memory consumption and / or runtimes (but of course a lot is still to do).

However, could it be that you changed to the 64 bit version? This version uses the double amount of memory for all internal references and hence double memory usage. So the 64 bit version is only recommended for systems with more than approx. 3,5 Gb system memory.

Cheers,
Ingo