The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Radoop Read Database - Out of Memory
JEdward
RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 578 Unicorn
Answers
To assist if anyone is able, here is a sample log file.
Basic gist is:
After some gubbins Radoop begins to execute the query.
2 mins later the memory usage on Server begins to rise.
4 later the memory runs out and an OOM flag is created
4 mins later the process halts due to an OOM error.
The table size is 14GB and appears to run without any significant memory rise on my local laptop. It is only on the Server where memory usage does this. (Bit awkward when wanting to execute it overnight) -- EDIT: mistake on my side. This is also occuring on the laptop too, but is much slower as it takes longer to run the query and begin fetching all rows into memory. --
Other tables have so far been much smaller. One table was a large size (5 GB), and trigged an OOM flag on the server execution, but still managed to continue and successfully stream the data despite this. It is just this 14GB table that is causing issues.
hi @JEdward - sorry some of the Radoop folks are on vacation at the moment (and this is out of my league!). I will see if I can track somebody down for you.
Scott
Hi,
That's not the expected behaviour, for sure. We'll try to reproduce on our side and we'll get back to you.
Jesús
Could you please send us your process xml? That would help a lot in the reproduction.
Thanks,
Attila
Not really much exciting to see.
Please note, I changed the connection names for posting. I am using Radoop BETA here, but the same occurred in the older version too. Database is moving from an AWS RDS MySQL database into a cluster hosted outside AWS. Could this be something to do with it?
Any luck replicating this error? It would be great if it is something simple that can be resolved with the RM 9 release.
I'm stumped this end as the error logs don't give me any clues.
@asimon @jpuente
Were you able to reproduce this on your side?
I've just upgraded from RM Server 8.2 to RM Server 9.01 and can confirm it is still happening on my side.
My "workaround" of reading in snapshots of the database and then combining them all at the end is rather cumbersome.
I would expect the database to open as a stream, but it doesn't seem to want to... just wants to load it all into memory.