The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
importing large XLS OR CSV dataset issue
Contributor I
Hi everyone!
I am just starting out with Rapidmier and I must say i'm really impressed by it.
I do however encounter an issue when i try to import my dataset.
the dataset is a 100.000 examples, 80 attributes xls file (about 16 MB).
It imports fine but gets stuck when rapidminer stries to show the" ExampleSet" results output.
Apparently the file is too big to actually view?
I already updated my java to 64bit (on my 64bit windows 8 OS). I have 12GB ram and changed the RapidMinerGUI.bat file so it allows to take 8GB of RAM instead of the default 512MB.
Is there a possibility to turn off viewing the dataset in the results? Or is it something else I should take into account to run this dataset?
I tried importing the same dataset but just the first 10.000 rows, and this went smoothly so I am quiet convinced it must by the amount of data which makes
rapidminer "hang".
Does anyone here have some experience with datasets this large and have some hints for me?
thanks in advance,
Arno
I am just starting out with Rapidmier and I must say i'm really impressed by it.
I do however encounter an issue when i try to import my dataset.
the dataset is a 100.000 examples, 80 attributes xls file (about 16 MB).
It imports fine but gets stuck when rapidminer stries to show the" ExampleSet" results output.
Apparently the file is too big to actually view?
I already updated my java to 64bit (on my 64bit windows 8 OS). I have 12GB ram and changed the RapidMinerGUI.bat file so it allows to take 8GB of RAM instead of the default 512MB.
Is there a possibility to turn off viewing the dataset in the results? Or is it something else I should take into account to run this dataset?
I tried importing the same dataset but just the first 10.000 rows, and this went smoothly so I am quiet convinced it must by the amount of data which makes
rapidminer "hang".
Does anyone here have some experience with datasets this large and have some hints for me?
thanks in advance,
Arno
0
Answers
in RapidMiner, please have a look at the System Monitor view. How much memory does it consume when you try to load the big data set? How much when loading the small one?
If you do not want to display the result, just don't connect the output of the last operator to the process output on the far right. If you only want to display parts of it, please use a Sample operator to reduce the size of the data.
It is however strange if the data really loads fine, but does not display.... we'll have a look at that.
Best regards,
Marius