The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Memory usage of Read CSV operator seems excessive
tennenrishin
Member Posts: 177 
Contributor II
It would seem like Read CSV uses far more memory than necessary, which drastically reduces the maximum size of files that it can read on a given machine.
This suspicion is motivated by the fact that a CSV file that is too large to read on a given machine with Read CSV, can very easily be read on that same machine by splitting the file longitudinally (with a common ID field in each part), reading and storing each part in the repository (using Read CSV), and finally reading and joining the parts from the repository.
If the Read CSV were to reuse memory internally, then RapidMiner would (for a given machine) be able to import much larger input data sets directly from CSV files without using the above workaround. This seems like an important constraint for an application such as RM.
Regards,
Isak
This suspicion is motivated by the fact that a CSV file that is too large to read on a given machine with Read CSV, can very easily be read on that same machine by splitting the file longitudinally (with a common ID field in each part), reading and storing each part in the repository (using Read CSV), and finally reading and joining the parts from the repository.
If the Read CSV were to reuse memory internally, then RapidMiner would (for a given machine) be able to import much larger input data sets directly from CSV files without using the above workaround. This seems like an important constraint for an application such as RM.
Regards,
Isak
0
