The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
"Segmenting millions of text segments with Textsegmenter"
Contributor II
Hi there,
I am building a text mining process that should be able to process various xml-files with following properties:
Each XML file contains several thousands blogposts and some information to each post (author, time, etc.).
My question: Is there a way to process this file, taking in account the segments but not necessarily dividing this file in millions of other files like the TextSegmenter does.
My assumption: It will take ages to process millions of files to mine for knowledge or do sentiment analysis
Any help will be greatly appreciated.
Thanks
Braulio
I am building a text mining process that should be able to process various xml-files with following properties:
Each XML file contains several thousands blogposts and some information to each post (author, time, etc.).
My question: Is there a way to process this file, taking in account the segments but not necessarily dividing this file in millions of other files like the TextSegmenter does.
My assumption: It will take ages to process millions of files to mine for knowledge or do sentiment analysis
Any help will be greatly appreciated.
Thanks
Braulio
Tagged:
0
Answers
if there is a clear splitting criterion in your original texts, you could load the text as the content of a single attribute, use the new split operator which will deliver one attribute per text (this can actually lead to memory problems since meta data for attributes is rather costly in terms of memory). After that, you could transpose the example set and work with the StringTextInput operator.
Cheers,
Ingo