"Text Mining On rapid miner (Dealing with Smileys)"

maccten

Hi,

I have a selection of emails which i wish to tokenize and then apply a custom dictionary to stem
The emails have many many smiley faces and other unorthodox symbols to convey emotions
I was wondering is there a way to use the Stem (Dictionary) operator to achieve this?
For example set in my text file (dictionary) Smile:
I have tried this but it keeps falling over. Could you suggest any changes i could implement for my dictionary

Thank you for your time

MariusHelf

Hi,

probably your Tokenize operator is configured to cut on anything but letters and numbers, which means that any punctuation marks (from which smileys are composed) will be ignored. Please be sure to define proper tokenization rules. If that does not help, please post your process xml as described in the post linked from my signature.

Best regards,
Marius

maccten

Hi Marius

Sorry to bring this topic up again. I just have a quick question on rapid miner along the same topic.
Again i have around 1000 documents which contain text speak in each document
I have created a dictionary for text speak and smileys and it sits in a MYSQL table with a definition format like that in the table below

Smiling, Happy

Is there an operator in rapid miner that will allow me to find exactly the text speak (based on the column in the table i created) and convert it to the normalized English in the second column in the table

Thanks for your time

MariusHelf

Hi, the easiest way to do that is with the Replace (Dictionary) operator. Put it into your process before the Process Documents operator.
Just connect the example set containing the documents to the exa input port. Then load the dictionary with Read Database and connect it to the dic input. Use the from_attribute parameter to declare the first column as search string and to_attribute to define the other column as replacement string.

Best regards,
Marius

maccten

Hi Marius

This is very good but unfortunately if i have a sentence that contains the word URGENT and the dictionary specifies that U is equal to YOU then i get  YOUrgent
Is there a way to just maybe look at the words individually?
Would it make more sense to tokenize the Documents first using a space as a delimiter?

Thanks for your time Marius
Its really appreciated

MariusHelf

You can also try to tokenize first, then, however, you won't replace smileys which are attached directly to a word... however, after tokenization you can use the Stem (Dictionary) operator.

Best regards,
Marius