The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Replacing special characters when text mining
Guru
Hi
I am attempting to replace special characters inside of a dataset but not having much luck.
André Shoémaker
Adrié Spéllman
It seems as though RapidMiner is unable to understand those characters and is simply marking them as �. Non of the replace operators can seem to fix this issue.
Should I be doing this inside of a generate attributes operator to account for this or is there a better way?
I am attempting to replace special characters inside of a dataset but not having much luck.
André Shoémaker
Adrié Spéllman
It seems as though RapidMiner is unable to understand those characters and is simply marking them as �. Non of the replace operators can seem to fix this issue.
Should I be doing this inside of a generate attributes operator to account for this or is there a better way?
Tagged:
0
Best Answers
-
sgenzer
Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 
Community Manager
hi @robin yep seems like an encoding issue. Try changing the settings:
Unfortunately there are a pile of options there (way too many IMHO)...my go-to ones that I try when I get that problem are ISO-8859-1, UTF-8, or windows-1250. If none of those work, I move on to ASCII (yuck) and so on. Sometimes you can look at the original data source and find out the encoding from there...but not always.
Scott
5 -
rfuentealba
RapidMiner Certified Analyst, Member, University Professor Posts: 568 
Unicorn
Hi @robin,
If you are using UNIX (Linux/macOS), you may be able to execute the following command: file -I filename.csv. I took a screenshot for you:
In the charset= field, you have the encoding you can use, then RapidMiner will save you from the hassle of converting characters. At least in my case (Chilean Spanish, Portuguese, Swiss German and sometimes struggling with EBCDIC because that's what the sensors I read at work send), this has been quite helpful.
All the best,
Rodrigo.5
Answers
Where is your dataset coming from? Operators like Read CSV that import external data have an "encoding" parameter. If you set the encoding correctly for your data import, you'll see all special characters.
Regards,
Balázs