The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

"a bug while getting the token in arabic"

RaedMarjiRaedMarji Member Posts: 13 Contributor II
edited June 2019 in Help
Hello,

Im trying to compare a word I get as a token and a word that is hardcoded in a dictionary but the result is always false. even though when printed to the log it shows the same word Im looking for I dont know what Im missing and I dont know how to debug this. I just tried to get the bytes for the token and the bytes for the preset word and they are different so why is this happening with me considering that it runs perfectly in english.

is there a flag I should set to support UTF8?
Tagged:

Answers

  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee-RapidMiner, Member, University Professor Posts: 1,996 RM Engineering
    Hi,

    the description is a bit vague. Did you set your process encoding to UTF-8 (click the empty white space of your process in the RapidMiner GUI and select UTF-8 as encoding parameter.

    Regards,
    Marco
  • RaedMarjiRaedMarji Member Posts: 13 Contributor II
    I tried what you said but I dont think the problem is in the encoding of the process as i used the remove stop words operator and it gave the expected results but when I use my own operator which is very similar to the remove stopwords in arabic operator it doesnt detect the tokens coming from the token sequence. so i was wondering if somthing that needs to be set in my java code to make this work.

    a sample usecase
    if (token.contains("arabic_word") then remove it

    when I print the token text it show the same value as the "arabic_word" but they are not equal when they are compared in the operator.

    this problem is specific to rapidminer because I wrote my code first in java and it worked just find but when I changed the source of the word from the tokens it started to give wrong results.
Sign In or Register to comment.