The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Arabic Light Stemming a CSV file

NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
edited March 2023 in Help
I have a CSV file with around 4000 rows of text. I want to use the Arabic Light Stemmer to stem each record.

I have done the following but the text is not being stemmed. The output is the same as the input.


and inside the Process

Tagged:

Answers

  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Hi!

    To stem words, first you need words. Use Tokenize before Stem to split the text into words.

    Regards,

    Balázs
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @BalazsBarany

    I did the following



    inside the Process, but the output is still exactly the same as the input.

    Is there a problem with reading Arabic text?

    I specified the Encoding method to be UTF-8 when i imported the CSV file. Is there anything else I should do?
  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Hi,

    put a breakpoint after on Tokenize and play with the settings. If you see the words in different colors, the tokenization is working correctly.

    I have no idea about the conventions with Arabic text, maybe a different word separator is necessary etc.

    If the text looks normal to you in RapidMiner, then the encoding is correct. You would see that it is broken with a wrong encoding.

    Regards,

    Balázs
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @BalazsBarany

    I am facing this issue, what could be a possible reason?


  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Hi,

    you need to use Nominal to Text before Process Documents in order to mark your nominal attributes as text (suitable for the Text Processing operators).

    Regards,
    Balázs
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @BalazsBarany

    When I put a break point after Stem, I can see the correctly stemmed sentence. But the final output in the Results is like the following. What can I do to fix this? I want the output to be rows of the stemmed sentence.


  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Hi!

    Use the "keep text" option that all "Process Documents" operators have. 

    The default operation mode of Process Documents is to create the wide table suitable for machine learning methods. 

    Tokenization can split your text into letters, words or sentences. Stemming works on words, at least in Western languages. 

    Regards,
    Balázs
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @BalazsBarany

    Great, that solved it. But now, when i use Write CSV, I don't get Arabic text in the output CSV file.

    I set the encoding to UTF-8 for Read CSV, Write CSV, and the process when pressing on the white canvas.

    What can I do to solve that?


  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Hi!

    Try using a software in which you can set the import encoding. Excel is not very smart when just opening a CSV file. Something with Import should also work in Excel, where you get a dialog for selecting the encoding.

    The encoding of text files is not obvious to most software. It often needs to be specified manually. You can use an advanced editor (GVim, Notepad++ etc.) to determine if the file itself is really in UTF-8.

    Regards,
    Balázs
  • jwpfaujwpfau Employee-RapidMiner, Member Posts: 303 RM Engineering
    Hi Noor,

    Excel seem to have moved the CSV Import to Data → From Text/CSV



    Greetings,
    Jonas
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @jwpfau

    Hello!

    After clicking from Text/CSV, what should I do?


  • jwpfaujwpfau Employee-RapidMiner, Member Posts: 303 RM Engineering
    Hi Noor, 

    For me the first dialog was the "Import Data" file selector, the second one the csv table preview from my screenshot.

    I fear the excel autodetection completely failed for your file, is there anything in the "Open As" menu that says csv or utf-8?

    Greetings,
    Jonas
  • NoorKhalifaNoorKhalifa Member Posts: 7 Learner I
    @jwpfau

    I didn't manage to do that in Excel, but importing the file in Notepad gave me the Arabic equivalent.

    Thanks!
  • jwpfaujwpfau Employee-RapidMiner, Member Posts: 303 RM Engineering
    Hi Noor,

    you can force csv parsing here.



    But you will stay in the more cumbersome Power Query Editor flow afterwards.

    Greetings,
    Jonas
Sign In or Register to comment.