The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

tfidf and entropy

rafeenarafeena Member Posts: 14 Contributor II
edited December 2019 in Help
hi i would like to do a feature selection using tfidf and entropy .my question are as below

1. i would like to know do i need to use the generate tfidf or is the tfidf in word vector is enough?
2. for entropy do i use weight by information gain of discretize by entropy

Best Answer

  • jmerglerjmergler Employee-RapidMiner, RapidMiner Certified Analyst, Member, University Professor Posts: 41 Guru
    Solution Accepted
    Hi @rafeena,
    I'm not entirely certain I understand the question, but I think I can help. 
    1. There are many valid ways of getting the TFIDF scores. You do not need to use any particular operator like Generate TFIDF. I think what you are looking for is a data structure with words as attributes and TFIDF scores as values. If you have that, then it does not matter how you got it. If you do not have that, then please share a sample of what you do have so we can better help.
    2. Your goal is feature selection, so the Weight by Information Gain operator would work. This will score the attributes. After getting the weights you may consider using the Select by Weights operator. 
    Please consider sharing your process with sample data, so we can provide more precise help if you need it.
    Jeff

Answers

  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    hi @rafeena I'm sorry no one has chimed in here. Is this still an issue? If so I can try to find someone internal who may know the answer.

    Scott
  • rafeenarafeena Member Posts: 14 Contributor II
    Hi. Unfortunately it is still an issue, i really want to understand the differences of the methods available... Hope you can help.. Thank you..
  • rafeenarafeena Member Posts: 14 Contributor II

    hi jmergler, thanks for your reply.my project is about cyberbullying detection using term weighting scheme and i planned to use tfidf and entropy. i have attached my sample data and i share my tfidf process first. put put a screenshot of what i did  
Sign In or Register to comment.