The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Can anyone help how Rapid Miner calculates TF-IDF
subhasisdasgupt
Member Posts: 15 
Contributor II
I know how TF-IDF is to be calculated but when I tried to calculate TF-IDF manually with simple data and tried to match the same with Rapid Miner, I got different results. TF for each word is 0.5 which I also got. But how IDF value calculated is not clear to me. For example, TF of "Cat" is 0.5 but how the TF-IDF value is coming out to be 0.707 is not clear. For TF-IDF to be 0.707, the IDF value should be 0.707/0.5 = 1.414 and how this value is being calculated is the problem that I am facing. Please help.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document" width="90" x="45" y="30">
<parameter key="text" value="Ram has a dog"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (2)" width="90" x="45" y="300">
<parameter key="text" value="He has a Rabbit"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (3)" width="90" x="45" y="210">
<parameter key="text" value="Shivam has a Parrot"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (4)" width="90" x="45" y="120">
<parameter key="text" value="Ravi has a Cat"/>
</operator>
<operator activated="true" class="text:process_documents" compatibility="5.3.001" expanded="true" height="148" name="Process Documents" width="90" x="179" y="30">
<process expanded="true">
<operator activated="true" class="text:tokenize" compatibility="5.3.001" expanded="true" height="60" name="Tokenize" width="90" x="45" y="30"/>
<connect from_port="document" to_op="Tokenize" to_port="document"/>
<connect from_op="Tokenize" from_port="document" to_port="document 1"/>
<portSpacing port="source_document" spacing="0"/>
<portSpacing port="sink_document 1" spacing="0"/>
<portSpacing port="sink_document 2" spacing="0"/>
</process>
</operator>
<connect from_op="Create Document" from_port="output" to_op="Process Documents" to_port="documents 1"/>
<connect from_op="Create Document (2)" from_port="output" to_op="Process Documents" to_port="documents 4"/>
<connect from_op="Create Document (3)" from_port="output" to_op="Process Documents" to_port="documents 3"/>
<connect from_op="Create Document (4)" from_port="output" to_op="Process Documents" to_port="documents 2"/>
<connect from_op="Process Documents" from_port="example set" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document" width="90" x="45" y="30">
<parameter key="text" value="Ram has a dog"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (2)" width="90" x="45" y="300">
<parameter key="text" value="He has a Rabbit"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (3)" width="90" x="45" y="210">
<parameter key="text" value="Shivam has a Parrot"/>
</operator>
<operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (4)" width="90" x="45" y="120">
<parameter key="text" value="Ravi has a Cat"/>
</operator>
<operator activated="true" class="text:process_documents" compatibility="5.3.001" expanded="true" height="148" name="Process Documents" width="90" x="179" y="30">
<process expanded="true">
<operator activated="true" class="text:tokenize" compatibility="5.3.001" expanded="true" height="60" name="Tokenize" width="90" x="45" y="30"/>
<connect from_port="document" to_op="Tokenize" to_port="document"/>
<connect from_op="Tokenize" from_port="document" to_port="document 1"/>
<portSpacing port="source_document" spacing="0"/>
<portSpacing port="sink_document 1" spacing="0"/>
<portSpacing port="sink_document 2" spacing="0"/>
</process>
</operator>
<connect from_op="Create Document" from_port="output" to_op="Process Documents" to_port="documents 1"/>
<connect from_op="Create Document (2)" from_port="output" to_op="Process Documents" to_port="documents 4"/>
<connect from_op="Create Document (3)" from_port="output" to_op="Process Documents" to_port="documents 3"/>
<connect from_op="Create Document (4)" from_port="output" to_op="Process Documents" to_port="documents 2"/>
<connect from_op="Process Documents" from_port="example set" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>
0
Answers
in RapidMiner the TF/IDF values are normalized. Here Ingo points out how this works: http://rapid-i.com/rapidforum/index.php?topic=3728.0
Best regards,
Marius