How to read Pairwise t-Test RapidMiner Studio

ra_aulia_rachma

Is anyone can help me to explain how to read the result of T-Test Perfomance? Which is the best method from this pic?

IngoRM

Hi,

 

Are small performance values good or large ones?  Let's assume small ones are good (i.e. the performance values are errors, not accuracies).  In this case, the second performance (0.32...) would be the best.  And not only that, but it is also a result which is significantly better than the other two performances.

 

Here is why: your three performances are 0.38, 0.32, and 0.41.  Those are shown in the first column of the table as well as in the first row of the table (ignore the header row with the A, B, C, D please).  If small values are better (again, that depends on the chosen performance measurement and I can't see it this from the picture), obviously 0.32 is the best result since it is the smallest value.  But since there are possible variations of the result (indicated by the standard deviation shown after the +/- sign), it could well be that this result is actually not really better but just looks better thanks to random variation during the validation.  The t-Test now calculates the probabilities that the difference in values is an actual difference and not just the result of this random variation.

 

Let's see: for the second row / column, i.e. the one we care for most since it looks as the most promising candidate for the best performance, we can see that the likelihood to be just a difference as an effect of random fluctuations is actually 3.3% compared to the first model with 0.38 performance and even less, namely 1.3% compared to the third performance of 0.41.  Since those probabilities are smaller than the common threshold of 5% we accept that this model is actually significantly better than the other two.

 

The argument is similar if high values are better.  In this case, the third performance (0.41) looks best at first.  And in fact, it is significantly better than the second of only 0.32 but it is not significantly better than the first performance of 0.38.  You can see that the probability that those two values actually only look different is roughly 20% so you can't be sure.

 

Probabilities lower than 5% are marked with the yellow color.

 

By the way, I have created a small example process (pasted below) in case somebody else wants to test this.

 

Hope this helps,

Ingo

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.3.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.3.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.3.001" expanded="true" height="68" name="Retrieve Sonar" width="90" x="45" y="34">
        <parameter key="repository_entry" value="//Samples/data/Sonar"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="7.3.001" expanded="true" height="124" name="Multiply" width="90" x="179" y="34"/>
      <operator activated="true" class="concurrency:cross_validation" compatibility="7.3.001" expanded="true" height="145" name="Cross Validation (3)" width="90" x="313" y="340">
        <process expanded="true">
          <operator activated="true" class="naive_bayes" compatibility="7.3.001" expanded="true" height="82" name="Naive Bayes" width="90" x="45" y="34"/>
          <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
          <connect from_op="Naive Bayes" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.3.001" expanded="true" height="82" name="Apply Model (3)" width="90" x="45" y="34">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="7.3.001" expanded="true" height="82" name="NB Performance" width="90" x="246" y="34"/>
          <connect from_port="model" to_op="Apply Model (3)" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model (3)" to_port="unlabelled data"/>
          <connect from_op="Apply Model (3)" from_port="labelled data" to_op="NB Performance" to_port="labelled data"/>
          <connect from_op="NB Performance" from_port="performance" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="7.3.001" expanded="true" height="145" name="Cross Validation (2)" width="90" x="313" y="187">
        <process expanded="true">
          <operator activated="true" class="h2o:gradient_boosted_trees" compatibility="7.3.000" expanded="true" height="103" name="Gradient Boosted Trees" width="90" x="45" y="34">
            <list key="expert_parameters"/>
          </operator>
          <connect from_port="training set" to_op="Gradient Boosted Trees" to_port="training set"/>
          <connect from_op="Gradient Boosted Trees" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.3.001" expanded="true" height="82" name="Apply Model (2)" width="90" x="45" y="34">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="7.3.001" expanded="true" height="82" name="GBT Performance" width="90" x="246" y="34"/>
          <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
          <connect from_op="Apply Model (2)" from_port="labelled data" to_op="GBT Performance" to_port="labelled data"/>
          <connect from_op="GBT Performance" from_port="performance" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="7.3.001" expanded="true" height="145" name="Cross Validation" width="90" x="313" y="34">
        <process expanded="true">
          <operator activated="true" class="parallel_decision_tree" compatibility="7.3.001" expanded="true" height="82" name="Decision Tree" width="90" x="45" y="34"/>
          <connect from_port="training set" to_op="Decision Tree" to_port="training set"/>
          <connect from_op="Decision Tree" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.3.001" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="7.3.001" expanded="true" height="82" name="DT Performance" width="90" x="246" y="34"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="DT Performance" to_port="labelled data"/>
          <connect from_op="DT Performance" from_port="performance" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="t_test" compatibility="7.3.001" expanded="true" height="145" name="T-Test" width="90" x="447" y="34"/>
      <operator activated="true" class="model_management:model_management_key" compatibility="7.3.000" expanded="true" height="103" name="Compare Models" width="90" x="715" y="187">
        <process expanded="true">
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test data" spacing="0"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve Sonar" from_port="output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Cross Validation" to_port="example set"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Cross Validation (2)" to_port="example set"/>
      <connect from_op="Multiply" from_port="output 3" to_op="Cross Validation (3)" to_port="example set"/>
      <connect from_op="Cross Validation (3)" from_port="performance 1" to_op="T-Test" to_port="performance 3"/>
      <connect from_op="Cross Validation (2)" from_port="performance 1" to_op="T-Test" to_port="performance 2"/>
      <connect from_op="Cross Validation" from_port="performance 1" to_op="T-Test" to_port="performance 1"/>
      <connect from_op="T-Test" from_port="significance" to_port="result 1"/>
      <connect from_op="T-Test" from_port="performance 1" to_port="result 2"/>
      <connect from_op="T-Test" from_port="performance 2" to_port="result 3"/>
      <connect from_op="T-Test" from_port="performance 3" to_port="result 4"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
    </process>
  </operator>
</process>