Cross Validation - Macro Averaged Precision Calculation

pipaso

I believe that the Macro-Averaged Precision is calculated by averaging each precision score obtained at each fold. However, averaging gives different results in my case. What could be the problem?

David_A

Hi @pipaso ,

perhaps this answer by Ingo helps you to understand how the macro average is calculated:

https://community.rapidminer.com/discussion/comment/13873#Comment_13873

Best,
David

pipaso

Thanks @David_A,

I have already read that answer. I have done the same thing that the answer suggested, but the results are different. For example, the average precision must be (0.8516916+0.8314766+0.8547022)/3 = 0.845957. However, it is 0.8444077 in rapidminer as shown in the image. The same results apply for kappa and recall, as well. What is the problem? By the way, there is no mistake for mikro result.

sgenzer

@pipaso have you tried checking the "use local random seed" box? It's possible you're getting different results due to different random seeds. You can use this feature to keep this constant.

pipaso

@sgenzer I have already tried that, but I do not think it is connected to this issue. There is something wrong with the rapidminer macro-average calculation code. Everyone can try a simple example to see that the results are different.

sgenzer

ok thx @pipaso let's push this to Engineering and see what they say

IngoRM

Thanks for reporting this but I am afraid that I cannot confirm this problem.  Check out this process below - the average of the single folds and the value delivered by the cross validation are the same for me...

Please submit a process using one of the sample data sets which is showing the problem and we can check again.

Thanks for your help,
Ingo

<?xml version="1.0" encoding="UTF-8"?><process version="9.6.000-SNAPSHOT">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.6.000-SNAPSHOT" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="UTF-8"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.6.000-SNAPSHOT" expanded="true" height="68" name="Retrieve Sonar" width="90" x="45" y="34">
        <parameter key="repository_entry" value="//Samples/data/Sonar"/>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="9.6.000-SNAPSHOT" expanded="true" height="145" name="Cross Validation" width="90" x="179" y="34">
        <parameter key="split_on_batch_attribute" value="false"/>
        <parameter key="leave_one_out" value="false"/>
        <parameter key="number_of_folds" value="3"/>
        <parameter key="sampling_type" value="automatic"/>
        <parameter key="use_local_random_seed" value="false"/>
        <parameter key="local_random_seed" value="1992"/>
        <parameter key="enable_parallel_execution" value="true"/>
        <process expanded="true">
          <operator activated="true" class="naive_bayes" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Naive Bayes" width="90" x="45" y="34">
            <parameter key="laplace_correction" value="true"/>
          </operator>
          <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
          <connect from_op="Naive Bayes" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
            <list key="application_parameters"/>
            <parameter key="create_view" value="false"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Performance" width="90" x="179" y="34">
            <parameter key="main_criterion" value="first"/>
            <parameter key="accuracy" value="true"/>
            <parameter key="classification_error" value="false"/>
            <parameter key="kappa" value="false"/>
            <parameter key="weighted_mean_recall" value="false"/>
            <parameter key="weighted_mean_precision" value="false"/>
            <parameter key="spearman_rho" value="false"/>
            <parameter key="kendall_tau" value="false"/>
            <parameter key="absolute_error" value="false"/>
            <parameter key="relative_error" value="false"/>
            <parameter key="relative_error_lenient" value="false"/>
            <parameter key="relative_error_strict" value="false"/>
            <parameter key="normalized_absolute_error" value="false"/>
            <parameter key="root_mean_squared_error" value="false"/>
            <parameter key="root_relative_squared_error" value="false"/>
            <parameter key="squared_error" value="false"/>
            <parameter key="correlation" value="false"/>
            <parameter key="squared_correlation" value="false"/>
            <parameter key="cross-entropy" value="false"/>
            <parameter key="margin" value="false"/>
            <parameter key="soft_margin_loss" value="false"/>
            <parameter key="logistic_loss" value="false"/>
            <parameter key="skip_undefined_labels" value="true"/>
            <parameter key="use_example_weights" value="true"/>
            <list key="class_weights"/>
          </operator>
          <operator activated="true" class="log" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Log" width="90" x="313" y="34">
            <list key="log">
              <parameter key="single_folds" value="operator.Performance.value.accuracy"/>
            </list>
            <parameter key="sorting_type" value="none"/>
            <parameter key="sorting_k" value="100"/>
            <parameter key="persistent" value="false"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="log_to_data" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Log to Data" width="90" x="179" y="238">
        <parameter key="log_name" value="Log"/>
      </operator>
      <operator activated="true" class="aggregate" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Aggregate" width="90" x="313" y="238">
        <parameter key="use_default_aggregation" value="false"/>
        <parameter key="attribute_filter_type" value="all"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="default_aggregation_function" value="average"/>
        <list key="aggregation_attributes">
          <parameter key="single_folds" value="average"/>
        </list>
        <parameter key="group_by_attributes" value=""/>
        <parameter key="count_all_combinations" value="false"/>
        <parameter key="only_distinct" value="false"/>
        <parameter key="ignore_missings" value="true"/>
      </operator>
      <connect from_op="Retrieve Sonar" from_port="output" to_op="Cross Validation" to_port="example set"/>
      <connect from_op="Cross Validation" from_port="performance 1" to_port="result 1"/>
      <connect from_op="Log to Data" from_port="exampleSet" to_op="Aggregate" to_port="example set input"/>
      <connect from_op="Aggregate" from_port="example set output" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="147"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

pipaso

@IngoRM thank you very much for your answer. I have found my mistake, but do not now what it means. If you check the screenshots in my previous message, the problem results from the Multiply (2) object in the third screenshot. I erased it and the problem was solved. I noticed that my example set includes 2400 examples. However, the confusion matrix that I get from my design without erasing the object includes much more examples. I still do not know how "multiply (2)" object caused such a problem. 

IngoRM

Hi,

Thanks for coming back.  I am not sure if it is really the multiply TBH.  But it seems that the connection to the second "per" port inside the cross validation is triggering the problem.  Below is a minimal process showing the problem you are describing: if you connect the second "per" port, the average seems to be wrong.  If you disconnect it though (even with still using the multiply inside), then the problem goes away and the averages are correct.

I must admit that I do not have any idea what is causing this - so our engineering team should have a deeper look then...

Thanks again for reporting this and diving deeper.

Best,
Ingo

<?xml version="1.0" encoding="UTF-8"?><process version="9.6.000-SNAPSHOT">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.6.000-SNAPSHOT" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="UTF-8"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.6.000-SNAPSHOT" expanded="true" height="68" name="Retrieve Sonar" width="90" x="45" y="34">
        <parameter key="repository_entry" value="//Samples/data/Sonar"/>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="9.6.000-SNAPSHOT" expanded="true" height="166" name="Cross Validation" width="90" x="179" y="34">
        <parameter key="split_on_batch_attribute" value="false"/>
        <parameter key="leave_one_out" value="false"/>
        <parameter key="number_of_folds" value="3"/>
        <parameter key="sampling_type" value="automatic"/>
        <parameter key="use_local_random_seed" value="false"/>
        <parameter key="local_random_seed" value="1992"/>
        <parameter key="enable_parallel_execution" value="true"/>
        <process expanded="true">
          <operator activated="true" class="naive_bayes" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Naive Bayes" width="90" x="45" y="34">
            <parameter key="laplace_correction" value="true"/>
          </operator>
          <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
          <connect from_op="Naive Bayes" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
            <list key="application_parameters"/>
            <parameter key="create_view" value="false"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Performance" width="90" x="179" y="34">
            <parameter key="main_criterion" value="first"/>
            <parameter key="accuracy" value="true"/>
            <parameter key="classification_error" value="false"/>
            <parameter key="kappa" value="false"/>
            <parameter key="weighted_mean_recall" value="false"/>
            <parameter key="weighted_mean_precision" value="false"/>
            <parameter key="spearman_rho" value="false"/>
            <parameter key="kendall_tau" value="false"/>
            <parameter key="absolute_error" value="false"/>
            <parameter key="relative_error" value="false"/>
            <parameter key="relative_error_lenient" value="false"/>
            <parameter key="relative_error_strict" value="false"/>
            <parameter key="normalized_absolute_error" value="false"/>
            <parameter key="root_mean_squared_error" value="false"/>
            <parameter key="root_relative_squared_error" value="false"/>
            <parameter key="squared_error" value="false"/>
            <parameter key="correlation" value="false"/>
            <parameter key="squared_correlation" value="false"/>
            <parameter key="cross-entropy" value="false"/>
            <parameter key="margin" value="false"/>
            <parameter key="soft_margin_loss" value="false"/>
            <parameter key="logistic_loss" value="false"/>
            <parameter key="skip_undefined_labels" value="true"/>
            <parameter key="use_example_weights" value="true"/>
            <list key="class_weights"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="9.6.000-SNAPSHOT" expanded="true" height="103" name="Multiply" width="90" x="313" y="136"/>
          <operator activated="true" class="log" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Log" width="90" x="447" y="34">
            <list key="log">
              <parameter key="single_folds" value="operator.Performance.value.accuracy"/>
            </list>
            <parameter key="sorting_type" value="none"/>
            <parameter key="sorting_k" value="100"/>
            <parameter key="persistent" value="false"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Multiply" from_port="output 2" to_port="performance 2"/>
          <connect from_op="Log" from_port="through 1" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="84"/>
          <portSpacing port="sink_performance 3" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="log_to_data" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Log to Data" width="90" x="179" y="238">
        <parameter key="log_name" value="Log"/>
      </operator>
      <operator activated="true" class="aggregate" compatibility="9.6.000-SNAPSHOT" expanded="true" height="82" name="Aggregate" width="90" x="313" y="238">
        <parameter key="use_default_aggregation" value="false"/>
        <parameter key="attribute_filter_type" value="all"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="default_aggregation_function" value="average"/>
        <list key="aggregation_attributes">
          <parameter key="single_folds" value="average"/>
        </list>
        <parameter key="group_by_attributes" value=""/>
        <parameter key="count_all_combinations" value="false"/>
        <parameter key="only_distinct" value="false"/>
        <parameter key="ignore_missings" value="true"/>
      </operator>
      <connect from_op="Retrieve Sonar" from_port="output" to_op="Cross Validation" to_port="example set"/>
      <connect from_op="Cross Validation" from_port="performance 1" to_port="result 1"/>
      <connect from_op="Cross Validation" from_port="performance 2" to_port="result 2"/>
      <connect from_op="Log to Data" from_port="exampleSet" to_op="Aggregate" to_port="example set input"/>
      <connect from_op="Aggregate" from_port="example set output" to_port="result 3"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="63"/>
      <portSpacing port="sink_result 3" spacing="84"/>
      <portSpacing port="sink_result 4" spacing="0"/>
    </process>
  </operator>
</process>

pipaso

@IngoRM I hope the team will find the problem. I am happy to help. Thanks again.