The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Parameter Optimization, gives different results?

petergao0528petergao0528 Member Posts: 6 Contributor II
edited November 2018 in Help
I was trying to use the grid parameter optimization to determine the optimal learning rate and momentum for a neural network classifier, I logged the AUC value as the model selection criteria, inside the parameter optimization operator I have 10-fold cross validation.

But I found out that if I run with the optimized parameter settings I got from above, just with 10-fold cross validation, the AUC value is different from the one obtained during parameter optimization. I didn't use any local random seed. The cross-validation operator is exactly the same. How could this happen?

Best regards,

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    You absolutely have to set a local random seed in the X-Validation. Otherwise the splits will be different on each execution, leading to (hopefully small) differences in the output.

    Best regards,
    Marius
  • petergao0528petergao0528 Member Posts: 6 Contributor II
    Hi, Thanks for your input.

    I used the same local random seed, but the results for AUC are still different, and not by a small amount but rather large.
    The AUC obtained from parameter optimization and logged using the log operator is higher than the one I got from running x-validation with the same parameter settings by 0.05 to 0.1 which is definitely wrong.

    I can replicate the results exactly the same by running the parameter optimization again they are consistent and running x-validation with the same parameter settings also have consistent results itself. It's just the two don't conincide with each other. This is really bugging me. Can anyone help and give some advice on this?
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Can you post a simple example process, if possible using one of the example data sets such that we can reproduce the behavior?

    Best regards,
    Marius
  • petergao0528petergao0528 Member Posts: 6 Contributor II
    I tried with the sonar dataset and still have the same problem.

    First, I run an experiment using parameter optimization to find the optimal learning rate for a neural network. For example, use learning rate starting from 0.1 to 1 in 10 steps, other parameters for neural network remain default. Use 10-fold stratified cross validation to get the accuracy of the classification performance. Set local random seed for both cross validation and neural network to 1992. And say for example,  I get accuracy of 0.667 for learning rate 0.1 in this experiment set up.

    Second, if I run the experiment without parameter optimization but only crossvalidation with neural network with learning rate 0.1 and other parameters remain default. Use the same local random seed for both cross validation and neural network. I got the accuracy of 0.6981.

    I thought these two numbers should be exactly the same. But they are not and they're off quite a lot in my point of view. I don't know if the parameter optimization process is reliable or not. This also happens when I try to select the features when looping through all possible subset of features.

    The two experiments are the following.
    First experiment 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.015">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.015" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="retrieve" compatibility="5.3.015" expanded="true" height="60" name="Retrieve Sonar" width="90" x="112" y="165">
            <parameter key="repository_entry" value="//Samples/data/Sonar"/>
          </operator>
          <operator activated="true" class="optimize_parameters_grid" compatibility="5.3.015" expanded="true" height="94" name="Optimize Parameters (Grid)" width="90" x="514" y="210">
            <list key="parameters">
              <parameter key="Neural Net.learning_rate" value="[0.1;1.0;10;linear]"/>
            </list>
            <process expanded="true">
              <operator activated="true" class="x_validation" compatibility="5.3.015" expanded="true" height="112" name="Validation" width="90" x="112" y="120">
                <parameter key="use_local_random_seed" value="true"/>
                <process expanded="true">
                  <operator activated="true" class="neural_net" compatibility="5.3.015" expanded="true" height="76" name="Neural Net" width="90" x="112" y="75">
                    <list key="hidden_layers"/>
                    <parameter key="learning_rate" value="1.0"/>
                    <parameter key="shuffle" value="false"/>
                    <parameter key="normalize" value="false"/>
                    <parameter key="use_local_random_seed" value="true"/>
                  </operator>
                  <connect from_port="training" to_op="Neural Net" to_port="training set"/>
                  <connect from_op="Neural Net" from_port="model" to_port="model"/>
                  <portSpacing port="source_training" spacing="0"/>
                  <portSpacing port="sink_model" spacing="0"/>
                  <portSpacing port="sink_through 1" spacing="0"/>
                </process>
                <process expanded="true">
                  <operator activated="true" class="apply_model" compatibility="5.3.015" expanded="true" height="76" name="Apply Model" width="90" x="179" y="120">
                    <list key="application_parameters"/>
                  </operator>
                  <operator activated="true" class="performance_classification" compatibility="5.3.015" expanded="true" height="76" name="Performance" width="90" x="313" y="120">
                    <list key="class_weights"/>
                  </operator>
                  <connect from_port="model" to_op="Apply Model" to_port="model"/>
                  <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
                  <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
                  <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
                  <portSpacing port="source_model" spacing="0"/>
                  <portSpacing port="source_test set" spacing="0"/>
                  <portSpacing port="source_through 1" spacing="0"/>
                  <portSpacing port="sink_averagable 1" spacing="0"/>
                  <portSpacing port="sink_averagable 2" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="log" compatibility="5.3.015" expanded="true" height="76" name="Log" width="90" x="380" y="165">
                <list key="log">
                  <parameter key="LearningRate" value="operator.Neural Net.parameter.learning_rate"/>
                  <parameter key="Accuracy" value="operator.Performance.value.accuracy"/>
                </list>
              </operator>
              <connect from_port="input 1" to_op="Validation" to_port="training"/>
              <connect from_op="Validation" from_port="averagable 1" to_op="Log" to_port="through 1"/>
              <connect from_op="Log" from_port="through 1" to_port="performance"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_performance" spacing="0"/>
              <portSpacing port="sink_result 1" spacing="0"/>
            </process>
          </operator>
          <operator activated="false" class="x_validation" compatibility="5.3.015" expanded="true" height="112" name="Validation (2)" width="90" x="380" y="30">
            <parameter key="use_local_random_seed" value="true"/>
            <process expanded="true">
              <operator activated="true" class="neural_net" compatibility="5.3.015" expanded="true" height="76" name="Neural Net (2)" width="90" x="221" y="30">
                <list key="hidden_layers"/>
                <parameter key="learning_rate" value="0.1"/>
                <parameter key="shuffle" value="false"/>
                <parameter key="normalize" value="false"/>
                <parameter key="use_local_random_seed" value="true"/>
              </operator>
              <connect from_port="training" to_op="Neural Net (2)" to_port="training set"/>
              <connect from_op="Neural Net (2)" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="5.3.015" expanded="true" height="76" name="Apply Model (2)" width="90" x="45" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="5.3.015" expanded="true" height="76" name="Performance (2)" width="90" x="336" y="30">
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
              <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Retrieve Sonar" from_port="output" to_op="Optimize Parameters (Grid)" to_port="input 1"/>
          <connect from_op="Optimize Parameters (Grid)" from_port="performance" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    Second experiment 

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.015">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.015" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="retrieve" compatibility="5.3.015" expanded="true" height="60" name="Retrieve Sonar" width="90" x="112" y="165">
            <parameter key="repository_entry" value="//Samples/data/Sonar"/>
          </operator>
          <operator activated="false" class="optimize_parameters_grid" compatibility="5.3.015" expanded="true" height="94" name="Optimize Parameters (Grid)" width="90" x="514" y="210">
            <list key="parameters">
              <parameter key="Neural Net.learning_rate" value="[0.1;1.0;10;linear]"/>
            </list>
            <process expanded="true">
              <operator activated="true" class="x_validation" compatibility="5.3.015" expanded="true" height="112" name="Validation" width="90" x="112" y="120">
                <parameter key="use_local_random_seed" value="true"/>
                <process expanded="true">
                  <operator activated="true" class="neural_net" compatibility="5.3.015" expanded="true" height="76" name="Neural Net" width="90" x="112" y="75">
                    <list key="hidden_layers"/>
                    <parameter key="learning_rate" value="1.0"/>
                    <parameter key="shuffle" value="false"/>
                    <parameter key="normalize" value="false"/>
                    <parameter key="use_local_random_seed" value="true"/>
                  </operator>
                  <connect from_port="training" to_op="Neural Net" to_port="training set"/>
                  <connect from_op="Neural Net" from_port="model" to_port="model"/>
                  <portSpacing port="source_training" spacing="0"/>
                  <portSpacing port="sink_model" spacing="0"/>
                  <portSpacing port="sink_through 1" spacing="0"/>
                </process>
                <process expanded="true">
                  <operator activated="true" class="apply_model" compatibility="5.3.015" expanded="true" height="76" name="Apply Model" width="90" x="179" y="120">
                    <list key="application_parameters"/>
                  </operator>
                  <operator activated="true" class="performance_classification" compatibility="5.3.015" expanded="true" height="76" name="Performance" width="90" x="313" y="120">
                    <list key="class_weights"/>
                  </operator>
                  <connect from_port="model" to_op="Apply Model" to_port="model"/>
                  <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
                  <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
                  <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
                  <portSpacing port="source_model" spacing="0"/>
                  <portSpacing port="source_test set" spacing="0"/>
                  <portSpacing port="source_through 1" spacing="0"/>
                  <portSpacing port="sink_averagable 1" spacing="0"/>
                  <portSpacing port="sink_averagable 2" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="log" compatibility="5.3.015" expanded="true" height="76" name="Log" width="90" x="380" y="165">
                <list key="log">
                  <parameter key="LearningRate" value="operator.Neural Net.parameter.learning_rate"/>
                  <parameter key="Accuracy" value="operator.Performance.value.accuracy"/>
                </list>
              </operator>
              <connect from_port="input 1" to_op="Validation" to_port="training"/>
              <connect from_op="Validation" from_port="averagable 1" to_op="Log" to_port="through 1"/>
              <connect from_op="Log" from_port="through 1" to_port="performance"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_performance" spacing="0"/>
              <portSpacing port="sink_result 1" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="x_validation" compatibility="5.3.015" expanded="true" height="112" name="Validation (2)" width="90" x="380" y="30">
            <parameter key="use_local_random_seed" value="true"/>
            <process expanded="true">
              <operator activated="true" class="neural_net" compatibility="5.3.015" expanded="true" height="76" name="Neural Net (2)" width="90" x="221" y="30">
                <list key="hidden_layers"/>
                <parameter key="learning_rate" value="0.1"/>
                <parameter key="shuffle" value="false"/>
                <parameter key="normalize" value="false"/>
                <parameter key="use_local_random_seed" value="true"/>
              </operator>
              <connect from_port="training" to_op="Neural Net (2)" to_port="training set"/>
              <connect from_op="Neural Net (2)" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="5.3.015" expanded="true" height="76" name="Apply Model (2)" width="90" x="45" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="5.3.015" expanded="true" height="76" name="Performance (2)" width="90" x="336" y="30">
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
              <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Retrieve Sonar" from_port="output" to_op="Validation (2)" to_port="training"/>
          <connect from_op="Validation (2)" from_port="averagable 1" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
  • petergao0528petergao0528 Member Posts: 6 Contributor II
    Is there still someone on this issue now?
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Hi,

    you are probably looking at the output of the Log operator, however that one is not configured correctly: you are logging the accuracy of the Performance operator, but that will deliver only the performance of the last iteration of the cross validation. What you want to know is the performance of the complete cross validation, so you have to log the performance value of the X-Validation.

    Hope this helps!

    Best regards,
    Marius
  • dan_agapedan_agape Member Posts: 106 Maven
    Hi,

    I thought these two numbers should be exactly the same. But they are not and they're off quite a lot in my point of view. I don't know if the parameter optimization process is reliable or not.
    You may wish to know, for your application, also whether or not the calculation of AUC is reliable, as basically you want to optimise this parameter. Unfortunately the AUC calculation (as well as the ROC analysis) are not reliable in RM, even in its professional commercial version 6 (not to mention the free versions).

    For more details search the forum for 'AUC error'.

    Dan
  • IngoRMIngoRM Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    The thread mentioned "accuracy" originally, not AUC.  I doubt that here is any connection and, more important, Marius has already posted the solution / reason.
  • dan_agapedan_agape Member Posts: 106 Maven
    As far as I can see now the thread does mention the AUC rather than the accuracy, and thus hopefully you'll see the connection finally.

    Dan
Sign In or Register to comment.