Problem with Parameter Optimization

christian1983

Hi everybody,

in order to improve the performance of a neural network, i´m trying to determine the best network parameter with the optimization operator.
But i get always an error: Process Failed Cannot reset network to a smaller learning rate.

Here is my process:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.0.8" expanded="true" name="Process">
    <process expanded="true" height="377" width="622">
      <operator activated="true" class="retrieve" compatibility="5.0.8" expanded="true" height="60" name="Retrieve" width="90" x="52" y="138">
        <parameter key="repository_entry" value="//MyRepository/Data/DM_InputData_C15_Umformgrad"/>
      </operator>
      <operator activated="true" class="nominal_to_numerical" compatibility="5.0.8" expanded="true" height="94" name="Nominal to Numerical" width="90" x="179" y="120"/>
      <operator activated="true" class="optimize_parameters_grid" compatibility="5.0.8" expanded="true" height="94" name="Optimize Parameters (Grid)" width="90" x="380" y="75">
        <list key="parameters">
          <parameter key="Neural Net.learning_rate" value="[4.9E-324;20;10;quadratic]"/>
          <parameter key="Neural Net.momentum" value="[0.0;10;10;quadratic]"/>
        </list>
        <process expanded="true" height="395" width="640">
          <operator activated="true" class="x_validation" compatibility="5.0.8" expanded="true" height="112" name="Validation" width="90" x="105" y="44">
            <process expanded="true" height="395" width="295">
              <operator activated="true" class="neural_net" compatibility="5.0.8" expanded="true" height="76" name="Neural Net" width="90" x="53" y="31">
                <list key="hidden_layers"/>
                <parameter key="learning_rate" value="1.7999999999999998"/>
                <parameter key="momentum" value="0.0"/>
              </operator>
              <connect from_port="training" to_op="Neural Net" to_port="training set"/>
              <connect from_op="Neural Net" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true" height="395" width="295">
              <operator activated="true" class="apply_model" compatibility="5.0.8" expanded="true" height="76" name="Apply Model" width="90" x="23" y="62">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance" compatibility="5.0.8" expanded="true" height="76" name="Performance" width="90" x="165" y="116"/>
              <connect from_port="model" to_op="Apply Model" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
              <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <connect from_port="input 1" to_op="Validation" to_port="training"/>
          <connect from_op="Validation" from_port="averagable 1" to_port="performance"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Nominal to Numerical" to_port="example set input"/>
      <connect from_op="Nominal to Numerical" from_port="example set output" to_op="Optimize Parameters (Grid)" to_port="input 1"/>
      <connect from_op="Optimize Parameters (Grid)" from_port="performance" to_port="result 2"/>
      <connect from_op="Optimize Parameters (Grid)" from_port="parameter" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

What am I doing wrong?

I hope someone could help me.

Thank you.

land

Hi,
It seems to me the net cannot be set to a smaller learning rate During fitting the neural net, the initial learning rate will be decreased with every iteration. Now the implementation seems to avoid setting it to a too small rate. I would suggest to use a higher initial rate? Doesn't this solve the problem.

By the way: It seems to me, you have asked the question already somewhere else. And as you might understand, we are not happy to repeat ourselves...

Greetings,
  Sebastian

david

I had the same problem with a dataset and for me it turned out to be a zero-divide prioblem. I have odds in my dataset and when i calculate percentages for them by doing 1/odds and the odds turned out to be a zero i got this error while trying to do a parameter optimization. Just removing the zero-odds from the dataset solved my problem. These showed up as∞ in my example dataset.