"Cannot evaluate performance for current parameter combination"

Legacy User

Hi,

I'm doing some GridParameterOptimization for RandomForests. My model
is pretty much the same as the one given in the samples.

Three parameters are evaluated:

RandomForest.number_of_trees, min=1.0, max=100.0, steps=10, logarithmic
RandomForest.criterion, all possible criterion
and RandomForest.maximal_depth, min=-1.0, max=100.0, steps=10, logarithmic.

Sometimes, the learner terminates with:

[NOTE] OperatorChain: Process stopped.
[Warning] GridParameterOptimization: Cannot evaluate performance for current
parameter combination: Process stopped in OperatorChain
[NOTE] GridParameterOptimization: Process stopped.

Why Is it possible that some parameter combinations are not valid? And
how can I figure out which combination failed?
To avoid such "crashes" in the future, is it possible to tell RapidMiner to
ignore these invalid combinations without terminating?

Thanks,
ben

land

Hi Ben,
could you post your process file here?  Thats probably not the behavior intended And at the first glance I don't see, why this can happen...

Greetings,
  Sebastian

Legacy User

Hi Sebastian,

that's my model:

<operator name="Root" class="Process" expanded="yes">
    <operator name="CSVExampleSource" class="CSVExampleSource">
        <parameter key="filename"	value="examples.csv"/>
        <parameter key="label_name"	value="mylabel"/>
    </operator>
    <operator name="GridParameterOptimization" class="GridParameterOptimization" expanded="yes">
        <list key="parameters">
          <parameter key="RandomForest.number_of_trees"	value="[1.0;100.0;10;logarithmic]"/>
          <parameter key="RandomForest.criterion"	value="gain_ratio,information_gain,gini_index,accuracy"/>
          <parameter key="RandomForest.maximal_depth"	value="[-1.0;100.0;10;logarithmic]"/>
        </list>
        <operator name="XValidation" class="XValidation" expanded="yes">
            <parameter key="leave_one_out"	value="true"/>
            <parameter key="sampling_type"	value="shuffled sampling"/>
            <operator name="RandomForest" class="RandomForest">
                <parameter key="criterion"	value="gain_ratio"/>
                <parameter key="maximal_depth"	value="-1"/>
                <parameter key="number_of_trees"	value="5405"/>
            </operator>
            <operator name="OperatorChain" class="OperatorChain" expanded="yes">
                <operator name="ModelApplier" class="ModelApplier">
                    <list key="application_parameters">
                    </list>
                </operator>
                <operator name="ClassificationPerformance" class="ClassificationPerformance">
                    <parameter key="absolute_error"	value="true"/>
                    <parameter key="accuracy"	value="true"/>
                    <list key="class_weights">
                    </list>
                    <parameter key="classification_error"	value="true"/>
                </operator>
            </operator>
        </operator>
        <operator name="ProcessLog" class="ProcessLog">
            <parameter key="filename"	value="/tmp/process.log"/>
            <list key="log">
              <parameter key="iteration"	value="operator.XValidation.value.iteration"/>
              <parameter key="time"	value="operator.XValidation.value.time"/>
              <parameter key="deviation"	value="operator.XValidation.value.deviation"/>
              <parameter key="variance"	value="operator.XValidation.value.variance"/>
              <parameter key="accuracy"	value="operator.XValidation.value.performance"/>
              <parameter key="classification_error"	value="operator.XValidation.value.performance2"/>
              <parameter key="absolute_error"	value="operator.XValidation.value.performance3"/>
              <parameter key="num_trees"	value="operator.RandomForest.parameter.number_of_trees"/>
              <parameter key="criterion"	value="operator.RandomForest.parameter.criterion"/>
              <parameter key="max_depth"	value="operator.RandomForest.parameter.maximal_depth"/>
            </list>
            <parameter key="persistent"	value="true"/>
        </operator>
    </operator>
    <operator name="PerformanceWriter" class="PerformanceWriter">
        <parameter key="performance_file"	value="/tmp/performance.per"/>
    </operator>
    <operator name="ParameterSetWriter" class="ParameterSetWriter">
        <parameter key="parameter_file"	value="/tmp/parameters.par"/>
    </operator>
</operator>

Maybe you can find something.

Thank you.

Ben

land

Hi Ben,
it seems to me, that RapidMiner does not have the permission to write into the file you specified. Did you check that?
I'm only able to reproduce the error then.

Greetings,
  Sebastian