The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Accuracy remains constant
Maven
I want to optimize the parameters of the decision tree learner using EvolutionaryParameterOptimization.
That's my model:
lines of the generated log file:
the same. But I can't imagine that the number of correctly classified examples is always the same. Any
ideas what I'm dong wrong?
Chris
That's my model:
The strange thing is that the accuracy remains always (almost) the same. Here are the first
<?xml version="1.0" encoding="US-ASCII"?>
<process version="4.3">
<operator name="Root" class="Process" expanded="yes">
<operator name="CSVExampleSource" class="CSVExampleSource">
<parameter key="filename" value="example.csv"/>
<parameter key="label_name" value="label"/>
</operator>
<operator name="EvolutionaryParameterOptimization" class="EvolutionaryParameterOptimization" expanded="yes">
<list key="parameters">
<parameter key="DecisionTree.maximal_depth" value="[-1.0;10000.0]"/>
<parameter key="DecisionTree.minimal_leaf_size" value="[1.0;10000.0]"/>
<parameter key="DecisionTree.confidence" value="[1.0E-7;0.5]"/>
<parameter key="DecisionTree.minimal_size_for_split" value="[1.0;10000.0]"/>
<parameter key="DecisionTree.minimal_gain" value="[0.0;Infinity]"/>
</list>
<parameter key="population_size" value="10"/>
<operator name="IteratingPerformanceAverage" class="IteratingPerformanceAverage" expanded="yes">
<parameter key="iterations" value="3"/>
<operator name="XValidation" class="XValidation" expanded="yes">
<parameter key="sampling_type" value="shuffled sampling"/>
<operator name="DecisionTree" class="DecisionTree">
<parameter key="confidence" value="0.33223030703745715"/>
<parameter key="maximal_depth" value="6341"/>
<parameter key="minimal_gain" value="Infinity"/>
<parameter key="minimal_leaf_size" value="1424"/>
<parameter key="minimal_size_for_split" value="2961"/>
</operator>
<operator name="OperatorChain" class="OperatorChain" expanded="yes">
<operator name="ModelApplier" class="ModelApplier">
<list key="application_parameters">
</list>
</operator>
<operator name="ClassificationPerformance" class="ClassificationPerformance">
<parameter key="absolute_error" value="true"/>
<parameter key="accuracy" value="true"/>
<list key="class_weights">
</list>
<parameter key="classification_error" value="true"/>
</operator>
</operator>
</operator>
</operator>
<operator name="ProcessLog" class="ProcessLog">
<parameter key="filename" value="process3.log"/>
<list key="log">
<parameter key="iteration" value="operator.XValidation.value.iteration"/>
<parameter key="time" value="operator.XValidation.value.time"/>
<parameter key="deviation" value="operator.XValidation.value.deviation"/>
<parameter key="accuracy" value="operator.XValidation.value.performance1"/>
<parameter key="max_depth" value="operator.DecisionTree.parameter.maximal_depth"/>
<parameter key="max_leaf_size" value="operator.DecisionTree.parameter.minimal_leaf_size"/>
<parameter key="confidence" value="operator.DecisionTree.parameter.confidence"/>
</list>
<parameter key="persistent" value="true"/>
</operator>
</operator>
<operator name="PerformanceWriter" class="PerformanceWriter">
<parameter key="performance_file" value="final_performance.per"/>
</operator>
<operator name="ParameterSetWriter" class="ParameterSetWriter">
<parameter key="parameter_file" value="parameters.par"/>
</operator>
</operator>
</process>
lines of the generated log file:
Even when I replace the decision tree learned by a neural network, the accuracy values are always
# Generated by ProcessLog[com.rapidminer.operator.visualization.ProcessLogOperator]
# iteration time deviation accuracy max_depth max_leaf_size confidence
10.0 23.0 0.05129744389973656 0.7932203389830508 9839.0 1251.0 0.34154054826704505
10.0 23.0 0.05507822647211116 0.7932203389830507 6341.0 7767.0 0.32651644400635255
10.0 23.0 0.0392401250942013 0.793220338983051 3707.0 48.0 0.397380892497813
10.0 22.0 0.0726246958834861 0.7932203389830509 9008.0 5164.0 0.40415961507086146
10.0 25.0 0.03774755500223652 0.7932203389830509 7652.0 3992.0 0.27277836592326715
10.0 28.0 0.055078226472110144 0.7932203389830507 293.0 1424.0 0.03868159703154637
10.0 27.0 0.06287198979997247 0.7932203389830507 615.0 6926.0 0.12025785562389255
10.0 46.0 0.026037782196170964 0.7932203389830507 4846.0 1825.0 0.4842464140080626
the same. But I can't imagine that the number of correctly classified examples is always the same. Any
ideas what I'm dong wrong?
Chris
Tagged:
0
Answers
Are you really sure that you should have 'infinity' as your minimal split parameter?
HAPPY NEW YEAR TO ALL!
OK, maybe infinity as minimal gain is little bit too large. :-)
However, this seems not to be the problem. After reducing this
parameter to max. 100, I still get same results where the accuracy
is remaining (almost) constant. Any other ideas what might be wrong?
By the way, what exactly is the definition of an "minimal size of
a leaf" for the DecisionTree operator? This is used in the parameter
"minimal_size_for_split". Is it the value computed by the criterion?
Also happy new year to all.
Chris
as far as I can see from your posted log, the parameters have never been within a region, where changes might occur.
One possibility for constant accuracy is the collapsing of the models into a default model. Perhabs you smaller class is never discovered by the learner?
Greetings,
Sebastian
I've now simplified my model: and here is the complete log file: So, you can see that the accuracy is still constant. All parameters to be optimized seem also to be correctly
evaluated. Is this behavior really OK or do you have any other
suggestions what might went wrong?
Regards,
Chris
first at all:
What you did post here is not your modell, its your process. Models are the results of learning algorithms like trees, rulesets and so on.
The most basic model is called "default model". It means, that it always predicts the same class, which is the most frequent class in the trainingset, independent from the values of the current example.
All learning algorithms might produce models equivalent to this default models. For example this is the case if a tree only consists of one node. Another example would be an SVM hyperplane laid far beyond the data itself.
The learning process and hence the building process of the models are often controlled by parameters. Thats why you are optimizing this parameters, because you want to optimize the learning performance.
BUT not every distinct value will lead to a different model. Many values create the same result. So if the model building will change if the parameter value a exceeds 6, 7, 8, 1000 or greater have not to change it again.
For example the maximal depth of a tree is not important if the generation never reaches this limit. So 100, 100000, or 782379821412 will not result in a different tree than 15 if the maximal depth ever generate during leraning is 15.
Hope this will get you some ideas.
Greetings,
Sebastian
thank you for your detailed answer.This was very helpful. I didn't know that the "maximal" parameters
are just upper bounds for the parameter values. My assumption was that RapidMiner is always trying to
learn a model that tries to use the given values, i.e. RM learns a model with the maximal (or minimal)
parameter values specified.
So, parameter optimization requires some deeper knowledge on the model to be optimized, right? I mean
when "arbitrary" parameter values are evaluated, it might happen, as in my case, that many combinations
don't make sense since they create the same model, thus their evaluation is wasted time. Can you give me
any advices how parameter optimization should be effectively performed? Does it make sense to use just
small values for "maximal" parameters since they will more likely lead to different results? Or was my approach
OK and I can infer from the results that the learned DecisionTrees is not sensitive to my learning data, i.e. it
will mostly end up in the same accuracy?
Regards,
Chris