"Precision/Recall charts"

svm_friend

Is it possible to create 2D-Precision/Recall charts like this on:

http://proteomics.bioengr.uic.edu/malibu/docs/images/evaluate/spr_thumb.png

All I found were ROC-curves

land

Hi,
I'm not quite sure, that this is exactly what you want. But you could log the precision / recall and plot it using the plotter as you plot your data.
Here's a process showing how to log them:

<operator name="Root" class="Process" expanded="yes">
    <description text="#ylt#p#ygt# Often the different operators have many parameters and it is not clear which parameter values are best for the learning task at hand. The parameter optimization operator helps to find an optimal parameter set for the used operators. #ylt#/p#ygt#  #ylt#p#ygt# The inner crossvalidation estimates the performance for each parameter set. In this experiment two parameters of the SVM are tuned. The result can be plotted in 3D (using gnuplot) or in color mode. #ylt#/p#ygt#  #ylt#p#ygt# Try the following: #ylt#ul#ygt# #ylt#li#ygt#Start the experiment. The result is the best parameter set and the performance which was achieved with this parameter set.#ylt#/li#ygt# #ylt#li#ygt#Edit the parameter list of the ParameterOptimization operator to find another parameter set.#ylt#/li#ygt# #ylt#/ul#ygt# #ylt#/p#ygt# "/>
    <operator name="Input" class="ExampleSource">
        <parameter key="attributes"	value="..\data\iris.aml"/>
    </operator>
    <operator name="ParameterOptimization" class="GridParameterOptimization" expanded="yes">
        <list key="parameters">
          <parameter key="Training.C"	value="50,100,150,200,250"/>
          <parameter key="Training.degree"	value="1,2,3,4,5"/>
        </list>
        <operator name="Validation" class="XValidation" expanded="yes">
            <parameter key="sampling_type"	value="shuffled sampling"/>
            <operator name="Training" class="LibSVMLearner">
                <parameter key="C"	value="250.0"/>
                <list key="class_weights">
                </list>
                <parameter key="degree"	value="5"/>
                <parameter key="epsilon"	value="0.01"/>
                <parameter key="kernel_type"	value="poly"/>
            </operator>
            <operator name="ApplierChain" class="OperatorChain" expanded="yes">
                <operator name="Test" class="ModelApplier">
                    <list key="application_parameters">
                    </list>
                </operator>
                <operator name="ClassificationPerformance" class="ClassificationPerformance">
                    <list key="class_weights">
                    </list>
                    <parameter key="main_criterion"	value="weighted_mean_recall"/>
                    <parameter key="weighted_mean_precision"	value="true"/>
                    <parameter key="weighted_mean_recall"	value="true"/>
                </operator>
            </operator>
        </operator>
        <operator name="Log" class="ProcessLog">
            <parameter key="filename"	value="paraopt.log"/>
            <list key="log">
              <parameter key="C"	value="operator.Training.parameter.C"/>
              <parameter key="degree"	value="operator.Training.parameter.degree"/>
              <parameter key="recall"	value="operator.Validation.value.performance1"/>
              <parameter key="precision"	value="operator.Validation.value.performance2"/>
            </list>
        </operator>
    </operator>
</operator>

Greetings,
  Sebastian

ema

In case of other classifiers like nearest neighbors and others
what parameters should we consider

land

Hi Ema,
do you ask, which parameters you should log into the processLog? I think this depends on your task at hand. Values you never change probably don't give you much information. So include everything you might optimize or change during the process.

Greetings,
  Sebastian