confidences and probability

babaluma

I am trying to feed RapidMiner output (predictions and probabilities)  into an external program. I understand how to output confidence scores using ExampleSetWriter, but not probabilities. I found a thread in the old SourceForge forum that suggests using PlattScaling to convert confidence scores to probabilities. I looked around a little bit, but could not get this operator to get what I want. How should I go about doing this? How do I find and write the probability that a particular instance belongs to a class?

Thanks,
B

steffen

Hello babaluma

For others: Here is the link to the mentioned sf-thread: click.

Regarding your problem:
I understand Platt Scaling this way:
input: the ExampleSet the original model was created with and the created model
output: the "corrected" model.

Applying the corrected model to your test set results in real probabilities in the confidence column, i.e. confidence(c)=p(y=c|x=current Example).

hope this was helpful

Steffen

PS: This topic is really interesting. Recently I stumbled upon a paper converting scores of multiple-class-classifiers, but unfortunately, I did not have the time to study it yet. Here is the link (PDF):Transforming Classifier Scores into Accurate Multiclass Probability Estimates

babaluma

thanks for the reply.

while that seems to work, I am having trouble in getting probabilities from the training set with cross-validation. Where exactly should PlattScaling operator be placed in such a scenario?
My cross validation step looks like the following:
-XValidation
--NaiveBayes
--OperatorChain
---ModelApplier
---BinomialClassificationPerformance
---ExampleWriter
---ResultWriter
-ModelWriter
-PerformanceWriter

It seems that PlattScaling should be performed after the learning step and before the ExampleWriter operator. However, by placing it before the ExampleWriter operator, ExampleSet is lost (PlattScaling takes ExampleSet and Model and returns Model). Placing it after the learning step, I somehow get an error in the PerformanceVector operator. The error is that "an object of PerformanceVector type is needed". Without PlattScaling, it works fine. I suppose if PlattScaling had the parameter "keep_example_set", this would work. I would appreciate any feedback.

Thanks
B

steffen

Hello again

If you want to create a "platt-scaled" model for each training step, I suggest something like this: (simple copy and paste it to the xml-tab of rm-interface)

<operator name="Root" class="Process" expanded="yes">
    <operator name="XValidation" class="XValidation" expanded="yes">
        <operator name="create_scaled_model" class="OperatorChain" expanded="yes">
            <operator name="NaiveBayes" class="NaiveBayes">
                <parameter key="keep_example_set"	value="true"/>
            </operator>
            <operator name="PlattScaling" class="PlattScaling">
            </operator>
        </operator>
        <operator name="OperatorChain" class="OperatorChain" expanded="yes">
            <operator name="ModelApplier" class="ModelApplier">
                <list key="application_parameters">
                </list>
            </operator>
            <operator name="BinominalClassificationPerformance" class="BinominalClassificationPerformance">
            </operator>
        </operator>
    </operator>
</operator>

Hope this is what you wanted to do

greetings

Steffen

babaluma

That seems to work. Thanks a lot.

B