[Solved] AUC for nominal features

aryan_hosseinza

Hi ,

I am working with a decision tree as a binominal classifier , I want to measure AUC , but I am confused with the result , what threshold is changing, all features are nominal,

Thanks
Arian

MariusHelf

Hi Arian,

please post the process you have so far, and describe what exactly confuses you.

Best, Marius

aryan_hosseinza

I want to know the general setting of AUC in binominal classification evaluation ,


When you want to draw AUC curve , you change a threshold and caluclate TPR and FPR for each threshold and you draw the corresponding curve, but what is threshold changing here ? 

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
    <process expanded="true" height="450" width="882">
      <operator activated="true" class="retrieve" compatibility="5.2.008" expanded="true" height="60" name="Retrieve" width="90" x="112" y="120">
        <parameter key="repository_entry" value="descritized_GI_FROM_MI50"/>
      </operator>
      <operator activated="true" class="sample_stratified" compatibility="5.2.008" expanded="true" height="76" name="Sample (2)" width="90" x="246" y="120">
        <parameter key="sample" value="relative"/>
      </operator>
      <operator activated="true" class="discretize_by_bins" compatibility="5.2.008" expanded="true" height="94" name="Discretize" width="90" x="380" y="120">
        <parameter key="number_of_bins" value="5"/>
        <parameter key="range_name_type" value="short"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.2.008" expanded="true" height="112" name="Validation (2)" width="90" x="514" y="120">
        <parameter key="parallelize_training" value="true"/>
        <parameter key="parallelize_testing" value="true"/>
        <process expanded="true" height="682" width="502">
          <operator activated="true" class="bagging" compatibility="5.2.008" expanded="true" height="76" name="Bagging" width="90" x="179" y="30">
            <parameter key="sample_ratio" value="0.4"/>
            <parameter key="iterations" value="40"/>
            <parameter key="parallelize_learning_process" value="true"/>
            <process expanded="true" height="682" width="1055">
              <operator activated="true" class="multiply" compatibility="5.2.008" expanded="true" height="94" name="Multiply (2)" width="90" x="45" y="30"/>
              <operator activated="true" class="filter_examples" compatibility="5.2.008" expanded="true" height="76" name="Filter Examples (4)" width="90" x="313" y="30">
                <parameter key="condition_class" value="attribute_value_filter"/>
                <parameter key="parameter_string" value="event=f"/>
              </operator>
              <operator activated="true" class="sample_stratified" compatibility="5.2.008" expanded="true" height="76" name="Sample (Stratified)" width="90" x="447" y="30">
                <parameter key="sample" value="relative"/>
                <parameter key="sample_ratio" value="0.8"/>
              </operator>
              <operator activated="true" class="filter_examples" compatibility="5.2.008" expanded="true" height="76" name="Filter Examples (3)" width="90" x="313" y="255">
                <parameter key="condition_class" value="attribute_value_filter"/>
                <parameter key="parameter_string" value="event=t"/>
              </operator>
              <operator activated="true" class="union" compatibility="5.2.008" expanded="true" height="76" name="Union" width="90" x="648" y="210"/>
              <operator activated="true" class="decision_tree" compatibility="5.2.008" expanded="true" height="76" name="Decision Tree" width="90" x="782" y="210">
                <parameter key="criterion" value="gini_index"/>
                <parameter key="minimal_size_for_split" value="160"/>
                <parameter key="minimal_leaf_size" value="80"/>
                <parameter key="minimal_gain" value="0.01"/>
                <parameter key="maximal_depth" value="10"/>
                <parameter key="confidence" value="0.1"/>
                <parameter key="number_of_prepruning_alternatives" value="10"/>
                <parameter key="no_pre_pruning" value="true"/>
              </operator>
              <operator activated="false" class="naive_bayes" compatibility="5.2.008" expanded="true" height="76" name="Naive Bayes" width="90" x="782" y="30"/>
              <connect from_port="training set" to_op="Multiply (2)" to_port="input"/>
              <connect from_op="Multiply (2)" from_port="output 1" to_op="Filter Examples (4)" to_port="example set input"/>
              <connect from_op="Multiply (2)" from_port="output 2" to_op="Filter Examples (3)" to_port="example set input"/>
              <connect from_op="Filter Examples (4)" from_port="example set output" to_op="Sample (Stratified)" to_port="example set input"/>
              <connect from_op="Sample (Stratified)" from_port="example set output" to_op="Union" to_port="example set 1"/>
              <connect from_op="Filter Examples (3)" from_port="example set output" to_op="Union" to_port="example set 2"/>
              <connect from_op="Union" from_port="union" to_op="Decision Tree" to_port="training set"/>
              <connect from_op="Decision Tree" from_port="model" to_port="model"/>
              <portSpacing port="source_training set" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
            </process>
          </operator>
          <connect from_port="training" to_op="Bagging" to_port="training set"/>
          <connect from_op="Bagging" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="682" width="502">
          <operator activated="true" class="apply_model" compatibility="5.2.008" expanded="true" height="76" name="Apply Model (2)" width="90" x="112" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance_binominal_classification" compatibility="5.2.008" expanded="true" height="76" name="Performance (2)" width="90" x="313" y="30">
            <parameter key="main_criterion" value="AUC"/>
            <parameter key="accuracy" value="false"/>
            <parameter key="AUC" value="true"/>
          </operator>
          <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
          <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
          <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Sample (2)" to_port="example set input"/>
      <connect from_op="Sample (2)" from_port="example set output" to_op="Discretize" to_port="example set input"/>
      <connect from_op="Discretize" from_port="example set output" to_op="Validation (2)" to_port="training"/>
      <connect from_op="Validation (2)" from_port="averagable 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

MariusHelf

Hi,

you are right about the threshold, however, the threshold is changed for the confidence of being of belonging to a certain class. This does never have anything to do with the type of the input attributes.
So if you have a decision tree, most leaves won't be pure. If such a leave consists of , let's say, 75% positive examples and 25% negatives, then a new examples which ends up in this leave has a confidence of 75% for being positive.
You are using bagging, that means you are growing several decision trees and then predict by majority vote. But the more of the decision trees make the same classification, the higher the confidence of the composed bagging model.
That means, that for decision trees and for bagging the confidences do not have a continuous range like svms or naive bayes, but only as many "steps" as there are leaves in the tree or classifiers in the bagging model.

Best, Marius

aryan_hosseinza

So you mean that in other algorithms (Naive bayes , SVM , etc.) the confidence is varying to draw the ROC ?

Thanks for your answer, but there arises a basic question , how does the bagging work with decision trees ? at each step , all decision trees vote on what ? I got little confused

Thanks ,
Arian

MariusHelf

aryan.hosseinzadeh wrote:
So you mean that in other algorithms (Naive bayes , SVM , etc.) the confidence is varying to draw the ROC ?

For these algorithms, any confidence value between 0 and 1 can be assigned to a new example, whereas for Decision Trees there are some discrete confidence levels which are defined by the class balances in the single leaves.

[/quote]
Thanks for your answer, but there arises a basic question , how does the bagging work with decision trees ? at each step , all decision trees vote on what ? I got little confused
[/quote]

Here we are talking about model application, obviously. So what bagging does you classify a new example is to pass it to each decision tree, and lets them make their decisions (i.e. classifications). Then, it collects the classifications and predicts the value which the majority of the trees predicted.

Hope this helps!

~Marius

aryan_hosseinza

Thanks , I got it ,

I just checked it and found bagging doesn't make one final model ! I thought that would make one final model , but what about X-Validation , because it results one single final model although it trains several models on different portions of the input data, how does that work in Rapid Miner ? 

MariusHelf

Hey Arian,

if you connect the model output of the X-Validation, in addition to the 10 folds, it creates one more model on the complete data set. This is just for your convenience and has nothing to do with the performance estimation process.

aryan_hosseinza

Now I get why it runs 11 rounds sometimes !

Thanks

MariusHelf

Exactly, that's it