"How to calculate accuracy for every attribute?"

archu92

Hi Experts,

 

In my data set, all attriibutes represents Region Of Interest (ROI) and i want to calculate which ROI gives best accuracy.

Is there any way to calculate accuracy for every attribute in a data set?

 

Thank you,

Archana

Thomas_Ott

I think you're trying to do some variable sensitivity. You sure can!

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.3.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.3.001" expanded="true" height="68" name="Retrieve" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="optimize_selection_forward" compatibility="7.3.001" expanded="true" height="103" name="Forward Selection" width="90" x="180" y="30">
        <parameter key="maximal_number_of_attributes" value="33"/>
        <parameter key="speculative_rounds" value="55"/>
        <process expanded="true">
          <operator activated="true" class="x_validation" compatibility="7.3.001" expanded="true" height="112" name="InsV" width="90" x="45" y="30">
            <process expanded="true">
              <operator activated="true" class="weka:W-J48" compatibility="7.3.000" expanded="true" height="76" name="W-J48" width="90" x="45" y="30"/>
              <connect from_port="training" to_op="W-J48" to_port="training set"/>
              <connect from_op="W-J48" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="7.1.001" expanded="true" height="76" name="InsA" width="90" x="45" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="7.3.001" expanded="true" height="76" name="Performance (2)" width="90" x="179" y="30">
                <parameter key="accuracy" value="false"/>
                <parameter key="kappa" value="true"/>
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="InsA" to_port="model"/>
              <connect from_port="test set" to_op="InsA" to_port="unlabelled data"/>
              <connect from_op="InsA" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
              <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="log" compatibility="7.3.001" expanded="true" height="76" name="Log" width="90" x="180" y="30">
            <list key="log">
              <parameter key="feature" value="operator.Forward Selection.value.feature_names"/>
              <parameter key="performance" value="operator.InsV.value.performance"/>
              <parameter key="deviation" value="operator.InsV.value.deviation"/>
              <parameter key="cpu time" value="operator.InsV.value.cpu-execution-time"/>
              <parameter key="apply count" value="operator.InsV.value.applycount"/>
              <parameter key="number of attributes" value="operator.Forward Selection.value.number of attributes"/>
            </list>
          </operator>
          <connect from_port="example set" to_op="InsV" to_port="training"/>
          <connect from_op="InsV" from_port="averagable 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="performance"/>
          <portSpacing port="source_example set" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="select_by_weights" compatibility="7.3.001" expanded="true" height="103" name="Select by Weights" width="90" x="313" y="30"/>
      <operator activated="true" class="x_validation" compatibility="7.3.001" expanded="true" height="124" name="Validation" width="90" x="447" y="34">
        <process expanded="true">
          <operator activated="true" class="weka:W-J48" compatibility="7.3.000" expanded="true" height="76" name="W-J48 (2)" width="90" x="45" y="30"/>
          <connect from_port="training" to_op="W-J48 (2)" to_port="training set"/>
          <connect from_op="W-J48 (2)" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.1.001" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="7.3.001" expanded="true" height="76" name="Performance (3)" width="90" x="179" y="165">
            <parameter key="accuracy" value="false"/>
            <parameter key="kappa" value="true"/>
            <list key="class_weights"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance (3)" to_port="labelled data"/>
          <connect from_op="Performance (3)" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="log_to_data" compatibility="7.3.001" expanded="true" height="103" name="Log to Data" width="90" x="447" y="210">
        <parameter key="log_name" value="Log"/>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Forward Selection" to_port="example set"/>
      <connect from_op="Forward Selection" from_port="example set" to_op="Select by Weights" to_port="example set input"/>
      <connect from_op="Forward Selection" from_port="attribute weights" to_op="Select by Weights" to_port="weights"/>
      <connect from_op="Select by Weights" from_port="example set output" to_op="Validation" to_port="training"/>
      <connect from_op="Select by Weights" from_port="original" to_op="Log to Data" to_port="through 1"/>
      <connect from_op="Select by Weights" from_port="weights" to_port="result 3"/>
      <connect from_op="Validation" from_port="model" to_port="result 1"/>
      <connect from_op="Validation" from_port="averagable 1" to_port="result 2"/>
      <connect from_op="Log to Data" from_port="exampleSet" to_port="result 4"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
    </process>
  </operator>
</process>

 

 

archu92

Thanks a lot Thomas.

archu92

Hi @Thomas_Ott,

 

I have some doubts on how to choose parameter values:

For Forward seletion operator:

why <parameter key="maximal_number_of_attributes" value="33"/>  and   <parameter key="speculative_rounds" value="55"/> are choosen?

 

Thank you,

Archana

 

Thomas_Ott

I'm not sure by what you mean of "doubts?" Are you asking why those parameter values were chosen?  They were chosen arbitrarily for this example because we used the Iris Data set.  

 

The "Maximal Number of Attributes" tells the Foward Selection (FS) that after it's all done, you want a maximum (not to exceed) of 33 attribiutes. In the case of Iris, it has 4 so you get 4. "Number of Speculative Rounds" helps with avoiding the FS from getting stuck in a local optima. It essentially is the number of times where the stopping critera (next parameter below) can be ignored.