Weka model accuracy unknown...or random?

Fred12

hi,

I had previously problems with retrieving Weka models from Repository, but when I disabled several extensions, it worked afterwards...

I now have the problem that some Functional Tree operator models from Weka like W-FT and W-LMT bring me accuracy unknown when applying on test data:

 

 

I have the sa me problem with W-Rotation Forest and W-Random Forest brings me just 33% arbitrary random performance...

anyone knows what the problem is?

Thomas_Ott

Weka models? Do you mean models trained using Weka algorithms in RapidMiner and then stored in the RapidMiner format?

 

I usually never have a problem and need unload extensions. What if you use a RapidMiner Random Forest as a test to see if you get any P/R results?

Fred12

I get the usual good / Appropriate results with any Rapidminer models or operators... Random Forest model in Rapidminer works nice...

but when I train Weka Random Forest model and then store it in repository, retrieve it and test it again on test data the results are weird...

 

edit: I Just noticed, there is a "Read Weka Model" operator, but it seems not to work, at least it cannot read models stored from the repository... But I don't know how to else read / retrieve them, as there is no "Store Weka Model" operator... is that the reason the model doesn't work or does it not matter how to retrieve or store the model? Is the reason more the Rapidminer format / framework?

Thomas_Ott

I'm not sure why it's doing this but I can confirm something's not right. I just did a test with W-Random Forest, stored the model and tried to retrieve it.  I get this. 

 

Let me ping someone in development. 

 

 

 

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.4.000" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve Iris" width="90" x="112" y="34">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="7.4.000" expanded="true" height="145" name="Validation" width="90" x="313" y="34">
        <parameter key="sampling_type" value="stratified sampling"/>
        <process expanded="true">
          <operator activated="true" class="weka:W-RandomForest" compatibility="7.3.000" expanded="true" height="82" name="W-RandomForest" width="90" x="209" y="34"/>
          <connect from_port="training set" to_op="W-RandomForest" to_port="training set"/>
          <connect from_op="W-RandomForest" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.4.000" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="7.4.000" expanded="true" height="82" name="Performance" width="90" x="179" y="34"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="performance 1"/>
          <connect from_op="Performance" from_port="example set" to_port="test set results"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
          <description align="left" color="blue" colored="true" height="103" resized="true" width="315" x="38" y="137">The model created in the Training step is applied to the current test set (10 %).&lt;br/&gt;The performance is evaluated and sent to the operator results.</description>
        </process>
        <description align="center" color="transparent" colored="false" width="126">A cross-validation evaluating a decision tree model.</description>
      </operator>
      <operator activated="true" class="store" compatibility="7.4.000" expanded="true" height="68" name="Store" width="90" x="514" y="34">
        <parameter key="repository_entry" value="../data/WekaModel"/>
      </operator>
      <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve Iris (2)" width="90" x="45" y="289">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve WekaModel" width="90" x="514" y="136">
        <parameter key="repository_entry" value="../data/WekaModel"/>
      </operator>
      <operator activated="true" class="apply_model" compatibility="7.4.000" expanded="true" height="82" name="Apply Model (2)" width="90" x="380" y="289">
        <list key="application_parameters"/>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Validation" to_port="example set"/>
      <connect from_op="Validation" from_port="model" to_op="Store" to_port="input"/>
      <connect from_op="Validation" from_port="performance 1" to_port="result 1"/>
      <connect from_op="Retrieve Iris (2)" from_port="output" to_op="Apply Model (2)" to_port="unlabelled data"/>
      <connect from_op="Retrieve WekaModel" from_port="output" to_op="Apply Model (2)" to_port="model"/>
      <connect from_op="Apply Model (2)" from_port="labelled data" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

 

 

 

 

 

Telcontar120

I can confirm that I have experienced a similar problem with Weka models in recent Studio releases.  I am not sure which version this orginated with, but it is problematic.  I have had a similar problem where even trying to review the Weka model in the "model" view doesn't work, it just results in an endless-spinning-hourglass.  It would be great to have this corrected ASAP.

michal_moskal

Has this issue been resolved now? I think I can still reproduce this in RapidMiner 8

sgenzer

Issue still exists and and reported to dev team. I will update as needed.

SG