"Saving detail results (folder view) of unsupervised clustering"

B_

I've been running unsupervised learning processes and obtaining results.  I would like to save the detail results that show the cluster number and individual items in that cluster (folder view) so I can check the results quickly.  When I use ResultWriter it only writes the Text View (summary) to a file.

Is there another operator or a cluster parameter to set that will save the folder view to a file? 

Or do I have to save the cluster model then run supervised classification to see the detail results?

Thanks.

TobiasMalbrecht

Hi,

as you already noticed, a ResultWriter only saves the textual represenations of the objects that are output of a process. The operator you are looking for is called IOObjectWriter which lets you save any IOObject, in particular also a ClusterModel. Hence, you have to chose ClusterModel for the parameter "io_object" in order to save your ClusterModel. You may the load that ClusterModel back into RapidMiner by using the corresponding IOObjectReader which is in the same operator group (IO.Other).

Hope that solves your problem.
Regards,
Tobias

B_

Tobias

The IOObjectWriter does save the cluster model as an XML file (and when I load it I see the Cluster Model Data and Folder views) but there is no way to directly use the IO format file. 

I can write a parser to adapt the XML - this is good enough for now.  Thanks.

IngoRM

Hi,

if you want to show the folder view from your own application (?) maybe reloading it and just using the available cluster model visualization might be an option. This is simply one line for a loaded ClusterModel "cm":

Component folderView = new ExtendedJScrollPane(new ClusterTreeVisualization((FlatClusterModel) cm));

Maybe this helps (in case you want to display it in another application).

Cheers,
Ingo

B_

Tobias,  Ingo

While looking for something else I discovered Cluster2ExampleSet.  This does what I need, and then I can write the example set to a database, Excel, etc.

However, the row/record ID and cluster number are written at the end of each example line, and when I save to Excel the cluster info is dropped.  There is no switch on Exampl2Cluster or ExampleWriter to place the record ID and cluster number at the beginning of the line.  (Over 1800 terms in my small test set, which converts to 1800+ columns to save when the data is written.) 

Is it possible to add an ExampleWriter parameter to place the important information at the beginning of the line?  Then I can import directly into my database.

For now I can write a parser to read the end of the line in a csv file and extract what I need.

Thanks.  (I am sure you guys are being hammered with extra requests from the new release.)

B.

B_

Perhaps 3 choices of format
0- as it is now
1- place rec ID and cluster number at the beginning of the line and attributes after
2 - output only rec ID and cluster number, leave off cluster attributes

IngoRM

Hi,

well, this is actually already possible. Just use the ExampleSetWriter operator and set the "format" parameter to "special_format" and define an appropriate "special_format" parameter. The documentation (tutorial and the tooltip inside the program or F1) will show you the possible settings for the special format parameter. However, here is the solution for the three suggested scenarios:

0- as it is now

Here you can just use the "dense" format:

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="number_of_attributes"	value="2"/>
        <parameter key="target_function"	value="gaussian mixture clusters"/>
    </operator>
    <operator name="IdTagging" class="IdTagging">
    </operator>
    <operator name="FeatureNameFilter" class="FeatureNameFilter">
        <parameter key="filter_special_features"	value="true"/>
        <parameter key="skip_features_with_name"	value="label"/>
    </operator>
    <operator name="KMeans" class="KMeans">
        <parameter key="k"	value="4"/>
    </operator>
    <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
        <parameter key="keep_cluster_model"	value="false"/>
    </operator>
    <operator name="ExampleSetWriter" class="ExampleSetWriter">
        <parameter key="example_set_file"	value="cm_out_complete.dat"/>
    </operator>
</operator>

1- place rec ID and cluster number at the beginning of the line and attributes after

Here you can use the special format "$i $v[cluster] $a":

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="number_of_attributes"	value="2"/>
        <parameter key="target_function"	value="gaussian mixture clusters"/>
    </operator>
    <operator name="IdTagging" class="IdTagging">
    </operator>
    <operator name="FeatureNameFilter" class="FeatureNameFilter">
        <parameter key="filter_special_features"	value="true"/>
        <parameter key="skip_features_with_name"	value="label"/>
    </operator>
    <operator name="KMeans" class="KMeans">
        <parameter key="k"	value="4"/>
    </operator>
    <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
        <parameter key="keep_cluster_model"	value="false"/>
    </operator>
    <operator name="ExampleSetWriter" class="ExampleSetWriter">
        <parameter key="example_set_file"	value="cm_out_complete.dat"/>
        <parameter key="format"	value="special_format"/>
        <parameter key="special_format"	value="$i $v[cluster] $a"/>
    </operator>
</operator>

2 - output only rec ID and cluster number, leave off cluster attributes

Here you can just omit the parameter "$a" above like in this example:

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="number_of_attributes"	value="2"/>
        <parameter key="target_function"	value="gaussian mixture clusters"/>
    </operator>
    <operator name="IdTagging" class="IdTagging">
    </operator>
    <operator name="FeatureNameFilter" class="FeatureNameFilter">
        <parameter key="filter_special_features"	value="true"/>
        <parameter key="skip_features_with_name"	value="label"/>
    </operator>
    <operator name="KMeans" class="KMeans">
        <parameter key="k"	value="4"/>
    </operator>
    <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
        <parameter key="keep_cluster_model"	value="false"/>
    </operator>
    <operator name="ExampleSetWriter" class="ExampleSetWriter">
        <parameter key="example_set_file"	value="cm_out_complete.dat"/>
        <parameter key="format"	value="special_format"/>
        <parameter key="special_format"	value="$i $v[cluster]"/>
    </operator>
</operator>

Cheers,
Ingo

B_

OK

This is what I need.  I'll let you know what the results are.

Thanks.  Have a good weekend.

vijaypshah

How can I remove the word cluster and only use the number?
For example instead of having cluster2, i just want to save it as 2

Regards,
Vijay

IngoRM

Hi,

I don't get it. I don't get the word "cluster"...

Could you please post your process (XML from the XML tab) and attach a screenshot showing the place where the word "cluster" occurs? Thanks.

Cheers,
Ingo

vijaypshah

I had more time to play around with the tool now. Seems it might have to do with Weka -unsupervised learning maybe, i didn't get the cluster word with kmeans or support vector clustering...

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSource" class="ExampleSource">
        <parameter key="attributes" value="C:\Documents and Settings\shahv\My Documents\rm_workspace\temp.aml"/>
    </operator>
    <operator name="W-EM" class="W-EM">
        <parameter key="N" value="5.0"/>
    </operator>
</operator>

Results aml file

<?xml version="1.0" encoding="windows-1252"?>
<attributeset default_source="..\..\Program Files (x86)\Rapid-I\RapidMiner-4.1\temp.dat">

  <attribute
    name        = "feature_id.dat (2)"
    sourcecol    = "1"
    valuetype    = "integer"/>

  <attribute
    name        = "feature_id.dat (3)"
    sourcecol    = "2"
    valuetype    = "integer"/>

  <attribute
    name        = "feature_id.dat (4)"
    sourcecol    = "3"
    valuetype    = "integer"/>

  <attribute
    name        = "feature_id.dat (5)"
    sourcecol    = "4"
    valuetype    = "integer"/>

  <id
    name        = "feature_id.dat (1)"
    sourcecol    = "5"
    valuetype    = "integer"/>

  <cluster
    name        = "cluster"
    sourcecol    = "6"
    valuetype    = "nominal">
      <value>cluster4</value>
      <value>cluster0</value>
      <value>cluster3</value>
      <value>cluster2</value>
      <value>cluster1</value>
  </cluster>

</attributeset>

[attachment deleted by admin]

B_

Ingo

The Special Format output is working.  I'm able to get just the record ID and cluster number. 

Thanks for your help.

B.