The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
"Loop: store/view output from each iteration?"
Contributor II
Hi,
I have a question about the "Loop clusters" method. If I have a loop that loops over a number of clutsers, how can I get the output from all the X-validations I have going inside the loop? With this setup I only get the output of the last loop in the output. If there are say 5 clusters, how do I get all the results to the output? Alternatively, how can I save the "Perofrmance vector" classification matrix as simple text (I don't want the whole performance-thing save to a RM object).
Simple example:
I have a question about the "Loop clusters" method. If I have a loop that loops over a number of clutsers, how can I get the output from all the X-validations I have going inside the loop? With this setup I only get the output of the last loop in the output. If there are say 5 clusters, how do I get all the results to the output? Alternatively, how can I save the "Perofrmance vector" classification matrix as simple text (I don't want the whole performance-thing save to a RM object).
Simple example:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.0.11" expanded="true" name="Process">
<process expanded="true" height="325" width="480">
<operator activated="true" class="retrieve" compatibility="5.0.11" expanded="true" height="60" name="Retrieve (2)" width="90" x="45" y="75">
<parameter key="repository_entry" value="//Samples/data/Golf"/>
</operator>
<operator activated="true" class="set_role" compatibility="5.0.11" expanded="true" height="76" name="Set Role" width="90" x="179" y="75">
<parameter key="name" value="Outlook"/>
<parameter key="target_role" value="cluster"/>
</operator>
<operator activated="true" class="loop_clusters" compatibility="5.0.11" expanded="true" height="94" name="Loop Clusters" width="90" x="313" y="75">
<process expanded="true" height="559" width="720">
<operator activated="true" class="x_validation" compatibility="5.0.11" expanded="true" height="112" name="Validation (2)" width="90" x="246" y="30">
<parameter key="number_of_validations" value="3"/>
<parameter key="sampling_type" value="shuffled sampling"/>
<process expanded="true" height="559" width="335">
<operator activated="true" class="k_nn" compatibility="5.0.11" expanded="true" height="76" name="k-NN (3)" width="90" x="122" y="30">
<parameter key="k" value="9"/>
<parameter key="weighted_vote" value="true"/>
</operator>
<connect from_port="training" to_op="k-NN (3)" to_port="training set"/>
<connect from_op="k-NN (3)" from_port="model" to_port="model"/>
<connect from_op="k-NN (3)" from_port="exampleSet" to_port="through 1"/>
<portSpacing port="source_training" spacing="0"/>
<portSpacing port="sink_model" spacing="0"/>
<portSpacing port="sink_through 1" spacing="0"/>
<portSpacing port="sink_through 2" spacing="0"/>
</process>
<process expanded="true" height="559" width="335">
<operator activated="true" class="apply_model" compatibility="5.0.11" expanded="true" height="76" name="Apply Model (2)" width="90" x="45" y="30">
<list key="application_parameters"/>
</operator>
<operator activated="true" class="find_threshold" compatibility="5.0.11" expanded="true" height="76" name="Find Threshold (3)" width="90" x="179" y="30">
<parameter key="misclassification_costs_second" value="1.4"/>
</operator>
<operator activated="true" class="apply_threshold" compatibility="5.0.11" expanded="true" height="76" name="Apply Threshold (3)" width="90" x="45" y="210"/>
<operator activated="true" class="performance" compatibility="5.0.11" expanded="true" height="76" name="Performance (4)" width="90" x="179" y="210"/>
<connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
<connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
<connect from_op="Apply Model (2)" from_port="labelled data" to_op="Find Threshold (3)" to_port="example set"/>
<connect from_op="Find Threshold (3)" from_port="example set" to_op="Apply Threshold (3)" to_port="example set"/>
<connect from_op="Find Threshold (3)" from_port="threshold" to_op="Apply Threshold (3)" to_port="threshold"/>
<connect from_op="Apply Threshold (3)" from_port="example set" to_op="Performance (4)" to_port="labelled data"/>
<connect from_op="Performance (4)" from_port="performance" to_port="averagable 1"/>
<portSpacing port="source_model" spacing="0"/>
<portSpacing port="source_test set" spacing="0"/>
<portSpacing port="source_through 1" spacing="0"/>
<portSpacing port="source_through 2" spacing="0"/>
<portSpacing port="sink_averagable 1" spacing="0"/>
<portSpacing port="sink_averagable 2" spacing="0"/>
</process>
</operator>
<connect from_port="cluster subset" to_op="Validation (2)" to_port="training"/>
<connect from_op="Validation (2)" from_port="model" to_port="out 1"/>
<connect from_op="Validation (2)" from_port="averagable 1" to_port="out 2"/>
<portSpacing port="source_cluster subset" spacing="0"/>
<portSpacing port="source_in 1" spacing="0"/>
<portSpacing port="sink_out 1" spacing="0"/>
<portSpacing port="sink_out 2" spacing="0"/>
<portSpacing port="sink_out 3" spacing="0"/>
</process>
</operator>
<connect from_op="Retrieve (2)" from_port="output" to_op="Set Role" to_port="example set input"/>
<connect from_op="Set Role" from_port="example set output" to_op="Loop Clusters" to_port="example set"/>
<connect from_op="Loop Clusters" from_port="out 1" to_port="result 1"/>
<connect from_op="Loop Clusters" from_port="out 2" to_port="result 2"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
<portSpacing port="sink_result 3" spacing="0"/>
</process>
</operator>
</process>
Tagged:
0
Answers
you can use the Remember / Recall operators to remember anything you want. You can also use the Collect operator to bundle several objects to one collection.
The Text output can be writen using either the Write to Text operator or more complex using the reporting Extension. Anyway I would store the RM objects, too...
Greetings,
Sebastian
could you please provide an example of this? I tried a few iterations, but I find that I cannot get my head around what operators, and in which order, I need.
to be honest: No I never had
Here's a process that will solve your problem. But keep in mind the process execution order when using side effects like Remember Recall...
Greetings,
Sebastian
Thank you for the code, it really help me on the way. Still, I do get an error when I apply it to my own process:
When the "Recall" within the "Loop clusters" operator is executed for the first time I get:
"No object with the name m1 was found during retrieval from the object store"
This does not happen when I try you example, but my own code & data is of identical layout (and not parallelized, for example). Why cannot I find the "m1" object, or... why don't your code give this same error? Should the m1 object me initialized somehow? I have the "m1" name correct, as a matter of fact, I copy-pasted all your operators to my original project.
//frankie
this is probably because of execution order. In my process there are the two operators Collect(2) and Remember(2) that are executed BEFORE Loop Clusters. You can display the execution order with one of the buttons in the top right corner of the Process View.
This first remember ensures that there's an empty collection remembered that can be recalled inside the loop.
Greetings,
Sebastian
Why is it that I can run a process of say 500 000 samples without problems, but with this addition (the recalls/collects/remembers...) get this error message with about 1000 samples (with a loop of 8 categories)?! Should I free up memory at some point in the loop?
//frankie
Looks like a similar issue: http://rapid-i.com/rapidforum/index.php/topic,2714.0.html
without the process it is difficult to give any hints. Please provide the process together with your questions
Greetings,
Sebastian