The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

"Remove Samples With Many Missing Values"

dragoljubdragoljub Member Posts: 241 Contributor II
edited June 2019 in Help
I would like to filter examples that have more than a certain number of missing values, so I can then apply attribute filtering and not loose my entire dataset.

Any way to do this, the filter operator currently filters samples with any missing values.

Thanks,
-Gagi
Tagged:

Answers

  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi,
    you have already posted in another thread that was presenting the solution for this problem. Why did you open just another one?

    Greetings,
      Sebastian
  • dragoljubdragoljub Member Posts: 241 Contributor II
    Actually the other thread was about removing 'attributes'. I want to remove 'examples'.

    For example: Filter all examples that have more than 10 missing values for the current set of attributes.

    This way we can do data shrinking from both dimensions. I have a lot of samples so this would be useful.  ;D

    Thanks,
    -Gagi
  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi,
    excuse me, I should have read more carefully. Especially after I was surprised, that suddenly you are  starting to spam the forum. Well, back to your original questions:
    The computer scientist in me wants to answer: Transpose the ExampleSet and solve the old attribute problem, transpose it again and the things are cleared. But it doesn't seem to be appropriate :) So let's see...

    After I tried quite some time, I didn't come up with a really satisfying solution. Here's the ground idea: Count each missing value into a new attribute and then filter the examples accordingly. Here's the process for counting the missings: 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.0">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" expanded="true" name="Process">
        <process expanded="true" height="251" width="748">
          <operator activated="true" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
            <parameter key="repository_entry" value="//Samples/data/Labor-Negotiations"/>
          </operator>
          <operator activated="true" class="rename_by_replacing" expanded="true" height="76" name="Rename by Replacing" width="90" x="179" y="30">
            <parameter key="replace_what" value="-"/>
            <parameter key="replace_by" value="_"/>
          </operator>
          <operator activated="true" class="generate_empty_attribute" expanded="true" height="76" name="Generate Empty Attribute" width="90" x="313" y="30">
            <parameter key="name" value="numberOfMissings"/>
          </operator>
          <operator activated="true" class="replace_missing_values" expanded="true" height="94" name="Replace Missing Values" width="90" x="447" y="30">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="numberOfMissings"/>
            <parameter key="default" value="zero"/>
            <list key="columns"/>
          </operator>
          <operator activated="true" class="loop_attributes" expanded="true" height="60" name="Loop Attributes" width="90" x="581" y="30">
            <parameter key="iteration_macro" value="currentAttribute"/>
            <process expanded="true" height="581" width="764">
              <operator activated="true" class="generate_attributes" expanded="true" height="76" name="Generate Attributes" width="90" x="45" y="30">
                <list key="function_descriptions">
                  <parameter key="newNumberOfMissings" value="numberOfMissings + if(%{currentAttribute}!=%{currentAttribute},1,0)"/>
                </list>
              </operator>
              <operator activated="true" class="select_attributes" expanded="true" height="76" name="Select Attributes" width="90" x="179" y="30">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="numberOfMissings"/>
                <parameter key="invert_selection" value="true"/>
              </operator>
              <operator activated="true" class="rename" expanded="true" height="76" name="Rename" width="90" x="313" y="30">
                <parameter key="old_name" value="newNumberOfMissings"/>
                <parameter key="new_name" value="numberOfMissings"/>
              </operator>
              <connect from_port="example set" to_op="Generate Attributes" to_port="example set input"/>
              <connect from_op="Generate Attributes" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
              <connect from_op="Select Attributes" from_port="example set output" to_op="Rename" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_port="example set"/>
              <portSpacing port="source_example set" spacing="0"/>
              <portSpacing port="sink_example set" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Retrieve" from_port="output" to_op="Rename by Replacing" to_port="example set input"/>
          <connect from_op="Rename by Replacing" from_port="example set output" to_op="Generate Empty Attribute" to_port="example set input"/>
          <connect from_op="Generate Empty Attribute" from_port="example set output" to_op="Replace Missing Values" to_port="example set input"/>
          <connect from_op="Replace Missing Values" from_port="example set output" to_op="Loop Attributes" to_port="example set"/>
          <connect from_op="Loop Attributes" from_port="example set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    Why this isn't satisfying? Because it only works on real valued attributes. So if this is a solution depends on what you need...
    This again is a valuable feature, we should include in future versions. Please add it to the bug tracker as a feature request.

    Greetings,
      Sebastian
  • ui3oui3o Member Posts: 9 Contributor II
    Hi Sebastian,

    can you share a word on the if(%{currentAttribute}!=%{currentAttribute}...) statement. Seems to be very helpful. What's behind it?
    Thx


    ui3o
  • dragoljubdragoljub Member Posts: 241 Contributor II
    Thanks Sebastian,

    I was able to remove attributes and then filter examples for a similar effect.

    The if statement looks like it follows this syntax: if(statement, true_action, false_action), correct?

    Is there a list of supported functions in RM that we can use in expressions?

    Thanks Again,
    -Gagi
  • haddockhaddock Member Posts: 849 Maven
    Greets Seb,

    Late night waiting for US calls so I thought I'd take a look at making a missing value count column, and then bolting that on to the original data, like this.. 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.0">
      <context>
        <input/>
        <output/>
        <macros>
          <macro>
            <key/>
            <value/>
          </macro>
        </macros>
      </context>
      <operator activated="true" class="process" expanded="true" name="Process">
        <process expanded="true" height="353" width="808">
          <operator activated="true" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
            <parameter key="repository_entry" value="//Samples/data/Labor-Negotiations"/>
          </operator>
          <operator activated="true" class="multiply" expanded="true" height="94" name="Multiply" width="90" x="179" y="30"/>
          <operator activated="true" class="generate_id" expanded="true" height="76" name="Original Data" width="90" x="380" y="120"/>
          <operator activated="true" class="subprocess" expanded="true" height="76" name="Missing Count Column" width="90" x="380" y="30">
            <process expanded="true" height="353" width="748">
              <operator activated="true" class="transpose" expanded="true" height="76" name="Transpose" width="90" x="180" y="30"/>
              <operator activated="true" class="loop_attributes" expanded="true" height="60" name="Loop Attributes" width="90" x="315" y="30">
                <process expanded="true" height="353" width="808">
                  <operator activated="true" class="extract_macro" expanded="true" height="60" name="Count unknowns" width="90" x="45" y="30">
                    <parameter key="macro" value="Count"/>
                    <parameter key="macro_type" value="statistics"/>
                    <parameter key="statistics" value="unknown"/>
                    <parameter key="attribute_name" value="%{loop_attribute}"/>
                    <parameter key="example_index" value="%{b}"/>
                  </operator>
                  <operator activated="true" class="provide_macro_as_log_value" expanded="true" height="76" name="Provide Count as Log Value" width="90" x="180" y="30">
                    <parameter key="macro_name" value="Total"/>
                  </operator>
                  <operator activated="true" class="log" expanded="true" height="76" name="Log" width="90" x="494" y="30">
                    <list key="log">
                      <parameter key="Missings" value="operator.Count unknowns.value.macro_value"/>
                    </list>
                  </operator>
                  <connect from_port="example set" to_op="Count unknowns" to_port="example set"/>
                  <connect from_op="Count unknowns" from_port="example set" to_op="Provide Count as Log Value" to_port="through 1"/>
                  <connect from_op="Provide Count as Log Value" from_port="through 1" to_op="Log" to_port="through 1"/>
                  <connect from_op="Log" from_port="through 1" to_port="example set"/>
                  <portSpacing port="source_example set" spacing="0"/>
                  <portSpacing port="sink_example set" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="log_to_data" expanded="true" height="94" name="Log to Data" width="90" x="179" y="165"/>
              <operator activated="true" class="parse_numbers" expanded="true" height="76" name="Parse Numbers" width="90" x="313" y="165"/>
              <operator activated="true" class="generate_id" expanded="true" height="76" name="Generate ID" width="90" x="447" y="165"/>
              <connect from_port="in 1" to_op="Transpose" to_port="example set input"/>
              <connect from_op="Transpose" from_port="example set output" to_op="Loop Attributes" to_port="example set"/>
              <connect from_op="Loop Attributes" from_port="example set" to_op="Log to Data" to_port="through 1"/>
              <connect from_op="Log to Data" from_port="exampleSet" to_op="Parse Numbers" to_port="example set input"/>
              <connect from_op="Parse Numbers" from_port="example set output" to_op="Generate ID" to_port="example set input"/>
              <connect from_op="Generate ID" from_port="example set output" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="join" expanded="true" height="76" name="Data plus Missing Count" width="90" x="565" y="31"/>
          <connect from_op="Retrieve" from_port="output" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Missing Count Column" to_port="in 1"/>
          <connect from_op="Multiply" from_port="output 2" to_op="Original Data" to_port="example set input"/>
          <connect from_op="Original Data" from_port="example set output" to_op="Data plus Missing Count" to_port="right"/>
          <connect from_op="Missing Count Column" from_port="out 1" to_op="Data plus Missing Count" to_port="left"/>
          <connect from_op="Data plus Missing Count" from_port="join" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    The computer scientist in me wants to answer: Transpose the ExampleSet and solve the old attribute problem, transpose it again and the things are cleared. But it doesn't seem to be appropriate
    Be wary of the part of you which doubted the appropriateness of transposition  :D

  • dragoljubdragoljub Member Posts: 241 Contributor II
    Thanks Haddock,

    This will be very useful.

    The transpose is quite smart and lets you play within the confines of the RM space.  ;)

    -Gagi
Sign In or Register to comment.