About the Normalize operator.

Rafa

I have a simple question about the Normalize operator.

Suppose this

• I have a dataset (m1 examples, n attributes)
• I normalize them and I get two things: the normalized data and a normalization model.
• To the original data I add m2 new examples.
• To this augmented set of data (m1 + m2 examples) I apply the normalization model obtained previously.

Two things must happen:

1. The new m2 data have been normalized with respect to a model in which they were not present and consequently they do not correspond exactly to what would have been obtained when considering means and variances of the whole set (old plus new)
2. The m1 normalized values from the initial examples must be identical in both results.

Is this correct?

 Thanks!!!!!!!!!!

BalazsBarany

Hi @Rafa, 

if I understand your question correctly, this is what you're doing:

1. Build a normalization model from m1. This gives you the parameters for mean and standard deviation and transforms m1.
2. You combine m1 (original) and m2 and instead of creating a new normalization model (which would lead to a new calculation of mean and std.dev.) you apply the existing model built from m1 to the combination. 
3. You compare identical examples (e. g. based on some ID) between the two normalization results and they are not equal.

This is what I did in this example process, and they *are* identical. 

<?xml version="1.0" encoding="UTF-8"?><process version="9.8.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.8.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="-1"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.8.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="112" y="34">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="9.8.001" expanded="true" height="103" name="Filter Examples" width="90" x="246" y="34">
        <parameter key="parameter_expression" value=""/>
        <parameter key="condition_class" value="custom_filters"/>
        <parameter key="invert_filter" value="false"/>
        <list key="filters_list">
          <parameter key="filters_entry_key" value="label.does_not_equal.Iris-virginica"/>
        </list>
        <parameter key="filters_logic_and" value="true"/>
        <parameter key="filters_check_metadata" value="true"/>
      </operator>
      <operator activated="true" class="normalize" compatibility="9.8.001" expanded="true" height="103" name="Normalize M1" width="90" x="447" y="34">
        <parameter key="return_preprocessing_model" value="false"/>
        <parameter key="create_view" value="false"/>
        <parameter key="attribute_filter_type" value="all"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="numeric"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="real"/>
        <parameter key="block_type" value="value_series"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_series_end"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="method" value="Z-transformation"/>
        <parameter key="min" value="0.0"/>
        <parameter key="max" value="1.0"/>
        <parameter key="allow_negative_values" value="false"/>
      </operator>
      <operator activated="true" class="apply_model" compatibility="9.8.001" expanded="true" height="82" name="Apply normalization on original" width="90" x="514" y="187">
        <list key="application_parameters"/>
        <parameter key="create_view" value="false"/>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_op="Normalize M1" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="original" to_op="Apply normalization on original" to_port="unlabelled data"/>
      <connect from_op="Normalize M1" from_port="example set output" to_port="result 1"/>
      <connect from_op="Normalize M1" from_port="preprocessing model" to_op="Apply normalization on original" to_port="model"/>
      <connect from_op="Apply normalization on original" from_port="labelled data" to_port="result 3"/>
      <connect from_op="Apply normalization on original" from_port="model" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
    </process>
  </operator>
</process>

Would you like to share your process so I can check if it is built in the same way? The expected result of applying the model built from M1 to the (original, unmodified) M1 examples is to receive the same result.

Best regards,

Balázs

Rafa

Add this:

Normalize method  =  z-transformation.

It`s clear, answer to 2 in my first post is NOT.

So, the new question is, WHY NOT???

Thanks again!!

R