Reverse of word list: how to generate a text attribute by repeating a value?

die_eike

Hello Community,

what I am trying to do is somehow the reverse of the aggregate operator or the word list. I have an example set (document) with a nominal attribute (word) and a numerical attribute (frequency=n). I want to generate a new text attribute with the word repeated n times, separated by space.

I can't find an operator to do this. I tried generate attributes, but I can't find a mathematical expression for what I want to do. As it seems to me rather simple a process, I hope I did not miss anything posted in the community already. I searched but I couldn't find anything.

Anybody who can help me?

BalazsBarany

Hi,

assuming that @rfuentealba and I interpret the problem correctly, here's an example process.

<?xml version="1.0" encoding="UTF-8"?><process version="9.2.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.2.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="utility:create_exampleset" compatibility="9.2.001" expanded="true" height="68" name="Create ExampleSet" width="90" x="45" y="34">
        <parameter key="generator_type" value="comma separated text"/>
        <parameter key="number_of_examples" value="100"/>
        <parameter key="use_stepsize" value="false"/>
        <list key="function_descriptions"/>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)"/>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="input_csv_text" value="word,freq&#10;abc,10&#10;def,5&#10;ghi,8"/>
        <parameter key="column_separator" value=","/>
        <parameter key="parse_all_as_nominal" value="false"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="extract_macro" compatibility="9.2.001" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="34">
        <parameter key="macro" value="maxFreq"/>
        <parameter key="macro_type" value="statistics"/>
        <parameter key="statistics" value="max"/>
        <parameter key="attribute_name" value="freq"/>
        <list key="additional_macros"/>
      </operator>
      <operator activated="true" class="generate_macro" compatibility="9.2.001" expanded="true" height="82" name="Generate Macro" width="90" x="313" y="34">
        <list key="function_descriptions">
          <parameter key="maxFreq" value="replaceAll(%{maxFreq}, &quot;\\.0&quot;, &quot;&quot;)"/>
        </list>
      </operator>
      <operator activated="true" class="utility:create_exampleset" compatibility="9.2.001" expanded="true" height="68" name="Create ExampleSet (2)" width="90" x="112" y="136">
        <parameter key="generator_type" value="attribute functions"/>
        <parameter key="number_of_examples" value="%{maxFreq}"/>
        <parameter key="use_stepsize" value="false"/>
        <list key="function_descriptions"/>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)"/>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="column_separator" value=","/>
        <parameter key="parse_all_as_nominal" value="false"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="generate_id" compatibility="9.2.001" expanded="true" height="82" name="Generate ID" width="90" x="246" y="136">
        <parameter key="create_nominal_ids" value="false"/>
        <parameter key="offset" value="0"/>
      </operator>
      <operator activated="true" class="cartesian_product" compatibility="9.2.001" expanded="true" height="82" name="Cartesian" width="90" x="447" y="85">
        <parameter key="remove_double_attributes" value="false"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="9.2.001" expanded="true" height="103" name="Filter Examples" width="90" x="581" y="85">
        <parameter key="parameter_expression" value="id &lt;= freq"/>
        <parameter key="condition_class" value="expression"/>
        <parameter key="invert_filter" value="false"/>
        <list key="filters_list"/>
        <parameter key="filters_logic_and" value="true"/>
        <parameter key="filters_check_metadata" value="true"/>
      </operator>
      <operator activated="true" class="aggregate" compatibility="9.2.001" expanded="true" height="82" name="Aggregate" width="90" x="715" y="85">
        <parameter key="use_default_aggregation" value="false"/>
        <parameter key="attribute_filter_type" value="all"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="default_aggregation_function" value="average"/>
        <list key="aggregation_attributes">
          <parameter key="word" value="concatenation"/>
        </list>
        <parameter key="group_by_attributes" value="word|freq"/>
        <parameter key="count_all_combinations" value="false"/>
        <parameter key="only_distinct" value="false"/>
        <parameter key="ignore_missings" value="true"/>
      </operator>
      <operator activated="true" class="replace" compatibility="9.2.001" expanded="true" height="82" name="Replace" width="90" x="849" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="concat(word)"/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="nominal"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="file_path"/>
        <parameter key="block_type" value="single_value"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="single_value"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="replace_what" value="\|"/>
        <parameter key="replace_by" value=" "/>
      </operator>
      <connect from_op="Create ExampleSet" from_port="output" to_op="Extract Macro" to_port="example set"/>
      <connect from_op="Extract Macro" from_port="example set" to_op="Generate Macro" to_port="through 1"/>
      <connect from_op="Generate Macro" from_port="through 1" to_op="Cartesian" to_port="left"/>
      <connect from_op="Create ExampleSet (2)" from_port="output" to_op="Generate ID" to_port="example set input"/>
      <connect from_op="Generate ID" from_port="example set output" to_op="Cartesian" to_port="right"/>
      <connect from_op="Cartesian" from_port="join" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_op="Aggregate" to_port="example set input"/>
      <connect from_op="Aggregate" from_port="example set output" to_op="Replace" to_port="example set input"/>
      <connect from_op="Replace" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

The upper row generates some example data (you would put your own there). Extract Macro determines the highest number of repetitions. This creates a macro value like 10.0, so we have to change that to just 10 with Generate Macro.
The lower row generates an example set with as many rows as max(repetitions). It adds a series from 1 to max(repetitions) with Generate Id. 
Then we do a cartesian join, which combines all records from both sources, repeating the words as many times as there are repetitions. With Filter Examples we only keep the items with the Id being less than or equal to the number of repetitions. 
We then aggregate by word and repetitions, concatenating the word as many times as it is left after the filter.
The last operator is a Replace. The result was concatenated with pipes like this: abc|abc|abc. Here we change it to "abc abc abc" by replacing the | with a space. Feel free to change it.

This would be much easier if we had a simple "repeat" function in Generate Attributes.
We could also optimize the join by having <= based joins instead of just = based. (There's a building block for this out there.)

Regards,

Balázs

lionelderkrikor

Congratulations @BalazsBarany for solving this interesting challenge !

For those who are interested, it can be solved by a simple line of code in Python...

<?xml version="1.0" encoding="UTF-8"?><process version="9.5.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.5.000" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="utility:create_exampleset" compatibility="9.5.000" expanded="true" height="68" name="Create ExampleSet" width="90" x="112" y="85">
        <parameter key="generator_type" value="comma separated text"/>
        <parameter key="number_of_examples" value="100"/>
        <parameter key="use_stepsize" value="false"/>
        <list key="function_descriptions"/>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)"/>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="input_csv_text" value="word,freq&#10;test,2&#10;test_2,3&#10;test_3,4"/>
        <parameter key="column_separator" value=","/>
        <parameter key="parse_all_as_nominal" value="false"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="python_scripting:execute_python" compatibility="9.3.001" expanded="true" height="103" name="Execute Python" width="90" x="313" y="85">
        <parameter key="script" value="import pandas as pd&#10;&#10;# rm_main is a mandatory function, &#10;# the number of arguments has to be the number of input ports (can be none)&#10;def rm_main(data):&#10;&#10;   &#10;  for i in range(len(data)) :&#10;    &#10;    data.loc[i,'generated'] = &quot; &quot;.join(data.loc[i,'word'] for k in range(data.loc[i,'freq']))&#10;&#10;  return data"/>
        <parameter key="notebook_cell_tag_filter" value=""/>
        <parameter key="use_default_python" value="true"/>
        <parameter key="package_manager" value="conda (anaconda)"/>
      </operator>
      <connect from_op="Create ExampleSet" from_port="output" to_op="Execute Python" to_port="input 1"/>
      <connect from_op="Execute Python" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Regards,

Lionel

 

Telcontar120

Perhaps you could provide a pre and post-output sample dataset?  I am not quite envisioning exactly what you want to produce here or the exact format of your original data.

rfuentealba

Let me understand:

You have this dataset (word and freq) and want the "generated" attribute?:

na na na na <code>na na na na na na na na word | freq | generated<br>--------------------------------------------------------------<br>a    | 5    | a a a a a<br>ze   | 3    | ze ze ze<br>nini | 2    | nini nini nini<br>na   | 16   | na na na na 

Do you want it with or without spaces?

All the best,

Rod.

Telcontar120

Wow, these other unicorns have already solved the problem!  That was fast, great work guys!

die_eike

Dear Unicorns,

thanks for your help!

@rfuentealba, this is exactly what I wanted.

@BalazsBarany, thanks for the process and the detailed description. I will try it out for my data now and report if it has done the trick for me.

Just one question beforehand, would it make sense to have an operator in Rapidminer which performs such an operation? I don't know, but it could be useful in several cases to have an operator which repeats the (nominal or other) values of an attribute with a given frequency (constant, or indicated by a value in another attribute).

Best,

Eike

die_eike

Dear @BalazsBarany,

your process is giving me a "Required parameter missing" and indeed you did not specify the function description in the lower "Create Example Set". I am a bit at a loss here. Could you help me out?

Best,

Eike

P.S.: Sorry for my earlier question, you just said it here and I overlooked it:

"This would be much easier if we had a simple "repeat" function in Generate Attributes."

BalazsBarany

Hi,

I was in an older Studio and there it worked.
You can add a dummy function in "Edit List...": attribute name = x, function expressions = 1.

Regards,

Balázs

die_eike

Hello @BalazsBarany and others,

sorry for the late replay. I had to work on another project last couple of days and to delay this one. Your process works now and I was able to adjust it for my data  

Thanks a lot!