Dates / Date_parse / Automodel Preprocessing

Noel

Hi All-
I have a process, exported from AutoModel, that decomposes all dates into attributes like "date_name:month_of_year", "date_name:year", "date_name:day_of_month", etc. I'm trying to recoup the original dates from the underlying data (thinking this would be "easy" to do by stringing a bunch of concat() s together and feeding them to date_parse()... e.g. concat(concat(date_name:month_of_year,"/"),date_name:day_of_month))...

This is turning out to be more difficult than anticipated. Any suggestions for a different approach?

Thanks, Noel

tftemme

Hi @Noel

Concerning the problem with the parsing. I am not sure what exactly goes wrong. So I am assuming here that maybe the parsing of the numerical month_of, day_of, year_of attributes for the concat is not working. So you have to add a str() function around the attributes. Also I would use the date_parse_custom function to explicitly specify the format of the string to be parsed into the date.

Another possibility is to use the Nominal to Date operator (where you also have to specify the date format of the nominal attribute). 

I attached an xml file where I created random 'month of year', 'day of year', 'year' attributes and then use generate attribute to concat them to the date string. In the Generate Attribute I directly use this date string with date_parse_custom to convert it into a date_time attribute. For demonstrating I am doing the same again with the Nominal to Date operator.

Concerning the attribute names with white spaces. You just need to put them into squared brackets (can see this in the example xml below), then the function expression can correctly identify them. When you use the expression editor (small calculator symbol) and insert an attribute in your function by using the list in the bottom right, the squared brackets are inserted automatically, if needed.

Hopes this helps. Best regards,
Fabian

<process version="9.3.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.3.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="utility:create_exampleset" compatibility="9.3.001" expanded="true" height="68" name="Create ExampleSet" width="90" x="380" y="136">
        <parameter key="generator_type" value="attribute functions"/>
        <parameter key="number_of_examples" value="100"/>
        <parameter key="use_stepsize" value="false"/>
        <list key="function_descriptions">
          <parameter key="month of year" value="round(1+11*rand())"/>
          <parameter key="day of year" value="round(1+29*rand())"/>
          <parameter key="year" value="round(2016 + 2*rand())"/>
        </list>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)"/>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="column_separator" value=","/>
        <parameter key="parse_all_as_nominal" value="false"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="generate_attributes" compatibility="9.3.001" expanded="true" height="82" name="Generate Attributes" width="90" x="514" y="136">
        <list key="function_descriptions">
          <parameter key="date" value="concat(str([month of year]),&quot;/&quot;,str([day of year]),&quot;/&quot;,str(year))"/>
          <parameter key="date_parsed" value="date_parse_custom(date,&quot;MM/dd/yyyy&quot;)"/>
        </list>
        <parameter key="keep_all" value="true"/>
      </operator>
      <operator activated="true" class="nominal_to_date" compatibility="9.3.001" expanded="true" height="82" name="Nominal to Date" width="90" x="648" y="136">
        <parameter key="attribute_name" value="date"/>
        <parameter key="date_type" value="date"/>
        <parameter key="date_format" value="MM/dd/yyyy"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="locale" value="English (United States)"/>
        <parameter key="keep_old_attribute" value="true"/>
      </operator>
      <connect from_op="Create ExampleSet" from_port="output" to_op="Generate Attributes" to_port="example set input"/>
      <connect from_op="Generate Attributes" from_port="example set output" to_op="Nominal to Date" to_port="example set input"/>
      <connect from_op="Nominal to Date" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

MartinLiebig

Hi,

can't you just join the original dates?

Cheers,

Martin

Noel

Hi Martin- I thought of doing that, but what key would I use for the joining? I was using the date as an index...

Noel

On an unrelated note, I've noticed that "function expressions" (like in Generate Macro or Generate Attributes) don't seem to like attribute names with white space in them. In the past, I've Renamed or Generated Copies to remove the spaces. Perhaps single or double quotes can resolve this issue, but I've never tried...

Noel

Thanks @tftemme !