"windowing a discontinuous data set"

pop

Hello,

I have a set of intraday data over several days. I am using the windowing operator on it.
The window size is small with regard to the number of intraday data so most of the windowed examples are made of data from the same day, but this operator will, as expected,  create some examples that include data from 2 different days.
I would like to remove those examples as they make no sense.
Any idea for creating this kind of filter??

Thank you very much for your help.

haddock

Hi there Pop,

You can probably get what you want by using the "Date to Numerical" operator, which can strip the components from a date_time attribute. In the following example data is filtered by the minute of the timestamp, leaving only those from a quarter past the hour.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="258" width="915">
      <operator activated="true" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="30" y="56">
        <parameter key="repository_entry" value="Past_Deltas"/>
      </operator>
      <operator activated="true" class="nominal_to_date" expanded="true" height="76" name="Nominal to Date" width="90" x="179" y="30">
        <parameter key="attribute_name" value="DateTime"/>
        <parameter key="date_type" value="date_time"/>
        <parameter key="date_format" value="yyyy-MM-dd hh:mm:ss.0"/>
      </operator>
      <operator activated="true" class="date_to_numerical" expanded="true" height="76" name="Date to Numerical" width="90" x="313" y="30">
        <parameter key="attribute_name" value="DateTime"/>
        <parameter key="time_unit" value="minute"/>
        <parameter key="keep_old_attribute" value="true"/>
      </operator>
      <operator activated="true" class="filter_examples" expanded="true" height="76" name="Filter Examples" width="90" x="514" y="30">
        <parameter key="condition_class" value="attribute_value_filter"/>
        <parameter key="parameter_string" value="DateTime_minute = 15"/>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Nominal to Date" to_port="example set input"/>
      <connect from_op="Nominal to Date" from_port="example set output" to_op="Date to Numerical" to_port="example set input"/>
      <connect from_op="Date to Numerical" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Have fun  8)

pop

Hello Haddock,
Thank you very much for your reply.
I was thinking about something like this:
before windowing I have one attribute called date.
After windowing I have attributes date-99, date-98,...,date-0.
I would basically need to create a filter that keep the example if all values of date-99,date-98,....,date-0 are all the same and remove it otherwise.
Does it make any sense?
thank you

haddock

Hi Again,

You could use that operator to strip out the day from just the first and last timestamps, and trash examples where they weren't the same. Worth a try anyway..

Good luck !