"Time series filtering / intervals"

Alexus

Hey there,

I would like to know if I can prepare time series (financial, i.e. eurusd) in the following ways in rapidminer:

1. from all data, i.e. extract all mondays and make a new table
2. from all data, i.e. extract only 8:12am mondays and make a new table
3. given a table with dates and times, for example 10 datasets that contain different dates and times, rapidminer should then look at another table with finanical (tick) data and make a new table which contains only data starting with the dates from the first table, plus X hours afterwards.

and similar tasks. In short, I would like to make new time series from (tick)data with the parameters I choose, and/or from dates from other tables.

Is that possible with rapidminer?

And another question: Whats the technical term for such actions?

Alexus

And if so, which version of rapid miner is capable of doing that? Is the community edition enough?

wessel

Hey,

With high probability, yes to all questions.
Here is a demo.

Best regards,

Wessel



<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.006">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.006" expanded="true" name="Process">
    <process expanded="true" height="392" width="165">
      <operator activated="true" class="generate_sales_data" compatibility="5.2.006" expanded="true" height="60" name="Generate Sales Data" width="90" x="45" y="30"/>
      <operator activated="true" class="date_to_numerical" compatibility="5.2.006" expanded="true" height="76" name="Date to Numerical" width="90" x="200" y="29">
        <parameter key="attribute_name" value="date"/>
        <parameter key="time_unit" value="day"/>
        <parameter key="day_relative_to" value="week"/>
        <parameter key="keep_old_attribute" value="true"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="5.2.006" expanded="true" height="76" name="Filter Examples" width="90" x="324" y="27">
        <parameter key="condition_class" value="attribute_value_filter"/>
        <parameter key="parameter_string" value="date_day=1"/>
      </operator>
      <connect from_op="Generate Sales Data" from_port="output" to_op="Date to Numerical" to_port="example set input"/>
      <connect from_op="Date to Numerical" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Alexus

Hey, thats great! You have NO idea how long I have been searching for software thats capable of that, unfortunately I did not know what to search for exactly...

Is there a tutorial I can look at, that you know, to get me started?

Alexus

And whats the technical term for this kind of operations that I want to do?

wessel

There is no technical term that I am aware off.
The best definition I can come up with is:
http://en.wikipedia.org/wiki/Unix_time

All these operations boil down to the basics of converting a date back to an integer holding the number of milliseconds after 1970.
E.g. http://www.vogella.com/articles/JavaDateTimeAPI/article.html

Best regards,

Wessel

Alexus

I don't quite get it - how does i.e. extracting all mondays out of a time series that contains monday to friday into a new time series that only contains mondays relate to unix time?

Alexus

I don't know if we understand each other? in MYSQL terms I would try a command like (not a working example I guess):

select * from table where date = monday and time between 8-12

or something along those lines. I have not found out yet how to do similar things in rapidminer

wessel

Alexus wrote:
I don't quite get it - how does i.e. extracting all mondays out of a time series that contains monday to friday into a new time series that only contains mondays relate to unix time?

To make this calculations, Rapid MIner uses code that is provided in any standard Java Installation.
E.g. http://docs.oracle.com/javase/1.4.2/docs/api/java/util/Calendar.html

Think about it, how would you implement this code if you had to do it from scratch?
For a lot of stuff milliseconds are really useful.
If you represent time as an integer holding the number of milliseconds after 1970 you can simple compare dates using the ">" operator.
And you can do stuff like:
if A = Monday
then so is A+=7 *24 * 60 * 60 * 10000

Alexus

I see. Thanks for your explanation. I conclude from your examples that its not yet possible to to this kind of task via the gui, right? If I have to do this via coding(i.e. in java), then I think preparing the data via SQL might be the easier way.

wessel

I do not agree with your conclusion.
Did my first example not show you how to do this within Rapid Miner?
And then I tried to explain you the underlying methods that make it work.