The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Time Series data issue

xiangw02xiangw02 Member Posts: 10 Learner III
edited December 2018 in Help

Hi I'm new to Time Series ARIMA model. when I train this model with the data below. i got this error: Indices of the time series are not strictly monotonic increasing. I try to forcast energy usage for every 15 mins.  please advise.

Time                                                                                                                                                     Usage

2/20/2018 1:00 PM 1,327.68
2/20/2018 1:15 PM 1,330.56
2/20/2018 1:30 PM 1,351.68
2/20/2018 1:45 PM 1,350.72
2/20/2018 10:00 AM 1,302.72
2/20/2018 10:00 PM 1,140.48
2/20/2018 10:15 AM 1,305.60
2/20/2018 10:15 PM 1,131.84
2/20/2018 10:30 AM 1,305.60
Tagged:

Best Answer

  • tftemmetftemme Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, RMResearcher, Member Posts: 164 RM Research
    Solution Accepted
    Hi @Montse

    Yes your data is not sorted correctly. You first have all years for January, then all years for February (so your 6 th. example is earlier than your 5 th. example). Hence the Arima Trainer correctly throws the error that your date indices are not montonic increasing. Please first sort your data after the date attribute using for example the Sort operator

    Best regards,
    Fabian

Answers

  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @xiangw02 welcome to the community! Some quick recommendations for you:
    • Post your XML process here in this thread (see https://youtu.be/KkgB5QXWXJ8 and "Read Before Posting" on right when you reply)
    • Attach your dataset if possible (use a fictionalized version if there are privacy concerns)
    • Make sure you have all necessary extensions installed (see https://youtu.be/pjBqG3xtXx4)

    Scott

  • xiangw02xiangw02 Member Posts: 10 Learner III 
    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
      <operator activated="true" class="timeseries:arima_trainer" compatibility="0.1.002" expanded="true" height="103" name="ARIMA Trainer" width="90" x="179" y="85">
        <parameter key="time_series_attribute" value="kW"/>
        <parameter key="has_indices" value="true"/>
        <parameter key="indices_attribute" value="Period"/>
        <parameter key="plithiumorder_of_the_autoregressive_model" value="1"/>
        <parameter key="dlithiumdegree_of_differencing" value="0"/>
        <parameter key="qlithiumorder_of_the_moving-average_model" value="1"/>
        <parameter key="estimate_constant" value="true"/>
      </operator>
    </process>
  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi @xiangw02

    Can you verify your process, your XML is broken...

     

    Regards,

     

    Lionel

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi again @xiangw02,

     

    OK, I can visualize your data in RapidMiner,

     

    I understandand you want to forecast Energy (kW) in the future.....

     

    but can you be more precise (in the last day, week, month......??????)

     

     

    Regards,

     

     

    Lionel 

  • xiangw02xiangw02 Member Posts: 10 Learner III

    next 24 hours but by every 15 mins interval.

  • xiangw02xiangw02 Member Posts: 10 Learner III

    any one can help this?

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi again @xiangw02,

     

    I'm working on a process for your project.

    I will have an answer element tomorrow.

     

    Regards,

     

     

    Lionel

     

  • xiangw02xiangw02 Member Posts: 10 Learner III

    @lionelderkrikor, did you figure out a solution?  thanks!

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi @xiangw02 

     

    I'm not a specialist of ARIMA, but you can find here a process to forecast the Energy using

    the kNN model (you can test different models - SVM, Neural Nets etc. -  inside the Validation operator in order to find the best performance).

     

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
      <context>
        <input/>
        <output/>
        <macros>
          <macro>
            <key>futureMonths</key>
            <value>15</value>
          </macro>
          <macro>
            <key>horizon</key>
            <value>1</value>
          </macro>
          <macro>
            <key>windowSize</key>
            <value>6</value>
          </macro>
        </macros>
      </context>
      <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="read_csv" compatibility="8.1.001" expanded="true" height="68" name="Read CSV" width="90" x="45" y="136">
            <parameter key="csv_file" value="C:\Users\Lionel\Documents\Formations_DataScience\Rapidminer\Tests_Rapidminer\Energy_forecast\Copy of HJD_Electricity_Interval_3.16-3.18 (002).csv"/>
            <parameter key="column_separators" value=","/>
            <parameter key="grouped_digits" value="true"/>
            <parameter key="date_format" value="MM/dd/yyyy"/>
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <parameter key="encoding" value="windows-1252"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="Period.true.date_time.id"/>
              <parameter key="1" value="kW.true.polynominal.attribute"/>
            </list>
          </operator>
          <operator activated="true" class="sort" compatibility="8.1.001" expanded="true" height="82" name="Sort" width="90" x="179" y="136">
            <parameter key="attribute_name" value="Period"/>
          </operator>
          <operator activated="true" class="rename" compatibility="8.1.001" expanded="true" height="82" name="Rename (2)" width="90" x="313" y="136">
            <parameter key="old_name" value="Period"/>
            <parameter key="new_name" value="DateTime"/>
            <list key="rename_additional_attributes"/>
          </operator>
          <operator activated="true" class="subprocess" compatibility="8.1.001" expanded="true" height="82" name="Set Predictions_Params" width="90" x="447" y="136">
            <process expanded="true">
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Window_Size" width="90" x="45" y="34">
                <parameter key="macro" value="WindowSize"/>
                <parameter key="value" value="40"/>
              </operator>
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Horizon" width="90" x="179" y="34">
                <parameter key="macro" value="horizon"/>
                <parameter key="value" value="1"/>
              </operator>
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Future_Day" width="90" x="313" y="34">
                <parameter key="macro" value="futureDay"/>
                <parameter key="value" value="96"/>
              </operator>
              <connect from_port="in 1" to_op="Set Window_Size" to_port="through 1"/>
              <connect from_op="Set Window_Size" from_port="through 1" to_op="Set Horizon" to_port="through 1"/>
              <connect from_op="Set Horizon" from_port="through 1" to_op="Set Future_Day" to_port="through 1"/>
              <connect from_op="Set Future_Day" from_port="through 1" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role" width="90" x="581" y="136">
            <parameter key="attribute_name" value="DateTime"/>
            <parameter key="target_role" value="id"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes" width="90" x="715" y="136">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attributes" value="kW"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="6.4.000" expanded="true" height="103" name="Filter Examples" width="90" x="916" y="85">
            <parameter key="condition_class" value="no_missing_attributes"/>
            <list key="filters_list"/>
          </operator>
          <operator activated="true" class="parse_numbers" compatibility="8.1.001" expanded="true" height="82" name="Parse Numbers" width="90" x="1050" y="136">
            <parameter key="grouped_digits" value="true"/>
          </operator>
          <operator activated="true" class="series:windowing" compatibility="5.2.000" expanded="true" height="82" name="Windowing for Training" width="90" x="1184" y="136">
            <parameter key="window_size" value="%{WindowSize}"/>
            <parameter key="create_label" value="true"/>
            <parameter key="label_attribute" value="kW"/>
            <parameter key="horizon" value="%{horizon}"/>
          </operator>
          <operator activated="true" class="series:sliding_window_validation" compatibility="7.4.000" expanded="true" height="124" name="Validation" width="90" x="1318" y="34">
            <parameter key="training_window_width" value="40"/>
            <parameter key="test_window_width" value="20"/>
            <process expanded="true">
              <operator activated="true" class="k_nn" compatibility="8.1.001" expanded="true" height="82" name="k-NN" width="90" x="179" y="34"/>
              <connect from_port="training" to_op="k-NN" to_port="training set"/>
              <connect from_op="k-NN" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="8.1.001" expanded="true" height="82" name="Apply Model (2)" width="90" x="112" y="34">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_regression" compatibility="8.1.001" expanded="true" height="82" name="Performance" width="90" x="246" y="34"/>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="series:windowing" compatibility="5.2.000" expanded="true" height="82" name="Windowing for Application" width="90" x="1318" y="187">
            <parameter key="window_size" value="%{WindowSize}"/>
            <parameter key="label_attribute" value="inputYt"/>
          </operator>
          <operator activated="true" class="extract_macro" compatibility="8.1.001" expanded="true" height="68" name="Extract Example Count" width="90" x="1519" y="187">
            <parameter key="macro" value="exampleCount"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="filter_example_range" compatibility="8.1.001" expanded="true" height="82" name="Filter Example Range" width="90" x="1653" y="187">
            <parameter key="first_example" value="%{exampleCount}"/>
            <parameter key="last_example" value="%{exampleCount}"/>
          </operator>
          <operator activated="true" class="remember" compatibility="8.1.001" expanded="true" height="68" name="Remember" width="90" x="1787" y="187">
            <parameter key="name" value="data"/>
          </operator>
          <operator activated="true" class="loop" compatibility="8.1.001" expanded="true" height="82" name="Loop" width="90" x="1519" y="34">
            <parameter key="iterations" value="%{futureDay}"/>
            <process expanded="true">
              <operator activated="true" class="recall" compatibility="8.1.001" expanded="true" height="68" name="Recall" width="90" x="45" y="136">
                <parameter key="name" value="data"/>
              </operator>
              <operator activated="true" class="apply_model" compatibility="7.1.001" expanded="true" height="82" name="Apply Model" width="90" x="179" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="multiply" compatibility="8.1.001" expanded="true" height="103" name="Multiply" width="90" x="447" y="30"/>
              <operator activated="true" class="materialize_data" compatibility="8.1.001" expanded="true" height="82" name="Materialize Data (2)" width="90" x="179" y="165"/>
              <operator activated="true" class="generate_attributes" compatibility="6.4.000" expanded="true" height="82" name="Increase Date (2)" width="90" x="313" y="165">
                <list key="function_descriptions">
                  <parameter key="DateTime" value="date_add(DateTime, 15, DATE_UNIT_MINUTE)"/>
                </list>
              </operator>
              <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role (2)" width="90" x="447" y="165">
                <parameter key="attribute_name" value="prediction(label)"/>
                <list key="set_additional_roles"/>
              </operator>
              <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes (3)" width="90" x="179" y="289">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="kW-39"/>
                <parameter key="invert_selection" value="true"/>
              </operator>
              <operator activated="true" class="rename" compatibility="8.1.001" expanded="true" height="82" name="Rename" width="90" x="313" y="289">
                <parameter key="old_name" value="kW-38"/>
                <parameter key="new_name" value="kW-39"/>
                <list key="rename_additional_attributes">
                  <parameter key="kW-37" value="kW-38"/>
                  <parameter key="kW-36" value="kW-37"/>
                  <parameter key="kW-35" value="kW-36"/>
                  <parameter key="kW-34" value="kW-35"/>
                  <parameter key="kW-33" value="kW-34"/>
                  <parameter key="kW-32" value="kW-33"/>
                  <parameter key="kW-31" value="kW-32"/>
                  <parameter key="kW-30" value="kW-31"/>
                  <parameter key="kW-29" value="kW-30"/>
                  <parameter key="kW-28" value="kW-29"/>
                  <parameter key="kW-27" value="kW-28"/>
                  <parameter key="kW-26" value="kW-27"/>
                  <parameter key="kW-25" value="kW-26"/>
                  <parameter key="kW-24" value="kW-25"/>
                  <parameter key="kW-23" value="kW-24"/>
                  <parameter key="kW-22" value="kW-23"/>
                  <parameter key="kW-21" value="kW-22"/>
                  <parameter key="kW-20" value="kW-21"/>
                  <parameter key="kW-19" value="kW-20"/>
                  <parameter key="kW-18" value="kW-19"/>
                  <parameter key="kW-17" value="kW-18"/>
                  <parameter key="kW-16" value="kW-17"/>
                  <parameter key="kW-15" value="kW-16"/>
                  <parameter key="kW-14" value="kW-15"/>
                  <parameter key="kW-13" value="kW-14"/>
                  <parameter key="kW-12" value="kW-13"/>
                  <parameter key="kW-11" value="kW-12"/>
                  <parameter key="kW-10" value="kW-11"/>
                  <parameter key="kW-9" value="kW-10"/>
                  <parameter key="kW-8" value="kW-9"/>
                  <parameter key="kW-7" value="kW-8"/>
                  <parameter key="kW-6" value="kW-7"/>
                  <parameter key="kW-5" value="kW-6"/>
                  <parameter key="kW-4" value="kW-5"/>
                  <parameter key="kW-3" value="kW-4"/>
                  <parameter key="kW-2" value="kW-3"/>
                  <parameter key="kW-1" value="kW-2"/>
                  <parameter key="kW-0" value="kW-1"/>
                  <parameter key="prediction(label)" value="kW-0"/>
                </list>
              </operator>
              <operator activated="true" class="remember" compatibility="8.1.001" expanded="true" height="68" name="Remember (2)" width="90" x="447" y="289">
                <parameter key="name" value="data"/>
              </operator>
              <connect from_port="input 1" to_op="Apply Model" to_port="model"/>
              <connect from_op="Recall" from_port="result" to_op="Apply Model" to_port="unlabelled data"/>
              <connect from_op="Apply Model" from_port="labelled data" to_op="Multiply" to_port="input"/>
              <connect from_op="Multiply" from_port="output 1" to_port="output 1"/>
              <connect from_op="Multiply" from_port="output 2" to_op="Materialize Data (2)" to_port="example set input"/>
              <connect from_op="Materialize Data (2)" from_port="example set output" to_op="Increase Date (2)" to_port="example set input"/>
              <connect from_op="Increase Date (2)" from_port="example set output" to_op="Set Role (2)" to_port="example set input"/>
              <connect from_op="Set Role (2)" from_port="example set output" to_op="Select Attributes (3)" to_port="example set input"/>
              <connect from_op="Select Attributes (3)" from_port="example set output" to_op="Rename" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_op="Remember (2)" to_port="store"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="8.1.001" expanded="true" height="82" name="Append" width="90" x="1653" y="34"/>
          <connect from_op="Read CSV" from_port="output" to_op="Sort" to_port="example set input"/>
          <connect from_op="Sort" from_port="example set output" to_op="Rename (2)" to_port="example set input"/>
          <connect from_op="Rename (2)" from_port="example set output" to_op="Set Predictions_Params" to_port="in 1"/>
          <connect from_op="Set Predictions_Params" from_port="out 1" to_op="Set Role" to_port="example set input"/>
          <connect from_op="Set Role" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_op="Parse Numbers" to_port="example set input"/>
          <connect from_op="Parse Numbers" from_port="example set output" to_op="Windowing for Training" to_port="example set input"/>
          <connect from_op="Windowing for Training" from_port="example set output" to_op="Validation" to_port="training"/>
          <connect from_op="Windowing for Training" from_port="original" to_op="Windowing for Application" to_port="example set input"/>
          <connect from_op="Validation" from_port="model" to_op="Loop" to_port="input 1"/>
          <connect from_op="Validation" from_port="averagable 1" to_port="result 2"/>
          <connect from_op="Windowing for Application" from_port="example set output" to_op="Extract Example Count" to_port="example set"/>
          <connect from_op="Extract Example Count" from_port="example set" to_op="Filter Example Range" to_port="example set input"/>
          <connect from_op="Filter Example Range" from_port="example set output" to_op="Remember" to_port="store"/>
          <connect from_op="Loop" from_port="output 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
        </process>
      </operator>
    </process>

    I hope it helps,

     

    Regards,

     

     

    Lionel

     

     

     

     

     

     

     

     

  • xiangw02xiangw02 Member Posts: 10 Learner III

    Thanks. @lionelderkrikor. my data goes to 3/9/2018. the forcasting date is 3/18/2018. where do i set to 3/10/2018?

  • xiangw02xiangw02 Member Posts: 10 Learner III

    BTW can someone help debug my ARIMA process?

    thanks

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi again @xiangw02,

     

    I think you have to preprocess your data, the time attribute (attribute Period) is not strictly increasing

    and it generate a bug in the Arima trainer operator.

    You have 2 times the same date-time in your dataset : 

     

    Energy_forecast.png

     

     

    I hope it helps

     

    Regards,

     

     

    Lionel

     

     

  • xiangw02xiangw02 Member Posts: 10 Learner III

    @lionelderkrikor thanks again.

    i reformat the date and Data Import wizard cannot format correct format. e.g. i attached the csv file for reference.

    what's correct date-time format Data Import wizard can take?

    2/20/18 18:15
  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi again @xiangw02,

     

    This process work but i don't know why  : the hours in the date_time attribute are not good displayed (always 12 : 00 :00 AM)

     

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="8.1.001" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="read_csv" compatibility="8.1.001" expanded="true" height="68" name="Read CSV" width="90" x="112" y="85">
            <parameter key="csv_file" value="C:\Users\Lionel\Documents\Formations_DataScience\Rapidminer\Tests_Rapidminer\Energy_forecast\Copy of HJD_Electricity_Interval_3.16-3.18 (002) (1).csv"/>
            <parameter key="column_separators" value=","/>
            <parameter key="grouped_digits" value="true"/>
            <parameter key="date_format" value="MM/dd/yyyy"/>
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <parameter key="encoding" value="windows-1252"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="Period.true.date.attribute"/>
              <parameter key="1" value="kW.true.real.attribute"/>
            </list>
          </operator>
          <operator activated="false" class="nominal_to_date" compatibility="8.1.001" expanded="true" height="82" name="Nominal to Date" width="90" x="313" y="238">
            <parameter key="attribute_name" value="Period"/>
            <parameter key="date_type" value="date_time"/>
            <parameter key="date_format" value="MM/dd/yyyy"/>
          </operator>
          <operator activated="false" class="parse_numbers" compatibility="8.1.001" expanded="true" height="82" name="Parse Numbers" width="90" x="447" y="238">
            <parameter key="grouped_digits" value="true"/>
          </operator>
          <operator activated="true" class="timeseries:arima_trainer" compatibility="0.1.002" expanded="true" height="103" name="ARIMA Trainer" width="90" x="581" y="85">
            <parameter key="time_series_attribute" value="kW"/>
            <parameter key="has_indices" value="true"/>
            <parameter key="indices_attribute" value="Period"/>
          </operator>
          <operator activated="true" class="timeseries:apply_forecast" compatibility="0.1.002" expanded="true" height="82" name="Apply Forecast" width="90" x="715" y="85"/>
          <connect from_op="Read CSV" from_port="output" to_op="ARIMA Trainer" to_port="example set"/>
          <connect from_op="ARIMA Trainer" from_port="forecast model" to_op="Apply Forecast" to_port="forecast model"/>
          <connect from_op="Apply Forecast" from_port="example set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>

    Regards,

     

     

    Lionel

  • xiangw02xiangw02 Member Posts: 10 Learner III

    @lionelderkrikor, can you look at the solution you provided?

     my data goes to 3/9/2018. the forcasting date is 3/18/2018. where do i set to 3/10/2018?

  • xiangw02xiangw02 Member Posts: 10 Learner III

    @lionelderkrikor, can you look at the solution you provided?

     my data goes to 3/9/2018. the forcasting date is 3/18/2018. where do i set to 3/10/2018?

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi @xiangw02,

     

    In deed, this is a strange behaviour.

    I will study that and will give you an answer element, i hope, soon .

     

    Regards,

     

    Lionel

     

  • lionelderkrikorlionelderkrikor RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn

    Hi again @xiangw02,

     

    I applied a "fix" to set the prediction to 3/10/2018.

     

    Here the new process : 

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
      <context>
        <input/>
        <output/>
        <macros>
          <macro>
            <key>futureMonths</key>
            <value>15</value>
          </macro>
          <macro>
            <key>horizon</key>
            <value>1</value>
          </macro>
          <macro>
            <key>windowSize</key>
            <value>6</value>
          </macro>
        </macros>
      </context>
      <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="read_csv" compatibility="8.1.001" expanded="true" height="68" name="Read CSV" width="90" x="45" y="136">
            <parameter key="csv_file" value="C:\Users\Lionel\Documents\Formations_DataScience\Rapidminer\Tests_Rapidminer\Energy_forecast\Copy of HJD_Electricity_Interval_3.16-3.18 (002).csv"/>
            <parameter key="column_separators" value=","/>
            <parameter key="grouped_digits" value="true"/>
            <parameter key="date_format" value="MM/dd/yyyy h:mm a"/>
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <parameter key="encoding" value="windows-1252"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="Period.true.date_time.id"/>
              <parameter key="1" value="kW.true.polynominal.attribute"/>
            </list>
          </operator>
          <operator activated="true" class="sort" compatibility="8.1.001" expanded="true" height="82" name="Sort" width="90" x="179" y="136">
            <parameter key="attribute_name" value="Period"/>
          </operator>
          <operator activated="true" class="rename" compatibility="8.1.001" expanded="true" height="82" name="Rename (2)" width="90" x="313" y="136">
            <parameter key="old_name" value="Period"/>
            <parameter key="new_name" value="DateTime"/>
            <list key="rename_additional_attributes"/>
          </operator>
          <operator activated="true" class="subprocess" compatibility="8.1.001" expanded="true" height="82" name="Set Predictions_Params" width="90" x="447" y="136">
            <process expanded="true">
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Window_Size" width="90" x="45" y="34">
                <parameter key="macro" value="WindowSize"/>
                <parameter key="value" value="40"/>
              </operator>
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Horizon" width="90" x="179" y="34">
                <parameter key="macro" value="horizon"/>
                <parameter key="value" value="1"/>
              </operator>
              <operator activated="true" class="set_macro" compatibility="8.1.001" expanded="true" height="82" name="Set Future_Day" width="90" x="313" y="34">
                <parameter key="macro" value="futureDay"/>
                <parameter key="value" value="96"/>
              </operator>
              <connect from_port="in 1" to_op="Set Window_Size" to_port="through 1"/>
              <connect from_op="Set Window_Size" from_port="through 1" to_op="Set Horizon" to_port="through 1"/>
              <connect from_op="Set Horizon" from_port="through 1" to_op="Set Future_Day" to_port="through 1"/>
              <connect from_op="Set Future_Day" from_port="through 1" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role" width="90" x="581" y="136">
            <parameter key="attribute_name" value="DateTime"/>
            <parameter key="target_role" value="id"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes" width="90" x="715" y="136">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attributes" value="kW"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="6.4.000" expanded="true" height="103" name="Filter Examples" width="90" x="916" y="85">
            <parameter key="condition_class" value="no_missing_attributes"/>
            <list key="filters_list"/>
          </operator>
          <operator activated="true" class="parse_numbers" compatibility="8.1.001" expanded="true" height="82" name="Parse Numbers" width="90" x="1050" y="136">
            <parameter key="grouped_digits" value="true"/>
          </operator>
          <operator activated="true" class="series:windowing" compatibility="5.2.000" expanded="true" height="82" name="Windowing for Training" width="90" x="1184" y="136">
            <parameter key="window_size" value="%{WindowSize}"/>
            <parameter key="create_label" value="true"/>
            <parameter key="label_attribute" value="kW"/>
            <parameter key="horizon" value="%{horizon}"/>
          </operator>
          <operator activated="true" class="series:sliding_window_validation" compatibility="7.4.000" expanded="true" height="124" name="Validation" width="90" x="1318" y="34">
            <parameter key="training_window_width" value="40"/>
            <parameter key="test_window_width" value="20"/>
            <process expanded="true">
              <operator activated="true" class="k_nn" compatibility="8.1.001" expanded="true" height="82" name="k-NN" width="90" x="179" y="34"/>
              <connect from_port="training" to_op="k-NN" to_port="training set"/>
              <connect from_op="k-NN" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="8.1.001" expanded="true" height="82" name="Apply Model (2)" width="90" x="112" y="34">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_regression" compatibility="8.1.001" expanded="true" height="82" name="Performance" width="90" x="246" y="34"/>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="series:windowing" compatibility="5.2.000" expanded="true" height="82" name="Windowing for Application" width="90" x="1318" y="187">
            <parameter key="window_size" value="%{WindowSize}"/>
            <parameter key="label_attribute" value="inputYt"/>
          </operator>
          <operator activated="true" class="extract_macro" compatibility="8.1.001" expanded="true" height="68" name="Extract Example Count" width="90" x="1519" y="187">
            <parameter key="macro" value="exampleCount"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="filter_example_range" compatibility="8.1.001" expanded="true" height="82" name="Filter Example Range" width="90" x="1653" y="187">
            <parameter key="first_example" value="%{exampleCount}"/>
            <parameter key="last_example" value="%{exampleCount}"/>
          </operator>
          <operator activated="true" class="remember" compatibility="8.1.001" expanded="true" height="68" name="Remember" width="90" x="1787" y="187">
            <parameter key="name" value="data"/>
          </operator>
          <operator activated="true" class="loop" compatibility="8.1.001" expanded="true" height="82" name="Loop" width="90" x="1452" y="34">
            <parameter key="iterations" value="%{futureDay}"/>
            <process expanded="true">
              <operator activated="true" class="recall" compatibility="8.1.001" expanded="true" height="68" name="Recall" width="90" x="45" y="136">
                <parameter key="name" value="data"/>
              </operator>
              <operator activated="true" class="apply_model" compatibility="7.1.001" expanded="true" height="82" name="Apply Model" width="90" x="179" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="multiply" compatibility="8.1.001" expanded="true" height="103" name="Multiply" width="90" x="447" y="30"/>
              <operator activated="true" class="materialize_data" compatibility="8.1.001" expanded="true" height="82" name="Materialize Data (2)" width="90" x="179" y="165"/>
              <operator activated="true" class="generate_attributes" compatibility="6.4.000" expanded="true" height="82" name="Increase Date (2)" width="90" x="313" y="165">
                <list key="function_descriptions">
                  <parameter key="DateTime" value="date_add(DateTime, 15, DATE_UNIT_MINUTE)"/>
                </list>
              </operator>
              <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role (2)" width="90" x="447" y="165">
                <parameter key="attribute_name" value="prediction(label)"/>
                <list key="set_additional_roles"/>
              </operator>
              <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes (3)" width="90" x="179" y="289">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="kW-39"/>
                <parameter key="invert_selection" value="true"/>
              </operator>
              <operator activated="true" class="rename" compatibility="8.1.001" expanded="true" height="82" name="Rename" width="90" x="313" y="289">
                <parameter key="old_name" value="kW-38"/>
                <parameter key="new_name" value="kW-39"/>
                <list key="rename_additional_attributes">
                  <parameter key="kW-37" value="kW-38"/>
                  <parameter key="kW-36" value="kW-37"/>
                  <parameter key="kW-35" value="kW-36"/>
                  <parameter key="kW-34" value="kW-35"/>
                  <parameter key="kW-33" value="kW-34"/>
                  <parameter key="kW-32" value="kW-33"/>
                  <parameter key="kW-31" value="kW-32"/>
                  <parameter key="kW-30" value="kW-31"/>
                  <parameter key="kW-29" value="kW-30"/>
                  <parameter key="kW-28" value="kW-29"/>
                  <parameter key="kW-27" value="kW-28"/>
                  <parameter key="kW-26" value="kW-27"/>
                  <parameter key="kW-25" value="kW-26"/>
                  <parameter key="kW-24" value="kW-25"/>
                  <parameter key="kW-23" value="kW-24"/>
                  <parameter key="kW-22" value="kW-23"/>
                  <parameter key="kW-21" value="kW-22"/>
                  <parameter key="kW-20" value="kW-21"/>
                  <parameter key="kW-19" value="kW-20"/>
                  <parameter key="kW-18" value="kW-19"/>
                  <parameter key="kW-17" value="kW-18"/>
                  <parameter key="kW-16" value="kW-17"/>
                  <parameter key="kW-15" value="kW-16"/>
                  <parameter key="kW-14" value="kW-15"/>
                  <parameter key="kW-13" value="kW-14"/>
                  <parameter key="kW-12" value="kW-13"/>
                  <parameter key="kW-11" value="kW-12"/>
                  <parameter key="kW-10" value="kW-11"/>
                  <parameter key="kW-9" value="kW-10"/>
                  <parameter key="kW-8" value="kW-9"/>
                  <parameter key="kW-7" value="kW-8"/>
                  <parameter key="kW-6" value="kW-7"/>
                  <parameter key="kW-5" value="kW-6"/>
                  <parameter key="kW-4" value="kW-5"/>
                  <parameter key="kW-3" value="kW-4"/>
                  <parameter key="kW-2" value="kW-3"/>
                  <parameter key="kW-1" value="kW-2"/>
                  <parameter key="kW-0" value="kW-1"/>
                  <parameter key="prediction(label)" value="kW-0"/>
                </list>
              </operator>
              <operator activated="true" class="remember" compatibility="8.1.001" expanded="true" height="68" name="Remember (2)" width="90" x="447" y="289">
                <parameter key="name" value="data"/>
              </operator>
              <connect from_port="input 1" to_op="Apply Model" to_port="model"/>
              <connect from_op="Recall" from_port="result" to_op="Apply Model" to_port="unlabelled data"/>
              <connect from_op="Apply Model" from_port="labelled data" to_op="Multiply" to_port="input"/>
              <connect from_op="Multiply" from_port="output 1" to_port="output 1"/>
              <connect from_op="Multiply" from_port="output 2" to_op="Materialize Data (2)" to_port="example set input"/>
              <connect from_op="Materialize Data (2)" from_port="example set output" to_op="Increase Date (2)" to_port="example set input"/>
              <connect from_op="Increase Date (2)" from_port="example set output" to_op="Set Role (2)" to_port="example set input"/>
              <connect from_op="Set Role (2)" from_port="example set output" to_op="Select Attributes (3)" to_port="example set input"/>
              <connect from_op="Select Attributes (3)" from_port="example set output" to_op="Rename" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_op="Remember (2)" to_port="store"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="8.1.001" expanded="true" height="82" name="Append" width="90" x="1586" y="34"/>
          <operator activated="true" class="generate_attributes" compatibility="6.4.000" expanded="true" height="82" name="Correct Date (3)" width="90" x="1720" y="34">
            <list key="function_descriptions">
              <parameter key="DateTime" value="date_add(DateTime, -8, DATE_UNIT_DAY)"/>
            </list>
          </operator>
          <connect from_op="Read CSV" from_port="output" to_op="Sort" to_port="example set input"/>
          <connect from_op="Sort" from_port="example set output" to_op="Rename (2)" to_port="example set input"/>
          <connect from_op="Rename (2)" from_port="example set output" to_op="Set Predictions_Params" to_port="in 1"/>
          <connect from_op="Set Predictions_Params" from_port="out 1" to_op="Set Role" to_port="example set input"/>
          <connect from_op="Set Role" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_op="Parse Numbers" to_port="example set input"/>
          <connect from_op="Parse Numbers" from_port="example set output" to_op="Windowing for Training" to_port="example set input"/>
          <connect from_op="Windowing for Training" from_port="example set output" to_op="Validation" to_port="training"/>
          <connect from_op="Windowing for Training" from_port="original" to_op="Windowing for Application" to_port="example set input"/>
          <connect from_op="Validation" from_port="model" to_op="Loop" to_port="input 1"/>
          <connect from_op="Validation" from_port="averagable 1" to_port="result 1"/>
          <connect from_op="Windowing for Application" from_port="example set output" to_op="Extract Example Count" to_port="example set"/>
          <connect from_op="Extract Example Count" from_port="example set" to_op="Filter Example Range" to_port="example set input"/>
          <connect from_op="Filter Example Range" from_port="example set output" to_op="Remember" to_port="store"/>
          <connect from_op="Loop" from_port="output 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_op="Correct Date (3)" to_port="example set input"/>
          <connect from_op="Correct Date (3)" from_port="example set output" to_port="result 2"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
        </process>
      </operator>
    </process>

    I hope it helps,

     

    Regards,

     

    Lionel

  • MontseMontse Member Posts: 19 Maven
    Hi @lionelderkrikor,

    I have the same problem with a time series but I have not found any repeated date.
    I attach you the file and next you can find the XML process, making only a classic decomposition of the data.
    The error is the same " The indices of the time series are not strictly monotonic increasing. Check Indices Attribute: date"
    I have checked out the "has indices parameter" and then works fine but I need the date associated with every row.

    Thanks in advance.
    Best regards,
    Montse


    <?xml version="1.0" encoding="UTF-8"?><process version="9.3.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.3.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="read_csv" compatibility="9.3.000" expanded="true" height="68" name="Read CSV" width="90" x="112" y="34">
            <parameter key="csv_file" value="C:\Users\monts\Documents\sales.csv"/>
            <parameter key="column_separators" value=";"/>
            <parameter key="trim_lines" value="false"/>
            <parameter key="use_quotes" value="true"/>
            <parameter key="quotes_character" value="&quot;"/>
            <parameter key="escape_character" value="\"/>
            <parameter key="skip_comments" value="true"/>
            <parameter key="comment_characters" value="#"/>
            <parameter key="starting_row" value="1"/>
            <parameter key="parse_numbers" value="true"/>
            <parameter key="decimal_character" value="."/>
            <parameter key="grouped_digits" value="false"/>
            <parameter key="grouping_character" value=","/>
            <parameter key="infinity_representation" value=""/>
            <parameter key="date_format" value="M/d/yy h:mm a"/>
            <parameter key="first_row_as_names" value="true"/>
            <list key="annotations"/>
            <parameter key="time_zone" value="SYSTEM"/>
            <parameter key="locale" value="English (United States)"/>
            <parameter key="encoding" value="windows-1252"/>
            <parameter key="read_all_values_as_polynominal" value="false"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="Month Number.true.integer.attribute"/>
              <parameter key="1" value="Year.true.integer.attribute"/>
              <parameter key="2" value="date.true.date_time.attribute"/>
              <parameter key="3" value="sales.true.integer.attribute"/>
            </list>
            <parameter key="read_not_matching_values_as_missings" value="false"/>
            <parameter key="datamanagement" value="double_array"/>
            <parameter key="data_management" value="auto"/>
          </operator>
          <operator activated="true" class="time_series:classic_decomposition" compatibility="9.3.000" expanded="true" height="82" name="Classic Decomposition" width="90" x="313" y="34">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="sales"/>
            <parameter key="attributes" value=""/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="numeric"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="real"/>
            <parameter key="block_type" value="value_series"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="value_series_end"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="has_indices" value="true"/>
            <parameter key="indices_attribute" value="date"/>
            <parameter key="decomposition_mode" value="additive"/>
            <parameter key="seasonality" value="12"/>
          </operator>
          <connect from_op="Read CSV" from_port="output" to_op="Classic Decomposition" to_port="example set"/>
          <connect from_op="Classic Decomposition" from_port="decomposition" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,533 RM Data Scientist
    Hi @Montse ,
    did you try to sort the dates before? i think some operators requiere this.

    Best,
    Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • MontseMontse Member Posts: 19 Maven
    Thank you @mschmitz and @tftemme . Now it works fine. Best regards, Montse
Sign In or Register to comment.