Why is RapidMiner Problematic?

SkyTrader

1) Over a month ago before the last two updates I used to be able to get RM to produce an attribute drop down and I could select "Close." I have tried on and off for over a month since and have not been able to get RM to recognise ANY attributes whatever method I use or sort out a host of other problems that I used to be able to do with RM.

2) I had to type "Close" in manually in the attribute drop down because the drop down is always blank nowadays whatever Excel file I use (and by that Excel files that I know have worked in the past with RM) and whether I use Import Wizard -- which has been not working as expected because I now get issues with a column called adj.close:





3) What's even more obvious is that I don't even have that adj close column anymore because I already deleted it from my Excel file (I've done everything to get round all these constant RM errors/warning messages). RM clearly has a Excel file caching problem: 
https://community.rapidminer.com/discussion/comment/67238#Comment_67238 

4) Why is the close non existent in RM drop down list when it's in my Excel file? 

5) Why does "Excluding" the adj close (which is really not there and is a ghost) make no difference as the adj close column still causes a warning?



6) I also have to ask why is RM referring to a FTSE file that was created today 9th Sept as "last modified" on 7th Sept? How can I clear this directory as it looks like a caching issue?

7) My Macbook FTSE folder is blank so what is this file with a ~?

MartinLiebig

Hi,

Please run the complete Wizard of the read excel operator or fill out the metadata information manually, otherwise RM cannot possible know what is in your excel file. Please do not use the import data functionality if you have changing files, the import wizard creates a copy of your data. If you change your excel file, you need to rerun the wizard or reset the meta data information.

For the ~$ files, those are lock files from excel. See for example: https://discussions.apple.com/thread/7628094 .

Best,

Martin

SkyTrader

Thanks Martin, @mschmitz

How do I reset the meta data information because I need to regularly update Excel with new date fro Read Excel?

I've done this (many, many times) but I still cannot get stop the "adj. close" column appearing within RM?



How do I clear "ghost" meta in RM?

Lastly, Excel Line 102 doesn't exist but it does in RM?



Cheers,

MartinLiebig

Hi,

You show the FTSE sheet in your excel screenshot, but try to read the DOW sheet in RapidMiner. This way this needs to differ?

Best,

Martin

SkyTrader

"This way this needs to differ?"

Sorry I don't understand your point? What does the Dow have to do with the current problems I have with the FTSE file? Besides the Dow Excel file doesn't produce anything but blank white space in the attributes drop down menu, although it used to work when I first used RM before the two updates. That's why I'm starting all over again and trying to use a new simple FTSE file (with no technical indicators and just 101 rows).

How can the warning I now get (pls see below) be saying the FTSE file doesn't exist? It's loaded okay in my results:
Note: I had to exclude the adj. close column using "Exclude" in Rm because RM still thinks it exists. How do I stop RM from including the "ghost" column adj close when NO such column exists in Excel?)



I still have blank attributes in the drop down (it never used to be a problem and always populated before the last two updates).



When I manually add the Close:



I get this warning saying the FTSE file does not exist in repository even though I can see it?:



Can you please explain how to permanently fix these time consuming problems using RM? Again, respectfully, please note I had none of these problems with RM when I first started using it, files loaded and attribute drops down self populated... and I was always constantly changing my Excel file without any issues too.

Lastly, in my previous post I asked why is the FTSE file referring to an error with row 102. There is no row 102 in my Excel file?

Thanks very much,

MartinLiebig

Hi,

Did you fix the wrong sheet name?

Best,

Martin

MartinLiebig

To your second question, as you can see in your repository there is no file named 'FTSE Daily', thats why you cannot load it with retrieve.

SkyTrader

"Did you fix the wrong sheet name?" Sorry, but I don't see any reference to a Dow Jones Excel file in my screenshots, only my new basic FTSE file?

I am using a new FTSE file to try and get RM to work. It says FTSE in all my screenshots.

SkyTrader

"there is no file named 'FTSE Daily'"

Right, I amended it to the simple "FTSE Daily" (so no date range in title) but despite "Excluding" the "Adj Close" column during the import, RM still thinks adj close exists (it doesn't in my Excel file) and will not import as it gives a warning:



This warning is because there is some kind of cache within RM? "the import wizard creates a copy of your data. If you change your excel file, you need to rerun the wizard or reset the meta data information."

How do I stop this RM thinking it's looking at an old non existent Excel file with adj close within the file? This was how the file was originally a few days ago but I got warning about the adj. close so removed it from the Excel file. I also never got any of these issues with my Dow Jones file that had an adj close column??

Cheers, once again,

MartinLiebig

Hi @SkyTrader ,

i am not talking about different files, but two different sheets in your excel file which is clearly visible in your screenshot. Files and Sheets are different things.

Best,

Martin

SkyTrader

Thanks Martin, @mschmitz
I've deleted the extra Dow sheet from Excel leaving just the FSTE data deleted the double sheet version in my directory for a single sheet version on my Macbook... not that it's made any difference as I still can not import the single sheet FTSE file and get it to recognise the close column which has been set as my label.

Why should I have to still "exclude" the adj close using the import wizard when the adj close simply does not exist in my Excel file? If I don't "exclude" the adj close it still appears in RM. What is specifically causing the non existent adj close to keep appearing in RM? 

I got the FTSE one sheet file to load but the Set Role operator now has an error saying the attribute close is not being found when using the import wizard even though I set close as my label with the import wizard using Read Excel (I never had any of these problems when I first started using RM with the single complex 5000 row Dow sheet Excel file).



I have been trying to resolve this issues for literally weeks now -- I quit RM and rebooted.

Now I get this warning (I used a breakpoint) :



I then decided to use an old Read Excel / Store process that I already have that works and set it to the FTSE file (instead of the Dow file that works fine). The FTSE data reads okay without bringing in the ghost adj close column. Why is that? What's wrong with the Read Excel operator I have in all the screenshots above?

As for Set Roll it doesn't populate with Open/High/Low/Close/Volume columns?
It doesn't work although I've manually written in "Close" either?




I not sure why something that was so simple to do with the Dow file is now causing such continued issues, particularly as this FTSE file is a test file (to get RM to work) and the most basic Yahoo Finance download with just 100 rows of Open/High/Low/Close/Volume data with none of the technical indicators (that initially caused some import issues with my complex Dow file).  

Thanks again for any help, I'd really like to actually get on with getting forecasts.

MartinLiebig

Hi @SkyTrader ,

sorry, but I cannot help you in this way. Can you maybe post the process and the data?

Best,

Martin

SkyTrader

Thanks. 
I normally pride myself on my attention to detail but you're not going to believe this Martin, @mschmitz, the Set Role was not able to find the Close attribute with the FTSE data because of an asterisk... As in: Excel Column E = Close*. Instead of the correct: Excel Column E = Close. Please see the 2nd image I posted at the top of this thread. 

I removed the *. Now it loads fine just like my Dow data did.

What's odd is when I went back to re-download the FTSE data just now (same link to Yahoo Finance that I used before from google, as it's purple highlighted in Chrome), no such asterisk appears with the Close in my new Excel download... What's even stranger when I ran the Forecast I was getting predicted close values from the 18th Sept. onwards. that's because I've just noticed that the 4 months worth of data in my FTSE file I downloaded last week has prices going to the 18th September 2020!? Prices that don't even exist yet... Yahoo isn't even currently showing a chart for the FTSE either...

Here's the process if anyone else would like to use Martin's ARIMA Forecast Univariate:

Cheers, thanks for your continued help and patience.

MartinLiebig

So.. Does this mean our Read operator did not handle '*' signs correctly?

SkyTrader

Yes, but where and why there was an asterisk in Excel in the first place when I've only ever seen if assigned to the adjusted close column I have no idea!