Downloading historical financial data since change of Yahoo API

qwertz2

Dear community,

Since Yahoo has changed its API, it is not that easy to access historical financial data anymore by pre-installed operators. Thanks to Thomas who posted a workaround based on the google finance service it is possible to retrieve stock data again.

But unfortunately google finance does not provide CSV download for other object types than stocks. Indices like the Dow Jones, commodities, currencies and bonds are not available.


Is there anyone who knows about free sources of historical data for
- indices
- commodities
- currencies
- bonds
which can be loaded into Rapidminer?



Kind regards
Sachs

Thomas_Ott

While not an elegant solution (i.e. plug right into RapidMiner), have you looked at YQL? https://developer.yahoo.com/yql/console/

 

They allow you get an SQL dump of data from different parts of their company.

Telcontar120

Quandl has a lot of financial data---some is free, some is paid, so you'll need to search specifically for what you are looking for, but they have many download formats available, and it can be used reliably with RapidMiner (I have done it).

https://www.quandl.com/search?query=

 

sgenzer

and I use the API from openexchangerates.com.  Take your pick!  

Scott

Thomas_Ott

@sgenzer that's only for Currencies, right?

sgenzer

yes correct - only currencies.  Cost is a freemium model where you get 500(?) calls/month free and pay for more.

qwertz2

 

Hi all,

 

I found that Yahoo still works when a parameter CRUMB is added. This parameter is individual to your session and stored in cookies. So you could still use Yahoo with but with the drawback that you have to start your browser first and call Yahoo page.

 

Not yet tested with Rapidminer...

 

 

Best

Sachs

qwertz2

 

Ok, this is obviously not as easy as expected...

 

1) I opened the Yahoo page

https://finance.yahoo.com/quote/SAN/history?period1=1483250400&period2=1495256400&interval=1d&filter=history&frequency=1d

 

2) There is a little download data button at the right. This is where I copied that link (which includes the session crumb).

https://query1.finance.yahoo.com/v7/finance/download/SAN?period1=1483250400&period2=1495256400&interval=1d&events=history&crumb=TAInnNixgdJ

 

3) This link can be opened from the browser several times. But once opened with Rapidminer it got en error

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="open_file" compatibility="7.5.001" expanded="true" height="68" name="Get Online Data as CSV" width="90" x="45" y="34">
        <parameter key="resource_type" value="URL"/>
        <parameter key="filename" value="http://www.google.com/finance/historical?q=NASDAQ:%{symbol}&amp;amp;ei=LRD-WJmhHIOrmgH_4KToAg&amp;amp;output=csv"/>
        <parameter key="url" value="https://query1.finance.yahoo.com/v7/finance/download/SAN?period1=1483250400&amp;period2=1495256400&amp;interval=1d&amp;events=history&amp;crumb=TAInnNixgdJ"/>
      </operator>
      <operator activated="true" class="read_csv" compatibility="7.5.001" expanded="true" height="68" name="Read CSV" width="90" x="179" y="34">
        <parameter key="csv_file" value="C:\Users\Abc\Desktop\bas.csv"/>
        <parameter key="column_separators" value=","/>
        <parameter key="use_quotes" value="false"/>
        <parameter key="date_format" value="dd-MMM-yy"/>
        <parameter key="first_row_as_names" value="false"/>
        <list key="annotations">
          <parameter key="0" value="Name"/>
        </list>
        <parameter key="encoding" value="UTF-8"/>
        <list key="data_set_meta_data_information">
          <parameter key="0" value="Date.true.date.id"/>
          <parameter key="1" value="Open.false.real.attribute"/>
          <parameter key="2" value="High.false.real.attribute"/>
          <parameter key="3" value="Low.false.real.attribute"/>
          <parameter key="4" value="%{symbol_name}.true.real.attribute"/>
          <parameter key="5" value="Volume.false.real.attribute"/>
        </list>
      </operator>
      <connect from_op="Get Online Data as CSV" from_port="file" to_op="Read CSV" to_port="file"/>
      <connect from_op="Read CSV" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

This is going to be a looooong night...

 

 

Cheers

Sachs

qwertz2

 

So far I could not get the historical data download running in Rapidminer. But as an intermediate result I would like to share three interesting links where other people managed to get it running in different programming languages like Java, Python or VBA:

 

https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working

https://stackoverflow.com/questions/44044263/yahoo-finance-historical-data-downloader-url-is-not-working

https://www.mrexcel.com/forum/excel-questions/1005785-yahoo-finance-api-stock-quotes-changed-3.html

 

One step closer to the goal for sure but no idea which distance is left on the way to go...

 

Cheers

Sachs

Thomas_Ott

Well if they can get it to run in Java, you could build a new extension for it. 

qwertz2

 

Hmm, I tried to get it working in an execute script operator as something to start with. However, sadly I was not able to get it running and I guess I am not the coder with the appropriate knowledge to find the root cause. This is where I have to give in. But hopefully this inspires other members with Java background to continue where I had to stop. Sometimes it's just a colon in the code that messes up everything...

 

Cheers

Sachs

Telcontar120

Did you ever check to see whether quandl had the data series you were interested in?  As I mentioned I am able to use their data with no additional coding required.

 

qwertz2

 

Yes, I tried to but unfortunately the indices data I am looking for is only available in the premium package.

Thanks for your help anyway

 

Kind regards

Sachs

martinvenich

The spreadsheet here works with Yahoo Finance to download bulk historical data. The VBA can be viewed and modified. The code is more complicated than the now defunct Yahoo API calls, but it still works

sgenzer

just to add to this thread...I wrote a KB article two weeks ago about using Alpha Venture API as a good alternative to Yahoo Finance.  You can see the article here: http://community.rapidminer.com/t5/RapidMiner-Studio-Knowledge-Base/Real-Time-Financial-Data-via-Alpha-Venture-API-alternative-to/ta-p/41119

 

Scott