"Twitter API error for multiple user details extraction"

m_oke

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="179" y="85">
        <parameter key="connection" value="TwitterConnection1"/>
        <parameter key="query" value="&quot;rapidminer&quot; -rt -http"/>
        <parameter key="limit" value="20"/>
        <parameter key="language" value="en"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="7.6.001" expanded="true" height="103" name="Filter Examples" width="90" x="447" y="85">
        <list key="filters_list">
          <parameter key="filters_entry_key" value="Retweet-Count.ge.1"/>
        </list>
      </operator>
      <operator activated="true" breakpoints="after" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="648" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="From-User"/>
      </operator>
      <operator activated="true" breakpoints="after" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values (2)" width="90" x="849" y="85">
        <parameter key="attribute" value="From-User"/>
        <process expanded="true">
          <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="514" y="34">
            <parameter key="connection" value="NewConnection"/>
            <parameter key="user" value="%{loop_value}"/>
          </operator>
          <connect from_op="Get Twitter User Details (2)" from_port="output" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="append" compatibility="7.6.001" expanded="true" height="82" name="Append" width="90" x="1050" y="85"/>
      <connect from_op="Search Twitter" from_port="output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Loop Values (2)" to_port="input 1"/>
      <connect from_op="Loop Values (2)" from_port="output 1" to_op="Append" to_port="example set 1"/>
      <connect from_op="Append" from_port="merged set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

I built an experiment in RapidMiner that does the following:

1) Using the 'Search Twitter', it searches twitter for keywords/queries

2) Using 'Filter keywords', it filters the results by number of retweets greater than 1

3) Using 'Selelect Attributes', it selects only the attribute 'From-User'

4) Using 'Loop Values' with emebedded 'Get Twitter user details', I extracted the user details of all users in (3) above

5) Using the 'Append' tool, I appended all the user details into 1

 

On running the experiment, it gave an error of 'Twitter API error' at the 'Get Twitter User Details' tool.

 

Not sure how to fix it, any help will be apreciated.

 

XML code of the experiment is attached:

 

Edin_Klapic

Hi @m_oke,

 

the following line in your log file explains the problem:

 

Caused by: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true

The Operator "Get Twitter User Details" with the parameter name searches for the Screen name of a user.

That is the one with @. Those do not have blanks. If you can obtain those names you can use them.

Otherwise you can use the parameter id within "Get Twitter User Details". The id is a number and is also available from the Operator Search Twitter (Attribute: From-User-Id).

 

An example process can be found in this community post.

 

Best regards,

Edin

 

Edin_Klapic

HI @m_oke,

 

can you please provide the file rapidminer-studio.log which is located in the directory <userhome>/.RapidMiner/

 

Thanks,

Edin

m_oke

Hi @Edin_Klapic

See the log below:

 

Nov 13, 2017 10:48:44 AM com.rapidminer.tools.ResultService init
INFO: No filename given for result file, using stdout for logging results!
Nov 13, 2017 10:48:44 AM com.rapidminer.Process run
INFO: Process starts
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread run
SEVERE: Process failed: Error on connecting to Twitter API.
com.rapidminer.operator.UserError: Error on connecting to Twitter API.
    at com.rapidminer.operator.web.services.twitter.GetTwitterUserDetailsOperator.createExampleSet(GetTwitterUserDetailsOperator.java:218)
    at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:49)
    at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:33)
    at com.rapidminer.operator.io.AbstractReader.doWork(AbstractReader.java:126)
    at com.rapidminer.operator.Operator.execute(Operator.java:1004)
    at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
    at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator.doIteration(AbstractLoopOperator.java:304)
    at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator$1.call(AbstractLoopOperator.java:191)
    at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator$1.call(AbstractLoopOperator.java:182)
    at com.rapidminer.extension.concurrency.execution.BackgroundExecutionService$ExecutionCallable.call(BackgroundExecutionService.java:357)
    at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=381aea58 or
    http://www.google.co.jp/search?q=3393bbb1
TwitterException{exceptionCode=[381aea58-3393bbb1 720b1cc8-3ee3b645 720b1cc8-3ee3b636], statusCode=-1, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=4.0.1}
    at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:179)
    at twitter4j.HttpClientBase.request(HttpClientBase.java:53)
    at twitter4j.HttpClientBase.get(HttpClientBase.java:71)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1950)
    at twitter4j.TwitterImpl.showUser(TwitterImpl.java:874)
    at com.rapidminer.operator.web.services.twitter.GetTwitterUserDetailsOperator.createExampleSet(GetTwitterUserDetailsOperator.java:168)
    ... 18 more
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
    at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
    at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
    at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
    at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at twitter4j.HttpResponseImpl.<init>(HttpResponseImpl.java:50)
    at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:143)
    ... 23 more
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
    at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
    at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
    at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at twitter4j.HttpResponseImpl.<init>(HttpResponseImpl.java:35)
    ... 24 more

Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE: Here:
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:           Process[1] (Process)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:            subprocess 'Main Process'
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:              +- Search Twitter[1] (Search Twitter)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:              +- Filter Examples[1] (Filter Examples)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:              +- Select Attributes[1] (Select Attributes)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:        ==>   +- Loop Values (2)[1] (Loop Values)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:            subprocess 'LoopValues'
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:              |     +- Get Twitter User Details (2)[4] (Get Twitter User Details)
Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
SEVERE:              +- Append[0] (Append)
Nov 13, 2017 10:48:46 AM com.rapidminer.operator.Operator stop
INFO: Loop Values (2): Process stopped.
Nov 13, 2017 10:48:46 AM com.rapidminer.operator.Operator stop
INFO: Loop Values (2): Process stopped.