How to get many user Twitter datas in a short time?
YYTheFenix
Member Posts: 4 
Contributor I
Hello , I'm Beginner in Rapid miner studio
I can only select tool box and match them each other to find the results.
Now, i want to extract twitter user datas, but i can extract it only one by one
such as i can get user twitter details one by one. but i want to get user twitter details of 250 users in one time
but input of tool box(parameter) can only insert 1 id or 1 name.
1.) How can i insert id or name more than one in one time? (want to find 250 users detail in one table)
2.) from 1.) Can i insert some condition more such as the users in 1.) must have friends <200 users
Thank you
Best Answers
-
Edin_Klapic
Employee-RapidMiner, RMResearcher, Member Posts: 299 
RM Data Scientist
Hi @YYTheFenix,
There is no built in functionality to get the data for all users in one request.
This has to be achieved by searching for every User detail separately e.g. using the Operator Loop Values.
Here you are looping over every username and search for the user details using Macros (more on Macros can be found here).
The attached example Process also checks for the number of friends of a Twitter user using the Operator Get Twitter Relations. Here you can set the parameter relation to friends. The following Branch Operator uses the condition type min_examples to only search for the User details if the user has at least 199 friends.
Important notes:
- Be aware that there is a limit on the number of free Twitter calls
- Usually I recommend to utilize the User ID for searching but currently there is a bug when using this parameter so the Twitter user name is used. This is the one with @ and may differ from the display name which you get from the Operator Search Twitter.
Best regards,
Edin
<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
<parameter key="query" value="rapidminer"/>
<parameter key="limit" value="1"/>
<description align="center" color="transparent" colored="false" width="126">get users</description>
</operator>
<operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="246" y="34">
<parameter key="attribute" value="From-User"/>
<parameter key="reuse_results" value="true"/>
<process expanded="true">
<operator activated="true" class="social_media:get_twitter_relations" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter Relations" width="90" x="112" y="34">
<parameter key="user" value="%{loop_value}"/>
<parameter key="relation" value="friends"/>
</operator>
<operator activated="true" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch" width="90" x="246" y="34">
<parameter key="condition_type" value="min_examples"/>
<parameter key="condition_value" value="199"/>
<process expanded="true">
<operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="246" y="34">
<parameter key="user" value="%{loop_value}"/>
</operator>
<connect from_op="Get Twitter User Details" from_port="output" to_port="input 1"/>
<portSpacing port="source_condition" spacing="0"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_input 1" spacing="0"/>
<portSpacing port="sink_input 2" spacing="0"/>
</process>
<process expanded="true">
<portSpacing port="source_condition" spacing="0"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_input 1" spacing="0"/>
<portSpacing port="sink_input 2" spacing="0"/>
</process>
</operator>
<connect from_op="Get Twitter Relations" from_port="output" to_op="Branch" to_port="condition"/>
<connect from_op="Branch" from_port="input 1" to_port="output 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="source_input 2" spacing="0"/>
<portSpacing port="sink_output 1" spacing="0"/>
<portSpacing port="sink_output 2" spacing="0"/>
</process>
</operator>
<connect from_op="Search Twitter" from_port="output" to_op="Loop Values" to_port="input 1"/>
<connect from_op="Loop Values" from_port="output 1" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>2 -
Edin_Klapic
Employee-RapidMiner, RMResearcher, Member Posts: 299 RM Data Scientist
Hi @YYTheFenix,
There are 2 problems when you want to search for Twitter User Details and using Search Twitter as base.
- Twitter changed the possible length of the From-User-Id which is not handled correct internally and therefore sometimes results in false ids which then probably lead to the error you mentioned.
- You cannot safely use the From-User you get from Search Twitter because this is only the Display name which does not uniquely identify a user. As I mentioned in my previous post you need to use the Twitter User name which is the one with the @ symbol.
We are investigating how and when we are able to fix the bug with the User-Ids but I cannot make any promises.
Best regards,
Edin
2
Answers
@YYTheFenix There's a sample process on the Community on how to do that with a Loop that I think @Edin_Klapic posted. It uses the Get Twitter User Details operator. <strike>Search for that.</strike>
Here it is: https://community.rapidminer.com/t5/Product-Feedback/HELP-obtaining-Twitter-user-details-I-m-new-to-RapidMiner/idc-p/38750#M90
@Edin_Klapic is the bug fixed?
@Edin_Klapic
thank you for nice xml code. when i try it and i found that this code can extract user's details from only one user per time (because i try to change limit more than 1 in search twitter box but it doesn't work , So i change it back to 1). And after that, i try to change condition in Branch to max_examples 200 . Could you tell me why don't have output result?
PS. my target is to find someone who has keyword rapidminer and has friend less than 200 users.
Thanks that nice topic @Thomas_Ott
I found one xml code from @Edin_Klapic which can extract user's detail more than 1 user in one time.(consider from user-id)
but i have some problem with that xml code when i change query in search twitter box to other word such as "police" it got error in GET USER TWITTER DETAIL box (twitter api error) Could you tell me why?
and if i want to add some conditions such as user must have friends<200 and followers<200 How should i do? i try to add Branch box connectd with get twitter user details box in Loop box but it doesn't work <<Below is original xml>>
Hi @YYTheFenix,
The attached process contains a solution with which you can use both search conditions in a nested Operator.
An easier way would be to get all User details and use Filter Examples afterwards since the number of Friends and Followers is retrieved from Get User Details.
Best regards,
Edin
Thank you @Edin_Klapic for nice code.
I try to combine all xml which i've got from you.
and from your previous answer . So, i choose user-id to use. (Although, it has problem in get twitter user detail box.)
finally i got below xml .(both condition friends and followers)
I found that it still has problem in get twitter user detail box when the input is a large number.
if below xml can be rewrite more easier you can tell me anytime Thanks.
Ps. i'm trying your new xml for better result
Hi @YYTheFenix,
the following process uses Filter Example to avoid a duplicate execution of Get User Details.
When you Search for Tweets some of the From-Ids have 16 or more digits. The current implementation produces a rounding error which results in a wrong id macro.
Thus for testing purposes I recommend to remove all Examples from your list where the id is longer than 15 digits.
Best regards,
Edin
Hi @YYTheFenix, @Thomas_Ott,
Just as a follow up on the Twitter user id issues.
The latest RapidMiner version 8.1 has an updated Twitter extension with fixed bugs and also new features.
More information can be found here: https://docs.rapidminer.com/latest/studio/releases/changes-8.1.0.html
Best,
Edin