Macro/url query parameter malfunctioning in RM web service.
Hi there,
I have setup a rapidminer webservice with a url query parameter bound to a macro with a default value in the underlying process context. But when testing the web service (in admin console or calling the API url), I enter a different value for the query parameter (or in other words the macro in the underlying process), I still get results for the default value of the macro and not the new value that I just tried to pass. It seems the user entered values are not getting passed to the underlying process. So when testing, even if I do not pass anything, I still get results with the default value for the macro. Any ideas why this might be happening? I remember there was an old video (RM 5 if I recall correctly) about this on youtube but I just can't find it.
Thank you very much!
Best Answer
-
pari1234
Member Posts: 26 
Maven
Hi @sgenzer, sorry about not being able to reply sooner and thank you very much for taking the time to go through this process. You are right about the server logs error messages and access tokens. Anyways, I made some changes to the process such that instead of passing url parameters, I now just send a JSON array in a POST request to the RM webservice API.
[{
"platformName": "twitter",
"username": "rel8edto"
}, {
"platformName": "facebook",
"username": "rel8edto"
}]This then gets converted to an example set using the read document, JSON to Data and De-Pivot operators and using appropriate filters and extracting macro inside the process, I can pass the username to the appropriate branch (facebook or twitter). I still haven't figured out exactly what happened with being able to passing macros. I created an identical server client setup at home and it worked just fine. So I'm guessing something might have led to a bug in our RM Server at some point. But thank you very much for your help. :smileyhappy:
0
Answers
This feature still works in v7+. I would double check the process to make sure the macro is defined in there as well.
hello @pari1234 - welcome to the community. Yes passing macros back and forth can be tricky. Have you set up the context parameters in the process to receive the macro from server?
Scott
Thank you @sgenzer and @Thomas_Ott for your responses. I have defined the macro in process context as well. My process contains auth tokens for Aylien, Twitter and Facebook, would you like me to post the xml here? I've attached a screenshot of my process with the context panel and a screenshot of the service settings page on server.
Ironically, the first version with only the twitter component worked perfectly. Then after I added the facebook component and overwrote the previous process, this issue started occurring. I then tried to isolate the issue by creating a fresh process with just the twitter components and the macro and the issue was there as well. Next, I kept just a single twitter operator with a macro and the issue persisted even then. Not sure if it's a server setting I'm missing or what....
Does this help? Let me know if you'd need anything else. Thank you.
Hi @pari1234,
Could it be a problem of the execution order of the Operators within your process?
I guess you use the Operator "Generate Data by User Specification" for some kind of if - else statement.
If the Twitter / Facebook branch is executed before this Operator the default macro value is taken.
To activate the execution order view click on the blue rectangle symbol in the upper right corner of your process canvas.
Then you can change the order and click on the symbol again to exit this view.
By the way, for debugging I propose to use the Operator "Print to Console" as first Operator in your process and enter the macro there.
Whenever the process is executed, the Macro value is printed in the Server log file.
Best,
Edin
Thank you @Edin_Klapic but this solution did not work. Besides I'm sorry but I found changing the execution order to be very cumbersome. It's difficult to simply drag and drop to change the execition order. I was able to set the top Generate-Data-By-User-Specificaiton operator as the first one in the execution order by using 'bring-to-front', but that was pretty much it. Also if you read my previous response to @sgenzerand @Thomas_Ott, I have said that this problem persists even when I use just a single twitter operator and a macro in the porcess context and nothing else in my entire process while testing as a web service.
For print to console, can you tell mewhere will I find the server log? I mean do I have to manually go into my server file system to find a log file? Found it. Not sure which log file should I be looking at under Administration/SystemInformation/ServerLogs in the admin console. Will it also log the events if I make a call to the webservice from an external tool such as 'curl' or 'Postman'?
Thank you.
hi @pari1234 - so to change the execution order, you just click on the numbered circles in the order you want.
Here's a quick video: https://rapidminer-my.sharepoint.com/personal/sgenzer_rapidminer_com/_layouts/15/guestaccess.aspx?docid=0a3710a88c8c249d686b86d8fac1780b3&authkey=Afqijaoku5aG2AMyyRBlvaE&expiration=2018-01-02T14%3A43%3A29.000Z&e=c2c861d081f542b5913b7da6653401ca
You'll notice that sometimes you're prevented from changing the order because of the way the operators are wired. This is purposeful of course. All I'm doing in the video is clicking the numbers.
Scott
Hi @pari1234,
The actual log file is always server.log.
Unfortunately, there is no auto refresh. So if you executed a process you need to reload the file manually.
Best,
Edin
I understand @sgenzer and thank you. Looking at your video, it seems I've done exactly what you show. And as you said, some of the orders cant be changed because of the way the operators are wired. But this still isn't solving the issue. Sorry and thanks.
Thank you @Edin_Klapic, it may sound weird but I do not even see a server log file for today's date 2017-11-03. The file name would be something like server.log.2017-11-03 right?
The macro name for print-to-console operator is 'log'.
The actual server.log has no date. Just "server.log"
The macro to enter in Print to Console should be the Macro name you defined in the context of your process (see screenshot)
Best,
Edin
@Edin_Klapic@sgenzer@Thomas_Ott Here's the process minus the fb auth token. Not sure if process context is included in the xml, if not, then the macro in process context is called - 'twitter', feel free to use whatever default value you want. I'm sure you guys have your own tokens for Twitter and Aylien. I couldn't demo it to our client today on the call so I guess I have more time now. If it runs fine for you guys then I think there might be something going on in our RM Server. Here's how you can get an access token from fb which you will then enter in the part of the url parameter for Enrich Data by Webservice operator.
It will return an access token that looks like (not a real token)-
-
{
"access_token": "1745625463738593|w_a8sakjhflaYsCHPTOTDr5H1r-wY",
"token_type": "bearer"
}
Thank you.
And here's the server log file. I could not figure out what's the issue. In a particular section it says something about twitter API rate limit, but that should not be related to the process macro simply called "twitter" right?
@sgenzer, @Edin_Klapic, @Thomas_Ott Okay so this seems to be a common issue regardless of what kind of process I have. Is this a bug? I created a simple webservice with Generate Data by User Specification (attribute: "data", attribute value: %{value}) and a Print to Console operator. A macro named 'value' was created in process context with some default value.
So all the process does is return the value of the attribute "data". When I test the service with some other macro value, it still returns the default value. Hope that helps.
Ah, there's something interesting in your server log. It fails right at execution at the Generate Exampleset operator. It says it's a dummy, which means that it's probably not installed on your RM Server installation. Check the operators and extensions tab to see if that operator is there.
OR it means that the data you are pulling from that exampleset is not on the Server but your local machine.
Thank you @Thomas_Ott. May I ask if you mean the "Create Exampleset" operator when you say Generate Exampleset? If yes, then I resolved it last night. That is why I have "Generate Data by User Specification" instead now and I double-checked to see whether that operator exists in server or not (it exists). If not, then I'm not sure what more can I try.
The process inherently does not contain any data. It is meant to be a part of a larger data delivery framework. It will receive incoming data via the url query parameter from another component within the framework and spit out social analysis results. And that is what I'm trying to test by passing it a twitter/facebook username as a url parameter bound to the macro.
Sincerely,
Pari
Yeah I meant Create Exampleset.
From inspecting your process, it appears that you are passing a twitter name as %{twitter} macro. Makes complete sense but you use the same twitter handle as a FB name. Will that even work? My Twitter handle is '@neuralmarket' but you can use it as my username for FB.
If you disabled the FB process and run only the Twitter one, does it work locally and then on the Server? Is the Twitter connection on the Server authenticated as well as your local machine?
And, where is your %{example} macro defined? I see you use it for your Filter Range but where are the default values stored/defined.
@Thomas_Ott, for now I'm only using it to test usernames that have the same name on both twitter and facebook. The production version will incorporate a separate macro for facebook. But I want to make sure that I can pass parameters successfully.
Even if I remove the facebook component, the problem still remains. Executing the process locally does not make sense to me since that means I'm just changing the default value of the macro everytime I run it. My Twitter connection is properly authenticated both on server and locally.
I'll mention this again- to see if the problem is specific to the process, I created a completely new process with just one Generate Data by User Specification operator and the process just returned the default value of context macro. When testing it as a webservice with a new value for the macro in url parameter, I still got the default value returned to me. So either I'm doing something dramatically wrong on the server, or something in RM server has gone corrupt since until yesterday afternoon, the twitter component was running perfectly on its own. Problems started popping up when I added the facebook component, but I don't understand what could be wrong there.
As for the macro %{example} in filter_example_range, I'm simply using that block from the solution provided by @sgenzer to my previous thread here in the community. The macro seems to be defined in the parameter settings for the Loop Examples operator which is being used after the Enrich Data by Webservice operator in two different places in the Facebook subprocess. That's where default values for %{example} are coming from.
Hope this helps. Thank you.
@Thomas_Ott I have a workaround for this that seems to be working when tested partially. It involves passing operator parameters in the url instead of macro values. So I can pass operator parameter to each of the twitter operators. But in the Generate Data by User Specification operator parameter settings (the facebook part), it takes in a list (arrtibutename, attribute value) and not simply a field value. How do I pass a list to that operator via the url on server?
Thank you.
Hi @pari1234 - sorry I'm not getting back to this until now. So I retraced your steps locally on Studio (not Server) and your process works fine. I'm going to try to retrace your steps on a Server and see how that goes.
Scott
hello again @pari1234 - I played around for a while on a server and could not get the Facebook API to work on Server at all. Could you? I did not even get to the part of passing macros back and forth, etc... There are some pieces to this thread that may shed some light on this...
- As others have mentioned above, whenever you see "Dummy" operators in RapidMiner Server, you have one of two problems: 1) as @Thomas_Ott said, you have not installed the extension needed to run that operator, or 2) the version of the extension installed on Server is not compatible with the version of Server. When I look at the top of your log, I see this:
So it appears that you're using the Operator Toolbox extension in this process when it generated the error - except that I don't see it in your XML! Hmmph. You are using the Web Mining, Text Processing, and Aylien extensions for sure - so you need to ensure that these extensions are installed on your server AND are compatible with the version of Server you're using.
- There are some other errors in that log that I will defer to the Server experts here to explain better (@Edin_Klapic or others?). Working from the bottom, I see the following:
So this is for sure a API query limit.
This is only a warning but it concerns me. If I remember correctly, cipher.key stores various security features for RapidMiner but my memory escapes me exactly why this would happen. @Edin_Klapic?
So here I'm assuming (?) that you deleted your token in the log manually and you actually did pass the token. If so, then I was getting the exact same error and I think it's a funky Facebook thing (see below). If not, clearly the token is not getting inserted in to the URL and this cannot be a macro issue.
- I looked at Facebook API tech docs for a while (I've been interested in using this for quite some time, so thank you for showing how to get the tokens, etc..- big help!). It seems that Facebook has quite the apparatus going on with their APIs, including a funky "appsecret_proof" security feature that I have never seen before in all my API work. They also say at the top of this link that, "Almost every Graph API call requires an access token. Malicious developers can steal access tokens and use them to send spam from your app. Facebook has automated systems to detect this, but you can help us secure your app by adding extra parameters to API requests. This document explains how." [emphasis is mine]. I always wondered about the security of sending access tokens directly in the url like it's normally done - it seems that Facebook agrees that more security is needed. I have NOT played with this at all but I have a sneaky suspicion that their "automated systems" are messing with things.
Anyway those are my 10 cents on this very interesting process. Maybe some of this helps?
Scott
oh that's great, @pari1234. Well done. Could you post your process so I can see? I'd be very interested in your solution.
Scott
Thank you @sgenzer, and below is the process. You'll need to use your own fb access token and twitter and aylien connections. I'm thinking of making a video about this process (since RM already has operators for twitter). Other than my main job, I also work as a Teaching Assistant for graduate business analytics students at the local university and I think this will be VERY helpful to those students who are scurrying for a social media data collection tool in their text mining semester project. Most of them do not come from a technical background and this would definitely expand the scope of their learning outside the classroom.
oh that's great. Thank you! Tell you what - if you can write this up for a KB article, we can publish it here and I may be able to get you a cool RapidMiner t-shirt as part of the deal.
Scott