What is the correct syntax to use Execute program on Linux?
UnicornHi there,
I currently have a CRON trigger that executes a script on a daily base on my server, but I want to be able to do the same from within Rapidminer on demand.
As far as I understand the Execute Command operator should allow me to do that, but as nothing seems to happen (also no errors thrown) I am not sure I use it correctly.
My command is a curl command that loads a json file to my elastic search environment. Running this as a standard bash or sh file works fine, all my data is loaded to elastic, but running the same command in the operator just does nothing. I get a green tick indicating the process ran fine, but no data is transfered.
My curl command is as follows :
curl -s -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@mydata.json"
And this is my process :
<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="productivity:execute_program" compatibility="7.6.001" expanded="true" height="103" name="Execute Program" width="90" x="112" y="34">
<parameter key="command" value="curl -s -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@mydata.json""/>
<parameter key="working_directory" value="/data/elastic/imports"/>
<list key="env_variables"/>
</operator>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
</process>
</operator>
</process>
Any idea if I am using it correctly?
or other tips?
Answers
Hi @kayman,
since `curl` uses the current path as the destination (if not specified differently) it might already work. But don't expect the downloaded file at the output port of the Execute Script Operator. Try Execute Script with `pwd` to output the current path of the working directory the process is executed in and check manually if the downloaded file exists there.
If this doesn't work, please try executing your command without the `-s` option and post the log here.
Regards,
Philipp
Hi @pschlunder, thanks already for looking into this.
Unfortunatly ommiting the -s variable does not make much of a difference. I am aware there will be no output, the basic idea is just to fire the job in the background so I do not have to wait for my next cron cycle to take care of the load. I'm just abusing rapidminer as a trigger :-)
Attached the log, bit it does not seem to tell a lot. It just confirms there was no problem noticed, and it ended as soon as it started, basically doing nothing with the actual command. Running the same command directly in my cmd shell works as expected.
If I play around with the command, like entering an incorrect path or so it does return an error, but it looks like the --data-binary part is happily ignored. Adding, removing or using an incorrect filename for the --data-binary part is making no difference at all.
Could it be that you need root-level access to run the script correctly? Cron-jobs are run by root, while RM Server is usually run by an user account.
Good point...
Not sure tbh, if I run the script as is (so apart from the cron level) I can run it without sudo with my user account so I'd expect RM server to be able to run the same.
We discussed it yerterday with Philipp, and we have doubts about the "@my_file.json". Have you tried using the full path for the file (i.e. "/home/user_name/my_file.json")?
Yeah, tried that one also. If I deliberatly mistype the actual path the script throws an error, it just seems to ignore the existence of the json file. Even if I mistype that one it simply will say there is no problem. So wheter the json file is there or not does not make any difference. Looks like the final step in the process, taking the json and loading it to Elastic is simply not starting and it doesn't cause a problem.