Help me understand how to implement Job Agents

Thomas_Ott

Hi Guys,

I have a brand new RapidMiner Server 8 installed and running. It's looking pretty darn good to me but I want to use multiple job-agents to control my executions.

 

I'm a bit confused on how to implement them. I did read the instructions but they're a bit terse and leave me confused. Help me understand how to implement them on the backend.

 

I have RM Server 8 running on my Windows laptop and installed to C:\RMServer8 directory. In there is the \job-agent diretory.  I go to Process Queues and see the Default queue, but now I want to create Queue1 and Queue2. So I add those queues and then click on install. I shutdown the Server and extract the new job-agent-queue1 zip file to the C:\RMServer8 directory.

 

Then what do I do I don't see any Queue 1 subdirectory. It gets really confusing for here. Do I do something like this:

 

|-job-agents

   |-queue1

   |-queue2

 

or something else

 

Thanks

 

 

Thomas_Ott

@sgenzer Bueller? Bueller?

MartinLiebig

Tom,

 

Job Agents are independed programs which can run on any computer. You can unzip them wereever you want - i guess even on your Pi. They need to have a network connection to the RM Server (JBOSS).

 

Each JA is attached to exactly one queue. You can specify these assignements in the JA's config.

 

Does this make sense?

 

~Martin

Thomas_Ott

@mschmitz Thanks, I suspected they were independent but I didn't know if they needed to be installed in the same directory where RM Server is installed.

 

So if I create a new queue called "General" and I download the Job Agent, is it preconfigured for the "General" queue or do I still need to go in the config file and edit it? I'm thinking no, but want your confirmation. 

 

This is kind of neat, actually, so the overhead of processing is really offloaded from the RM Server to these Job Agents. The Server is just an orchestration layer then. 

MartinLiebig

Tom,

 

i think they are already connected to the queue you are downloading it from. But you can also just copy your JA 10x and change it's name and queue it's connected to. Just have a look at the main config file. It's well documented.

 

~Martin

SGolbert

Hi Tom,

 

you must specify the queue they connect to in the configuration file. I recommend deleting the shipped zip file and just copying the included JA directory, so that you don't have to type in a lot of information.

 

If you copy it to another machine you also have to copy the server license file (and also all extentions that you use).

 

Then you have to change the following entries:

 

• jobagent.AppId: should be unique
• rapidminer.server.host: IP address of the server
• jobagent.queue.activemq.host: IP address of the server
• jobagent.queue.job.request: Name of the queue

 

Best,

Sebastian

 

Thomas_Ott

@SGolbert, thanks. I've already offloaded the Job Agents to their own directories. Since I'm working in a Windows environment, they tend to be a PITA. Eventually I'll move the Server off to seperate Linux box where I can make things smoother!

Thanks so much!

MartinLiebig

@Thomas_Ott didn't you try to run a server on a pi? I would love to see what happens if you run a JA on a pi

 

Best,

Martin

Thomas_Ott

@mschmitz I might just do that for a low level admin cron job stuff. Right now I have to run some jobs for clients. 

land

Hi Tom,

just be careful if you install / upgrade extensions. 

I usually overlook a job agent somewhere, who then sometimes gets the job requiring the (newer version of the) extension and then crashes the execution. Can be quite confusing and still PITA on Linux, especially if you regularly use extensions in projects that are under active development...
Of course it would be possible to write a RapidMiner process that distributes the job agents and executes them. Perfect job for your job agent running on the PI. It think that would improve the PITA to a completely new level of recursive PITA and help you being busy for a few weeks

Greetings,
Sebastian

RandyLeBlanc

The admin guide talks about something I would definitely, definitely do in a distributed, multi-server environment: share the server's extension directory. If you mount the extension dir of the server on all your JA's you'll never have an inconsistency or have to worry about distributing anything.

 

from https://docs.rapidminer.com/latest/server/administration/job-agents.html :

 

"To keep the extension installation effort and maintenance for RapidMiner Server and Job Agent to a minimum share the server’s extension directory between both. If it is not located on the same machine use your systems capabilities to enable a network share to provide the Job Agent access to the server’s extension directory. Open the Job Agent’s configuration at config/config.properties and add the path to the extension directory (jobagent.container.extensionsDir)."