Start RapidMiner Server with multiple threads

vipul_kumar

Hello all

 

I have recently updated to Rapid Miner Server version 8.0 . I wanted to have more understanding on the multi threading aspect of RM Server. I have a process designed and saved in RM Server and have exposed API's. A lot of requests will be fired to this API and I want them to be processed parallely using multiple threads , not sequentially.

What is the mechanism used by Rapid Miner to handle multi threading or parallel processing. 

SGolbert

Hi Kumal,

 

First have a look at these explanations of the new server structure:

 

https://community.rapidminer.com/t5/Community-Blog/RapidMiner-Server-8-0-will-be-out-soon-What-s-in-it-for-you/ba-p/42691

https://community.rapidminer.com/t5/Community-Blog/Get-ready-for-RapidMiner-Server-8-0/ba-p/42597

 

Then I will add in something: Parallelization is not new to Server 8.0. IF you have a Professional Licence there are a lot of operators that inherently work in parallel, and it was already possible in Server 7.x to run rapidminer processes at the same time. That said, the configuration that you need depends on which kind of parallelization you want.

 

Do you have computationally intensive processes with a lot of parameter optimization and feature selection? Then you probably need only one job agent with one job container.

 

Do you have multiple light processes (both on processor time and memory)? Then you probably can do better with one job agent and multiple job containers.

 

In my opinion it is not necessary to have more than one job agent per computer, unless you need redundancy. 

 

The configurations of RM Server 8.0 are a new and interesting topic!

 

 

vipul_kumar

Hi,

 

Thanks for the response.

I understand this concept of job agents that come into play in new RM 8 architecture which I can scale using multiple job agents connected to each queue.

But in my flow, I have exposed each process as api/webservice and I call this apis for scoring in which case, the flow does not go through job agents, instead it goes to execution engine. 

Is there any way of scaling the execution engine. Does is handle parallel requests. What is the official QPS(Queries per second) that execution engine can handle for a 16 GB Ram box.

SGolbert

The execution engine for web services is the same as in Server 7.x. It can handle parallel calls, however it still doesn't offer robustness or high availability. That will hopefully come in Server 8.1, stay tuned!