"Running Process on Server"

Alexandros_Tzekas51

Hello Guys,

I 'm trying to run processes on Server but i fail everytime! I did go to VIEW DETAILS and get the ERROR MESSAGE:

"Out of memory- The process has failed as it ran out of memory. Try to run the process again with less data or choose another queue with larger Job Containers."

I did as Edin_Klapic had said to me ("You need to assign more RAM to the job agent. The file is agent.properties located in job-agent/home/config.") and assign more RAM to the job agent. I even tried to run a proccess with very very low RAM requirements (some kB) but didn't work!

Any ideas?

I give you the LOG:

2018-12-19 15:00:42.111  INFO 5855 --- [main] c.r.execution.jobcontainer.Application   : Starting Application on alex-HP-Compaq-8000-Elite-CMT-PC with PID 5855 (/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container/rapidminer-jobcontainer-0.4.7.jar started by root in /home/alex/rapidminer-server/rapidminer-server-9.0.4/bin)
2018-12-19 15:00:42.135  INFO 5855 --- [main] c.r.execution.jobcontainer.Application   : No active profile set, falling back to default profiles: default
2018-12-19 15:00:50.205  INFO 5855 --- [main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@2d1ef81a: startup date [Wed Dec 19 15:00:50 EET 2018]; root of context hierarchy
2018-12-19 15:00:54.428  INFO 5855 --- [main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService  'taskScheduler'
2018-12-19 15:00:58.428  INFO 5855 --- [main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-12-19 15:00:58.442  INFO 5855 --- [main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2018-12-19 15:00:58.602  INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService  : Trying to update JWT token from auth properties.
2018-12-19 15:00:58.600  INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog      : Current memory usage is 544MB / 500000MB
2018-12-19 15:00:58.623  INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService  : Updated JWT token from auth properties file.
2018-12-19 15:00:58.686  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Changing 'user.home' system property to '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4'
2018-12-19 15:00:58.686  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Generating new in-memory cipher key
2018-12-19 15:00:59.017  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Loading JDBC properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/jdbc/jdbc_properties.xml'
2018-12-19 15:00:59.580  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Setting pre-defined RapidMiner properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4/rapidminer.properties'
2018-12-19 15:00:59.597  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Initializing the RapidMiner Core
2018-12-19 15:00:59.597  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Setting execution mode to 'JOB_CONTAINER'
2018-12-19 15:00:59.711  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Loading bundled extension from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container/plugins'
2018-12-19 15:00:59.713  INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer         : Loading custom extensions from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/extensions'
2018-12-19 15:00:59.730  INFO 5855 --- [main] com.rapidminer                           : Performing upgrade  to version 9.0.003
2018-12-19 15:00:59.946  INFO 5855 --- [main] com.rapidminer                           : Set locale to en.
2018-12-19 15:01:00.106  INFO 5855 --- [main] PlatformUtilities                        : rapidminer.home is '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container'.
2018-12-19 15:01:00.164  INFO 5855 --- [main] com.rapidminer                           : Initializing license manager.
2018-12-19 15:01:00.438  INFO 5855 --- [main] com.rapidminer                           : Cannot access file system in execution mode JOB_CONTAINER. Not loading repositories.
2018-12-19 15:01:03.663  INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog      : Current memory usage is 544MB / 500000MB
2018-12-19 15:01:05.307  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Model Simulator
2018-12-19 15:01:05.307  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Time Series
2018-12-19 15:01:05.307  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Professional
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Process Scheduling
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: PMML
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: JDBC Connectors
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Advanced File Connectors
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Productivity
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Social Media
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Legacy Result Access
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Concurrency
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Remote Repository
2018-12-19 15:01:05.308  INFO 5855 --- [main] com.rapidminer                           : Register plugin: H2O
2018-12-19 15:01:05.309  INFO 5855 --- [main] com.rapidminer                           : Register plugin: Cloud Connectivity
2018-12-19 15:01:06.900  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Twitter Connection.
2018-12-19 15:01:07.458  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Salesforce Connection.
2018-12-19 15:01:07.458  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Amazon S3 Connection.
2018-12-19 15:01:07.459  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Azure Blob Storage Connection.
2018-12-19 15:01:07.472  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Google Cloud Storage Connection.
2018-12-19 15:01:07.472  INFO 5855 --- [main] com.rapidminer                           : Registered configurator Dropbox Connection.
2018-12-19 15:01:08.664  INFO 5855 --- [taskScheduler-3] c.r.e.j.service.OutOfMemoryWatchdog      : Current memory usage is 576MB / 500000MB

Best Regards,

Tzekas Alexandros

SGolbert

Hi Alexandros,

you should check the memory and number of containers assigned to the job agent. The safest configuration would be to use only one job container with memory equal to the system memory (this will run only one process at the time!). You can find the configuration file in:

$PATH_TO_SERVER/job_agent/home/config/agent.properties

The file is very small and is easy to understand. Let me know if it has worked out.

Regards,

Sebastian

MarcoBarradas

Hi I have some experience with the job agents.
Go to your agents file and get the config.properties file and edit it with a notepad.
File should be on 
[RM Folder]\job-agent\config
The line you need to change is:
jobagent.container.memoryLimit = 10000 
The number is on MB so check how many GB you may allocate. remember that the agent shares memory with the server and all the things that are set up on it so you may have a constraint there.

After setting the new memory of the agent you will need to restart it.
If you are working with Windows Server go to the bin folder of the agent and run 
stop-job-agent.bat and then rapidminer-jobagent.bat

Then launch your studio or server an trigger your process. 
Do you really have 500GB on that server? The log you shared doesn´t show the actual part in which your process failed and it was just using 576MB. 
Other things you may check is trying to  run your process with the LoopBaches operator in order to handle the information. Hope this helps you.

JessForbesRM

Hello,

Is your process something you can execute on studio without memory or any other errors?  

Alexandros_Tzekas51

Yes, at Studio i get no Errors.

Alexandros_Tzekas51

Guys, please i really really need some answers on these questions cause i have to start my experiment. I have only 2 weeks to get my results!

sgenzer

hi @Alexandros_Tzekas51 I'm sorry but I don't know job agents well enough to help you here. Most RM folks are on holiday right now...

Scott

Edin_Klapic

Hi @Alexandros_Tzekas51,

Happy New Year to you.

As @MarcoBarradas already said: The log file extract you posted does not state where the process failed so helping is difficult at this point.

How much RAM does your machine have?

Happy Mining,

Edin

P.S.:

When you write the name of the community members with a leading @ - it results in a notification service and the names are colored in orange.