The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

"Rapidminer random crashes"

hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
edited June 2019 in Help

I am having some severe problems with random crashes. I added a loop parameters operator to incrementally increase the population size of an optimize selection (evolutionary,parallel) operator. Nothing extreme, just nine increments from 5-50 and I am using six threads. There is 50% available memory while running the process so memory is not the issue.

 

The problem is that sometimes it works fine and then other times it crashes with an error. I don't get an error if I run the process without looping parameters.  Here is a snapshot of the exception if that helps:-

 

Kind regards,

 

Alex

 

Exception: java.util.ConcurrentModificationException
Message: null
Stack trace:
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
java.util.LinkedList$ListItr.next(LinkedList.java:888)
java.util.AbstractList.hashCode(AbstractList.java:540)
com.rapidminer.example.SimpleAttributes.hashCode(SimpleAttributes.java:84)
com.rapidminer.example.set.AbstractExampleSet.hashCode(AbstractExampleSet.java:366)
com.rapidminer.example.set.SplittedExampleSet.hashCode(SplittedExampleSet.java:174)
com.rapidminer.tools.ReferenceCache$TransparentWeakReference.(ReferenceCache.java:139)
com.rapidminer.tools.ReferenceCache$TransparentWeakReference.(ReferenceCache.java:131)
com.rapidminer.tools.ReferenceCache$Reference.(ReferenceCache.java:71)
com.rapidminer.tools.ReferenceCache$Reference.(ReferenceCache.java:59)
com.rapidminer.tools.ReferenceCache.newReference(ReferenceCache.java:204)
com.rapidminer.operator.ports.impl.AbstractPort.setData(AbstractPort.java:72)
com.rapidminer.operator.ports.impl.OutputPortImpl.deliver(OutputPortImpl.java:56)
com.rapidminer.operator.ModelApplier.doWork(ModelApplier.java:112)
com.rapidminer.operator.Operator.execute(Operator.java:1002)
com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:50)
com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:798)
com.rapidminer.operator.validation.ValidationChain.executeEvaluator(ValidationChain.java:234)
com.rapidminer.operator.validation.ValidationChain.evaluate(ValidationChain.java:335)
com.rapidminer.operator.validation.SlidingWindowValidation.estimatePerformance(SlidingWindowValidation.java:141)
com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:285)
com.rapidminer.operator.Operator.execute(Operator.java:1002)
com.rapidminer.operator.executor.ParallelUnitExecutor$OperatorExecution.run(ParallelUnitExecutor.java:59)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

Tagged:

Answers

  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn

    Yes,

     

    i know these errors. Best to make a feature request to RapidMiner that the data sets become thread safe :)

     

    To avoid that you usually can sacrifice some memory and time by inserting Materialize Data operator as first operator in the subprocess. If that does not work: I would rather paralelize the outer loop and use the standard, serial optimize selection operator. Unfortunately for that you would need our Jackhammer extension as you otherwise don't have a parallel loop. See below link for more details.

     

    Greetings,

      Sebastian

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    Hi Sebastian,

     

    I tried many things to solve the problem. I finally took out the loop parameters operater and setup a subprocess for each parameter. In my case that meant creating 10 sub processes which was not too bad. I have run the new process many times now without any crashes using the optimize selection evolutionary parallel without any issues. 

     

    It seems that combining a loop parameters operater with a threaded operator inside may work in some cases and crash in others. I will certainly make a feature request on this issue as it should work better.

     

    Thank you for making me aware of your Jackhammer extension. I will take a look. 

     

    Kind regards,

     

    Alex

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    I finally managed to get my process stable. Even separating out the loop parameters operator and creating individual subprocesses did not solve the problem completely. I was still getting random crashes. Random in the sense that it would run three or four times and then crash on the fifth.

     

    Putting a Materialize Data operator inside the optimize parameters evolutionary seems to solve the problem. The side effect is greater cpu usage. I had to drop from six threads to four. Even with four threads I was hitting > 80% CPU usage. Processing times have gone up significantly (about 2x)

     

    To be honest, this fix seems more like a hack than anything else but I have run out of ideas. The parallel processing extention seems very hit and miss. I could easily justify adding CPU's to this problem but I am concerned about scalability. Is anyone running processes on workstations with with greater than 20 threads? Are there any other issues to consider?

     

    Kind regards,

     

    Alex

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,533 RM Data Scientist

    Dear Alex,

     

    any chance you can sent me the process and maybe even the data via pm or mail? I am happy to forward this to our development to have a look on it. Of course i cannot promise anything.

     

    ~Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee-RapidMiner, Member, University Professor Posts: 1,996 RM Engineering

    Hi,

     

    I can tell you that we are already aware of this particular issue and looking into it.

     

    Regards,

    Marco

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    Thanks Marco and Martin,

     

    I will note any errors if they pop up and send them in if that will be helpful.

     

    Alex

     

     

Sign In or Register to comment.