The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

RM Cloud benchmarking

sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
edited November 2018 in Help
Hi...is anyone out there creating some benchmarks on the performance of the various RM Cloud engines: regular, large, and x-large?  I have started and am seeing some odd results.

Thanks.

Scott

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,533 RM Data Scientist
    What do you mean by odd?
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee-RapidMiner, Member, University Professor Posts: 1,996 RM Engineering
    Hi,

    there are some things to consider when looking at this.

    1) The Cloud machines may have to be spawned when needed. Depending on the current usage, your job may immediately get a free machine or it will have to wait for a new one to spawn and be prepared. This can take a couple of minutes before your job is actually executed. However this time is NOT included in the consumed time of your process, so you do not pay for that.

    2) Jobs you submit to the Cloud run in a fully virtualized environment to protect the integrity of your own job and any subsequent jobs. This reduces performance a bit, but it provides a major boost in security.

    3) Once the machine is prepared, your job will be started on the machine in question. However this does not yet equal the press of your local "Play" button, as a fresh execution environment needs to be set up first. While this does not take long, it takes a few seconds.

    Taking the above factors into account, the execution time of your process usually beats your Desktop/Laptop.
    The below process with 1 million examples and 50 attributes executed by RapidMiner Studio 6.2 takes on my machine about 17 seconds, on Regular Cloud it takes about 12 seconds, on Large Cloud 9 seconds and on X-Large Cloud 7 seconds on average. Taking the startup time of a few seconds into account (see above), that's pretty good I'd say. Note that other operators which are not yet rewritten to utilize unlimited CPU cores (or simply cannot due to the nature of the operator) will not exhibit significant speed up when selecting a bigger machine. For those, the advantage comes to down to being able to use more memory. And that is quite useful for a lot of operators.


    What else can happen? Well it might occur that X-Large is slower for a certain process than Regular? Of course we also ran a multitude of performance tests on all machines, so here is what can happen:

    1) Your process might very short. Couple this with full virtualization and you are bound to get some time variance. In our tests for equal jobs (which were not maxing out memory usage) X-Large was usually faster, however there were also times where Regular beat X-Large by a bit. It depends on what you are actually doing.

    2) In terms of computing power, all machines use state-of-the-art Intel Server CPUs. The real difference (as shown in the description of each machine) is the available memory. If you submit a job that is neither memory nor parallel bound, your best bet is a Regular machine.


    I hope this information gave you some insight into things!

    Regards,
    Marco
Sign In or Register to comment.