Different Results on Different Versions

User45825

I have performed an experiment using auto model on two different versions i.e. RapidMiner Studio 9.0.003 and 9.3.001. My input data, task, target, and features were exactly the same. I ran the auto model process with keeping the default settings, however, I got a different set of results. For instance, Naïve Base Accuracy and AUC values on version 9.0.003 were 40% and 0.836 respectively, while on the version 9.1.003 they were 64% and 0.535. 
It will be helpful to get the possible reasons of such discrepancies.

BalazsBarany

Hi,

AutoModel was improved a lot between 9.0 and 9.3. It is able to try new attribute combinations, optimize parameters etc. 

Also, with a small example set, your results depend a lot on the random selection into training and test sets.

The model and the results should be identical or very similar between the version. The AutoModel result is not expected to be.

You can always inspect the generated processes and compare the training set and the resulting model.

Regards,
Balázs

Telcontar120

If you want your results to be perfectly reproducible, you also need to set the local random seed (any number will do) because this will remove any potential impact from quasi-random process steps such as training and testing splits in cross-validation, etc.