The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Different performance between grid search and real test

YADAOYADAO Member Posts: 1 Learner I
edited October 2020 in Help
Hi guys,
I used optimize parameters to find the best parameters' values with 10 fold cross-validation, and it showed out that the best kappa was 0.397. But when I applied these parameter values to 10 fold cross-validation myself, the kappa was only 0.337.
What's wrong?
Thank you.
Tagged:

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,533 RM Data Scientist
    Hi @YADAO ,
    can it just be sampling differences? What happens if you change the seed of your x-val?

    Best,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • swordfswordf Member Posts: 3 Learner II

    Is there still someone on this issue now?
    I have the same problem, different results are presented using the suggested parameter by grid optimizer,

    I don't understand why, even if it's not the same it should be close at least
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,533 RM Data Scientist
    Hi @swordf ,
    close in number of standard_deviations, yes.
    Best,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
Sign In or Register to comment.