[SOLVED] How Big ist Big

michaelhecht

Hello,

I've some problems with a "big" data set. The set contains about 150000 lines and (due to nominal to binominal conversion) 56 attributes. If I send this to a neural network (RM version, not Weka) I get an error " PM SEVERE: Process failed: Cannot reset network to a smaller learning rate"

With only one percent of the data all works fine. RM 5 64bit uses about 8GB of memory and I have 12 available, so no problem of memory.

What is the problem? Is the data set too big?

Thanks in advance,

Michael

michaelhecht

It seems that this problem is known but not yet solved, isn't it?

http://rapid-i.com/rapidforum/index.php/topic,2324.0.html

michaelhecht

Ok, it is obviously a problem of the data. Since I did sampling of al little amount I "missed" the three data sets with missing labels. Increasing the fraction of sampled data increased also the probability of getting one data set with missing label.

So, it would be nice, to get a better fitting error message, i.e. not "Cannot reset network to a smaller learning rate" but "Cannot handle missing labels".