[SOLVED] How to count 0 values in large set of attributes?

mikeb1029

I am working with TF-IDF data, and I want to count the number of terms (attributes) for each document (example) that have a TF-IDF value of 0.  At this time we have approximately 8000 examples and 4000 attributes.  It seems there is no way to count numeirc values, so I've tried several types of conversions.  I have tried converting the 0 values to missing attributes and counting those.  I have tried converting the numeric TF-IDF scores to polynomials and counting those.  I've also tried to discretize the TF-IDF values into 2 bins (0, and >0).  After the conversions I mentioned, I use an Extract Macro operator to count with parameters of macro type: statistics, statistics: count, attribute name: %{loop_attribute}, and the appropriate attribute value for the type of conversion I did previously in the process (an adaptation of Haddock's Missing Value Count workflow (http://www.myexperiment.org/workflows/1292.html).  All of these different conversions eventually provide the results I want, but the performance on a sample set of 850 examples and 165 attributes takes 5 minutes or more, so I know the performance with the full set of data would be way too slow for our needs.  

What am I doing wrong?  It seems that counting values like I want to do here should be something that RapidMiner could do in the blink of an eye?

MariusHelf

Hi,

what you could do is to convert non-zeros to missings (as you did before), and then use the Aggregate operator with count (no missings) as default aggregation.

Best regards,
Marius

mikeb1029

Thanks for your help, Marius.  Your suggestion made me realize that I could do this without resorting to the Loop Attributes operator, and that ended up speeding things up tremendously.

MariusHelf

:-) Nice to hear that!