The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Using the Select Attribute Operator during training
Contributor II
Hello,
I've trained a Neural Network by using a select attribute operator to reduce 85 attributes from a database view to 25 attributes used in training.
After training, I've saved the Neural Network to the repository and attempted to run it with unseen examples using the Read CSV operator.
The unseen sample contains a single record with only the 25 attributes that the select operator presented during training but I get an error from the neural Network:
Apr 11, 2012 9:54:23 PM SEVERE: java.lang.ArrayIndexOutOfBoundsException: DataRow: table index 85 of Attribute avg_score_4 is out of bounds.
If I present an example containing all 85 original variables, it works.
Have I misunderstood what the select operator is designed to do? I'm guessing I could make this work by making a database view containing only the variables I need but that seems redundant when the select operator is available.
What am I missing?
Many thanks,
M.
I've trained a Neural Network by using a select attribute operator to reduce 85 attributes from a database view to 25 attributes used in training.
After training, I've saved the Neural Network to the repository and attempted to run it with unseen examples using the Read CSV operator.
The unseen sample contains a single record with only the 25 attributes that the select operator presented during training but I get an error from the neural Network:
Apr 11, 2012 9:54:23 PM SEVERE: java.lang.ArrayIndexOutOfBoundsException: DataRow: table index 85 of Attribute avg_score_4 is out of bounds.
If I present an example containing all 85 original variables, it works.
Have I misunderstood what the select operator is designed to do? I'm guessing I could make this work by making a database view containing only the variables I need but that seems redundant when the select operator is available.
What am I missing?
Many thanks,
M.
0
Answers
with just your description it is hard to understand the problem you are having.
Please post your processes train and apply processes like it is described here: http://rapid-i.com/rapidforum/index.php/topic,4654.0.html
Best,
Nils
Apologies for not posting the process, I assumed I was doing something obviously stupid.
The training process: The "running" process:
Having tried changing various things around to try and understand this problem further it appears as if the only way I can get rid of this error is to store the data in the repository after the Set Role and Select Attributes operation and using it from there rather than the database (i.e not using a select or set role operator during training). I also need to remove the normalization stage.
Neither of those options are available to me for the real use case though, I need to try difference combinations of attributes during training rather than storing specific ones in the repository and the model works significantly better on the normalized data.
I can't stop thinking that I'm missing something obvious. Am I doing something stupid?
This is the only pair of processes I can get to work:
Learning: Applying: Regards,
M.
Probably got the wrong end of the stick, again, but if you want to train on one normalised set, and then apply that model on another set, then the 'Group Models' route looks the biz, it'll put the neural net and the normalisation models together into one model that can be applied elsewhere. Here's an example ... Hope that's useful.
I needed to set the "Create View" option on the Normalize data operator. Once I'd used the Group Models I started getting the error during the performance testing of the training process which led me to the solution.
Thanks for the help Haddock, I was starting to tear out what little hair I have remaining. The Group Models is a nifty tip that makes my life a lot simpler too.
Regards,
M.