Optimize Deep Learning's network structure and parameters
MavenHi RapidMiner,
I'm doing regression using "Deep Learning" operator, I have 480 input features (this is a predictive maintenance problem, each feature is a meter reading, and we want to apply regression to predict the next-time-to-fail of an asset). After training the "Deep Learning" operator, the Root Mean Square Error (RMSE) applied on the training datatset is still quite high (from 0.08 to 0.2), although the training dataset is normalized into [-1; 1]. I also tried a lot of network structures, including increasing the number of hidden layers (up to 15), and increasing the number of nodes in each hidden layer (up 1000 nodes/layer). In some cases, doing so even increases the RMSE on the training dataset, which means the model is under-fitting. I used the default values for the other deep learning parameters, including the adaptive learning rate and rectifier activation.
So do you have any piece of advices for this situation, or is there any way to optimize the network structure? (coz I already tried the "Optimize Parameters" operator for "Deep Learning" but could not find the operator's parameters for network structure). Or is there any way to make the deep learning operator fit the training data better?
Thank you very much for your help!
Best regards,
Phivu
Answers
I'm not sure what you mean by "coz I already tried the "Optimize Parameters" operator for "Deep Learning" but could not find the operator's parameters for network structure." You tried an Optimize Parameter operator, embedded a Cross Validation operator and inside a Deep Learning operator....and nothing showed up?
Hi Thomas,
Yes. I attached the process and a screenshot here. In the screenshot, you can see the "hidden_layer_sizes" parameter is disabled when I choose the "Deep Learning" operator for optimization. So how can i enable it?
Huh, that's interesting. Try using Set Macros to inside the hidden layers and vary those values.
Hi Thomas
I have encountered the same issue, where the network hyperparameters are greyed out in the grid search UI. May I know how do I implement the method you have mentioned? Where do I place the Set Macros?
Thank you
Regards
Corse
Ah I think you have to toggle on 'List.'
How would I do that? The options are greyed out actually (i.e. Grid/List)
You need to select the parameters you would like to optimize first (the middle column on top). Once you have done that the selection between "Grid" and "List" becomes available.
Yes it works with the other parameters, but in this case the Parameters of hidden_layer_sizes and hidden_dropout_ratios are greyed out, clicking on them yields no further selection options.
Has anyone else encountered this issue? Apart from this, doing a manual search is possible, but it will be a hair thinning exercise. I shuddered at that thought.
Ok. Apologies, I did not read the full thread up until now. It is indeed a bit unusual that this is greyed out. Have you taken a look at the process that @Thomas_Ott posted above? In it, he set macros for the layer sizes within the "Optimize Parameters" operator, which are then used in the hidden layers panel of the "Deep Learning" operator, instead of using actual numerical values --> see screenshot.
If you do that, you should be able to select those said "Set Macros" operators within the optimization panel and enter the values that you would like to iterate through.
No problem, and thank you for the clarification.
In fact, in Thomas' earlier video (https://www.youtube.com/watch?v=R5vPrTLMzng&list=PLLdyjsPklEJvpxLHZM-llt40FLw-qEiZO&index=1), around the 8:40 min mark, we also see that hidden_layers for Neural network is greyed out, although it was not highlighted.
I am still unsure how to use the Set Macros using the script Thomas posted. Where should it be placed? Nested within the Optimize Parameters operater before Cross Validation?
Regards
Ben
You can just copy the XML of Thomas' process and paste it into your XML panel in RapidMiner. Then press the green checkmark and the process will appear in the process panel.
Hi FBT
I see. In essence, its just creating the no. of Set Macro operators to be the no. of hidden layers? I'm assuming the Set Macros can be used instead of defining one Set Macro for one hidden layer. For the optimizing, I will still have to change the macro value ( = no. of nodes) manually correct?
Regards
Ben
Hi,
not really. The whole point about using "Set Macro" operators is to make the parameters available within the "Optimize" operator in which you can set the values you would like to iterate through. Take a look at the screenshot:
I have renamed the two "Set Macro" operators from Thomas' process to make it a bit clearer in the optimization pane. You can now either use the grid selection by setting a range of values and the distance between values for the layer, or you directly define your desired values with a list (you would just need to check the "List" radio button).
Using the "Set Macros" operator with which several macros can be defined at once, brings you back to the problem of the selection being "greyed out" in the optimization pane. Hence you would need to use one "Set Macro" operator for every hidden layer you want to iterate through.
Hi FBT
I see, I can get a clearer picture now, your first statement was key. Now I face another strange Problem Message. I defined 3 layers as per Set Macros connected to Cross-Validation with a Deep Learning operator within it.
Any chance this error could be diagnosed? Has it got to do with strings vs. integers in Rapidminer?
Ok, you are right. Values set in the "Set Macro" operator are strings. This is usually not a problem in other contexts, as you can use eval() or parse() to convert it to a number. In your context, however, this does not work and we need to get a bit more creative. What seems to work is the following:
This process is very similar to the one before, however, instead of using "Set Macro" operators, you start by reading in an Excel or CSV File that consists of one column --> "LayerSize". In this column you have the hidden layer sizes you want to iterate through. It is important that you read the values in as numerical types, otherwise you will run into the same issue as before. If in doubt, use the import wizard.
Now, instead of setting the macro values directly, you can extract them from the read-in file, which contains your layer sizes, with the "Extract Macro" operator. By setting the macro type to "data value" you get a new parameter called "example index". This is basically just the row number in your Excel sheet and luckily you can iterate through it within the Optimize operator.
My Excel sheet had six values for hidden layers, hence I am iterating through from 1 to 6. You will need to adapt accordingly, depending on how many values for hidden layers you have. You need to use one "Extract Macro" operator for each hidden layer you would like to iterate through. If the values you would like to iterate through for different hidden layers are not identical, you would need to create more than one Excel/CSV file and read them in, as shown in the process above.
Hi FBT
Oh I see, I was trying to put eval() into the hidden layer size in the Deep Learning operator. So it was an issue with 'strings' being read by the Set Macro operator. I have implemented the workaround that you have suggested, and it works well, although sometimes the Deep Learning shows a Warning message about the macro not being defined.
On a separate note, do you know what the 'standardize' parameter in DL does? It says:
Once again, thank you for your detailed clarifications on this topic.
Regards
I believe that this parameter rescales your attributes to have a mean of 0 and and a standard deviation of 1. Also, it assumes that your data has a Gaussian distribution.
I see, so its essentially an in-built Normalize operator with the Z-scaling method. I'm assuming the user can choose between a manual normalization or simply checking this parameter?