The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Determine multiple comparisons correction after evolutionary feature selection
University ProfessorI have been trying to employ Ingo's method from his recent posts on evolutionary feature selection. I have large dimension data sets that this has helped me with a lot. I now have performance about where I think it is close to optimized, based on previous work with more manual feature selection methods previously published. I'm interested in determining the P value for the classification the model is yielding. I can calculate the raw value from the binomial distribuition, but my feeling is I need to correct this for the multiple comparisons that go into the feature selection and cross validation steps. Is the log file, which lists only the number of rows, the number of different trials leading to the final result? Is there another way to know this if not?
Here is my xml:
<?xml version="1.0" encoding="UTF-8"?><process version="8.0.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="8.0.001" expanded="true" name="Process">
<parameter key="logverbosity" value="init"/>
<parameter key="random_seed" value="2001"/>
<parameter key="send_mail" value="never"/>
<parameter key="notification_email" value=""/>
<parameter key="process_duration_for_mail" value="30"/>
<parameter key="encoding" value="SYSTEM"/>
<process expanded="true">
<operator activated="true" class="read_excel" compatibility="8.0.001" expanded="true" height="68" name="Read Excel" width="90" x="45" y="34">
<parameter key="excel_file" value="/Users/scottsegalmd/Documents/AW computer study/Deep learning/All FaceGen data.xlsx"/>
<parameter key="sheet_number" value="1"/>
<parameter key="imported_cell_range" value="A1:DZ81"/>
<parameter key="encoding" value="SYSTEM"/>
<parameter key="first_row_as_names" value="false"/>
<list key="annotations">
<parameter key="0" value="Name"/>
</list>
<parameter key="date_format" value=""/>
<parameter key="time_zone" value="SYSTEM"/>
<parameter key="locale" value="English (United States)"/>
<list key="data_set_meta_data_information">
<parameter key="0" value="Cohort.true.binominal.label"/>
<parameter key="1" value="MRN.true.integer.id"/>
<parameter key="2" value="Age (years).true.integer.attribute"/>
<parameter key="3" value="Height (cm).true.integer.attribute"/>
<parameter key="4" value="Weight (kg).true.numeric.attribute"/>
<parameter key="5" value="BMI (kg/m2).true.numeric.attribute"/>
<parameter key="6" value="Mallampati Class.true.numeric.attribute"/>
<parameter key="7" value="Thyromental Distance (FBs).true.numeric.attribute"/>
<parameter key="8" value="Brow Ridge - high/low.true.numeric.attribute"/>
<parameter key="9" value="Brow Ridge - high/low ^2.true.numeric.attribute"/>
<parameter key="10" value="Brow Ridge Inner - down/up.true.numeric.attribute"/>
<parameter key="11" value="Brow Ridge Inner - down/up ^2.true.numeric.attribute"/>
<parameter key="12" value="Brow Ridge Outer - up/down.true.real.attribute"/>
<parameter key="13" value="Brow Ridge Outer - up/down ^2.true.real.attribute"/>
<parameter key="14" value="Cheekbones - low/high.true.numeric.attribute"/>
<parameter key="15" value="Cheekbones - low/high ^2.true.numeric.attribute"/>
<parameter key="16" value="Cheekbones - shallow/pronounced.true.real.attribute"/>
<parameter key="17" value="Cheekbones - shallow/pronounced ^2.true.real.attribute"/>
<parameter key="18" value="Cheekbones - thin/wide.true.real.attribute"/>
<parameter key="19" value="Cheekbones - thin/wide ^2.true.real.attribute"/>
<parameter key="20" value="Cheeks - concave/convex.true.real.attribute"/>
<parameter key="21" value="Cheeks - concave/convex ^2.true.real.attribute"/>
<parameter key="22" value="Cheeks - round/gaunt.true.numeric.attribute"/>
<parameter key="23" value="Cheeks - round/gaunt ^2.true.numeric.attribute"/>
<parameter key="24" value="Chin - forward/backward.true.numeric.attribute"/>
<parameter key="25" value="Chin - forward/backward ^2.true.numeric.attribute"/>
<parameter key="26" value="Chin - pronounced/recessed.true.real.attribute"/>
<parameter key="27" value="Chin - pronounced/recessed ^2.true.real.attribute"/>
<parameter key="28" value="Chin - retracted/jutting.true.numeric.attribute"/>
<parameter key="29" value="Chin - retracted/jutting ^2.true.numeric.attribute"/>
<parameter key="30" value="Chin - shallow/deep.true.real.attribute"/>
<parameter key="31" value="Chin - shallow/deep ^2.true.real.attribute"/>
<parameter key="32" value="Chin - small/large.true.real.attribute"/>
<parameter key="33" value="Chin - small/large ^2.true.real.attribute"/>
<parameter key="34" value="Chin - tall/short.true.real.attribute"/>
<parameter key="35" value="Chin - tall/short ^2.true.real.attribute"/>
<parameter key="36" value="Chin - wide/thin.true.numeric.attribute"/>
<parameter key="37" value="Chin - wide/thin ^2.true.numeric.attribute"/>
<parameter key="38" value="Eyes - down/up.true.numeric.attribute"/>
<parameter key="39" value="Eyes - down/up ^2.true.numeric.attribute"/>
<parameter key="40" value="Eyes - small/large.true.real.attribute"/>
<parameter key="41" value="Eyes - small/large ^2.true.real.attribute"/>
<parameter key="42" value="Eyes - tilt inward/outward.true.real.attribute"/>
<parameter key="43" value="Eyes - tilt inward/outward ^2.true.real.attribute"/>
<parameter key="44" value="Eyes - apart/together.true.real.attribute"/>
<parameter key="45" value="Eyes - apart/together ^2.true.real.attribute"/>
<parameter key="46" value="Face - brow-nose-chin ratio.true.real.attribute"/>
<parameter key="47" value="Face - brow-nose-chin ratio ^2.true.real.attribute"/>
<parameter key="48" value="Face - forehead-sellion-nose ratio.true.real.attribute"/>
<parameter key="49" value="Face - forehead-sellion-nose ratio ^2.true.real.attribute"/>
<parameter key="50" value="Face - heavy/light.true.real.attribute"/>
<parameter key="51" value="Face - heavy/light ^2.true.real.attribute"/>
<parameter key="52" value="Face - round/gaunt.true.numeric.attribute"/>
<parameter key="53" value="Face - round/gaunt ^2.true.numeric.attribute"/>
<parameter key="54" value="Face - tall/short.true.real.attribute"/>
<parameter key="55" value="Face - tall/short ^2.true.real.attribute"/>
<parameter key="56" value="Face - up/down.true.numeric.attribute"/>
<parameter key="57" value="Face - up/down ^2.true.numeric.attribute"/>
<parameter key="58" value="Face - wide/thin.true.numeric.attribute"/>
<parameter key="59" value="Face - wide/thin ^2.true.numeric.attribute"/>
<parameter key="60" value="Forehead - small/large.true.numeric.attribute"/>
<parameter key="61" value="Forehead - small/large ^2.true.numeric.attribute"/>
<parameter key="62" value="Forehead - tall/short.true.real.attribute"/>
<parameter key="63" value="Forehead - tall/short ^2.true.real.attribute"/>
<parameter key="64" value="Forehead - tilt forward/back.true.numeric.attribute"/>
<parameter key="65" value="Forehead - tilt forward/back ^2.true.numeric.attribute"/>
<parameter key="66" value="Head - thin/wide.true.real.attribute"/>
<parameter key="67" value="Head - thin/wide ^2.true.real.attribute"/>
<parameter key="68" value="Jaw - retracted/jutting.true.numeric.attribute"/>
<parameter key="69" value="Jaw - retracted/jutting ^2.true.numeric.attribute"/>
<parameter key="70" value="Jaw - wide/thin.true.real.attribute"/>
<parameter key="71" value="Jaw - wide/thin ^2.true.real.attribute"/>
<parameter key="72" value="Jaw - neck slope high/low.true.real.attribute"/>
<parameter key="73" value="Jaw - neck slope high/low ^2.true.real.attribute"/>
<parameter key="74" value="Jawline - concave/convex.true.numeric.attribute"/>
<parameter key="75" value="Jawline - concave/convex ^2.true.numeric.attribute"/>
<parameter key="76" value="Mouth - drawn/pursed.true.numeric.attribute"/>
<parameter key="77" value="Mouth - drawn/pursed ^2.true.numeric.attribute"/>
<parameter key="78" value="Mouth - happy/sad.true.real.attribute"/>
<parameter key="79" value="Mouth - happy/sad ^2.true.real.attribute"/>
<parameter key="80" value="Mouth - lips deflated/inflated.true.real.attribute"/>
<parameter key="81" value="Mouth - lips deflated/inflated ^2.true.real.attribute"/>
<parameter key="82" value="Mouth - lips large/small.true.real.attribute"/>
<parameter key="83" value="Mouth - lips large/small ^2.true.real.attribute"/>
<parameter key="84" value="Mouth - lips puckered/retracted.true.numeric.attribute"/>
<parameter key="85" value="Mouth - lips puckered/retracted ^2.true.numeric.attribute"/>
<parameter key="86" value="Mouth - lips thin/thick.true.real.attribute"/>
<parameter key="87" value="Mouth - lips thin/thick ^2.true.real.attribute"/>
<parameter key="88" value="Mouth - protruding/retracted.true.real.attribute"/>
<parameter key="89" value="Mouth - protruding/retracted ^2.true.real.attribute"/>
<parameter key="90" value="Mouth - tilt up/down.true.real.attribute"/>
<parameter key="91" value="Mouth - tilt up/down ^2.true.real.attribute"/>
<parameter key="92" value="Mouth - underbite/overbite.true.real.attribute"/>
<parameter key="93" value="Mouth - underbite/overbite ^2.true.real.attribute"/>
<parameter key="94" value="Mouth - up/down.true.real.attribute"/>
<parameter key="95" value="Mouth - up/down ^2.true.real.attribute"/>
<parameter key="96" value="Mouth - wide/thin.true.real.attribute"/>
<parameter key="97" value="Mouth - wide/thin ^2.true.real.attribute"/>
<parameter key="98" value="Mouth - chin distance - short/long.true.numeric.attribute"/>
<parameter key="99" value="Mouth - chin distance - short/long ^2.true.numeric.attribute"/>
<parameter key="100" value="Nose - bridge shallow/deep.true.real.attribute"/>
<parameter key="101" value="Nose - bridge shallow/deep ^2.true.real.attribute"/>
<parameter key="102" value="Nose - bridge short/long.true.real.attribute"/>
<parameter key="103" value="Nose - bridge short/long ^2.true.real.attribute"/>
<parameter key="104" value="Nose - down/up.true.numeric.attribute"/>
<parameter key="105" value="Nose - down/up ^2.true.numeric.attribute"/>
<parameter key="106" value="Nose - flat/pointed.true.numeric.attribute"/>
<parameter key="107" value="Nose - flat/pointed ^2.true.numeric.attribute"/>
<parameter key="108" value="Nose - nostril tilt down/up.true.real.attribute"/>
<parameter key="109" value="Nose - nostril tilt down/up ^2.true.real.attribute"/>
<parameter key="110" value="Nose - nostrils small/large.true.real.attribute"/>
<parameter key="111" value="Nose - nostrils small/large ^2.true.real.attribute"/>
<parameter key="112" value="Nose - nostrils wide/thin.true.real.attribute"/>
<parameter key="113" value="Nose - nostrils wide/thin ^2.true.real.attribute"/>
<parameter key="114" value="Nose - region concave/convex.true.real.attribute"/>
<parameter key="115" value="Nose - region concave/convex ^2.true.real.attribute"/>
<parameter key="116" value="Nose - sellion down/up.true.numeric.attribute"/>
<parameter key="117" value="Nose - sellion down/up ^2.true.numeric.attribute"/>
<parameter key="118" value="Nose - sellion shallow/deep (1).true.numeric.attribute"/>
<parameter key="119" value="Nose - sellion shallow/deep (1) ^2.true.numeric.attribute"/>
<parameter key="120" value="Nose - sellion shallow/deep (2).true.numeric.attribute"/>
<parameter key="121" value="Nose - sellion shallow/deep (2) ^2.true.numeric.attribute"/>
<parameter key="122" value="Nose - sellion thin/wide.true.numeric.attribute"/>
<parameter key="123" value="Nose - sellion thin/wide ^2.true.numeric.attribute"/>
<parameter key="124" value="Nose - short/long.true.real.attribute"/>
<parameter key="125" value="Nose - short/long ^2.true.real.attribute"/>
<parameter key="126" value="Nose - tilt down/up.true.numeric.attribute"/>
<parameter key="127" value="Nose - tilt down/up ^2.true.numeric.attribute"/>
<parameter key="128" value="Temples - thin/wide.true.real.attribute"/>
<parameter key="129" value="Temples - thin/wide ^2.true.real.attribute"/>
</list>
<parameter key="read_not_matching_values_as_missings" value="true"/>
<parameter key="datamanagement" value="double_array"/>
<parameter key="data_management" value="auto"/>
</operator>
<operator activated="true" class="optimize_selection_evolutionary" compatibility="8.0.001" expanded="true" height="103" name="Optimize Selection (Evolutionary)" width="90" x="313" y="34">
<parameter key="use_exact_number_of_attributes" value="false"/>
<parameter key="restrict_maximum" value="true"/>
<parameter key="min_number_of_attributes" value="1"/>
<parameter key="max_number_of_attributes" value="20"/>
<parameter key="exact_number_of_attributes" value="1"/>
<parameter key="initialize_with_input_weights" value="false"/>
<parameter key="population_size" value="5"/>
<parameter key="maximum_number_of_generations" value="30"/>
<parameter key="use_early_stopping" value="false"/>
<parameter key="generations_without_improval" value="2"/>
<parameter key="normalize_weights" value="true"/>
<parameter key="use_local_random_seed" value="false"/>
<parameter key="local_random_seed" value="1992"/>
<parameter key="user_result_individual_selection" value="false"/>
<parameter key="show_population_plotter" value="false"/>
<parameter key="plot_generations" value="10"/>
<parameter key="constraint_draw_range" value="false"/>
<parameter key="draw_dominated_points" value="true"/>
<parameter key="maximal_fitness" value="Infinity"/>
<parameter key="selection_scheme" value="tournament"/>
<parameter key="tournament_size" value="0.25"/>
<parameter key="start_temperature" value="1.0"/>
<parameter key="dynamic_selection_pressure" value="true"/>
<parameter key="keep_best_individual" value="false"/>
<parameter key="save_intermediate_weights" value="false"/>
<parameter key="intermediate_weights_generations" value="10"/>
<parameter key="p_initialize" value="0.5"/>
<parameter key="p_mutation" value="-1.0"/>
<parameter key="p_crossover" value="0.5"/>
<parameter key="crossover_type" value="uniform"/>
<process expanded="true">
<operator activated="true" class="concurrency:cross_validation" compatibility="8.0.001" expanded="true" height="145" name="Cross Validation" width="90" x="112" y="34">
<parameter key="split_on_batch_attribute" value="false"/>
<parameter key="leave_one_out" value="false"/>
<parameter key="number_of_folds" value="20"/>
<parameter key="sampling_type" value="automatic"/>
<parameter key="use_local_random_seed" value="false"/>
<parameter key="local_random_seed" value="1992"/>
<parameter key="enable_parallel_execution" value="true"/>
<process expanded="true">
<operator activated="true" class="naive_bayes" compatibility="8.0.001" expanded="true" height="82" name="Naive Bayes" width="90" x="179" y="187">
<parameter key="laplace_correction" value="true"/>
</operator>
<connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
<connect from_op="Naive Bayes" from_port="model" to_port="model"/>
<portSpacing port="source_training set" spacing="0"/>
<portSpacing port="sink_model" spacing="0"/>
<portSpacing port="sink_through 1" spacing="0"/>
</process>
<process expanded="true">
<operator activated="true" class="apply_model" compatibility="8.0.001" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
<list key="application_parameters"/>
<parameter key="create_view" value="false"/>
</operator>
<operator activated="true" class="performance_binominal_classification" compatibility="8.0.001" expanded="true" height="82" name="Performance" width="90" x="246" y="34">
<parameter key="main_criterion" value="first"/>
<parameter key="accuracy" value="true"/>
<parameter key="classification_error" value="false"/>
<parameter key="kappa" value="false"/>
<parameter key="AUC (optimistic)" value="false"/>
<parameter key="AUC" value="false"/>
<parameter key="AUC (pessimistic)" value="false"/>
<parameter key="precision" value="false"/>
<parameter key="recall" value="false"/>
<parameter key="lift" value="false"/>
<parameter key="fallout" value="false"/>
<parameter key="f_measure" value="false"/>
<parameter key="false_positive" value="false"/>
<parameter key="false_negative" value="false"/>
<parameter key="true_positive" value="false"/>
<parameter key="true_negative" value="false"/>
<parameter key="sensitivity" value="false"/>
<parameter key="specificity" value="false"/>
<parameter key="youden" value="false"/>
<parameter key="positive_predictive_value" value="false"/>
<parameter key="negative_predictive_value" value="false"/>
<parameter key="psep" value="false"/>
<parameter key="skip_undefined_labels" value="true"/>
<parameter key="use_example_weights" value="true"/>
</operator>
<connect from_port="model" to_op="Apply Model" to_port="model"/>
<connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
<connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
<connect from_op="Performance" from_port="performance" to_port="performance 1"/>
<connect from_op="Performance" from_port="example set" to_port="test set results"/>
<portSpacing port="source_model" spacing="0"/>
<portSpacing port="source_test set" spacing="0"/>
<portSpacing port="source_through 1" spacing="0"/>
<portSpacing port="sink_test set results" spacing="0"/>
<portSpacing port="sink_performance 1" spacing="0"/>
<portSpacing port="sink_performance 2" spacing="0"/>
</process>
</operator>
<operator activated="true" class="log" compatibility="8.0.001" expanded="true" height="82" name="Log" width="90" x="380" y="136">
<list key="log"/>
<parameter key="sorting_type" value="none"/>
<parameter key="sorting_k" value="100"/>
<parameter key="persistent" value="false"/>
</operator>
<connect from_port="example set" to_op="Cross Validation" to_port="example set"/>
<connect from_op="Cross Validation" from_port="performance 1" to_op="Log" to_port="through 1"/>
<connect from_op="Log" from_port="through 1" to_port="performance"/>
<portSpacing port="source_example set" spacing="0"/>
<portSpacing port="source_through 1" spacing="0"/>
<portSpacing port="sink_performance" spacing="0"/>
</process>
</operator>
<connect from_port="input 1" to_op="Read Excel" to_port="file"/>
<connect from_op="Read Excel" from_port="output" to_op="Optimize Selection (Evolutionary)" to_port="example set in"/>
<connect from_op="Optimize Selection (Evolutionary)" from_port="example set out" to_port="result 3"/>
<connect from_op="Optimize Selection (Evolutionary)" from_port="weights" to_port="result 1"/>
<connect from_op="Optimize Selection (Evolutionary)" from_port="performance" to_port="result 2"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="source_input 2" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
<portSpacing port="sink_result 3" spacing="0"/>
<portSpacing port="sink_result 4" spacing="0"/>
<portSpacing port="sink_result 5" spacing="0"/>
</process>
</operator>
</process>
One of my data files is also attached. Thanks in advance!
Tagged:
1
Answers
Hi,
I guess that ultimately you want to know how often a certain attribute was used? If yes, you could try to also log the used feature names changing the settings of the Log Operator. The resulting data set would need some post-processing but you can probably figure out the count for each attribute from it. Below is a quick and dirty process which shows this.
However, the way the log operator is used here means that only the best attribute set so far is logged. If you want to log ALL variants which have been tried, you would need to manually extract the feature names and store them. That process is much more complicated though...
Anyway, I hope this helps already,
Ingo
Thanks, I'll give this a try!