The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
"Comparing training and testing accuracy to check over-fitting."
Avichandra
Member Posts: 3 
Contributor I
<?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
<parameter key="logverbosity" value="init"/>
<parameter key="random_seed" value="2001"/>
<parameter key="send_mail" value="never"/>
<parameter key="notification_email" value=""/>
<parameter key="process_duration_for_mail" value="30"/>
<parameter key="encoding" value="SYSTEM"/>
<process expanded="true">
<operator activated="true" class="retrieve" compatibility="8.2.000" expanded="true" height="68" name="Retrieve totalfludata20180511" width="90" x="45" y="238">
<parameter key="repository_entry" value="//Local Repository/totalfludata20180511"/>
</operator>
<operator activated="true" class="split_data" compatibility="8.2.000" expanded="true" height="103" name="Split Data" width="90" x="45" y="34">
<enumeration key="partitions">
<parameter key="ratio" value="0.8"/>
<parameter key="ratio" value="0.2"/>
</enumeration>
<parameter key="sampling_type" value="automatic"/>
<parameter key="use_local_random_seed" value="false"/>
<parameter key="local_random_seed" value="1992"/>
</operator>
<operator activated="true" class="select_attributes" compatibility="8.2.000" expanded="true" height="82" name="Select Attributes" width="90" x="246" y="187">
<parameter key="attribute_filter_type" value="subset"/>
<parameter key="attribute" value=""/>
<parameter key="attributes" value="year2017_18|year2016_17|year2015_16|wherereside.factor|weightlb|travel.factor|slaughter.factor|preg0nt.factor|pastmedreproductiveother.factor|pastmedreproductive.factor|pastmedre0ldisother.factor|pastmedre0ldisendstage.factor|pastmedre0ldis.factor|pastmedpcos.factor|pastmedothermed.factor|pastmedorgtransp.factor|pastmedneurodisstroke.factor|pastmedneurodisspi0lcord.factor|pastmedneurodisseizepilep.factor|pastmedneurodispnh.factor|pastmedneurodisother.factor|pastmedneurodisms.factor|pastmedneurodisintelldisab.factor|pastmedneurodiscp.factor|pastmedneurodis.factor|pastmedmetenddisthyroid.factor|pastmedmetenddisother.factor|pastmedmetenddisdiab.factor|pastmedmetenddis.factor|pastmedhivcd4.factor|pastmedhiv.factor|pastmedhepdisother.factor|pastmedhepdishepc.factor|pastmedhepdishepb.factor|pastmedhepdiscirr.factor|pastmedhepdis.factor|pastmedhemdissicklecell.factor|pastmedhemdisother.factor|pastmedhemdislymph.factor|pastmedhemdisleuk.factor|pastmedhemdis.factor|pastmedesld.factor|pastmedendomet.factor|pastmeddialysis.factor|pastmedcvdisvalvdis.factor|pastmedcvdisother.factor|pastmedcvdiscorartdis.factor|pastmedcvdiscongesthrtfail.factor|pastmedcvdiscongenhdis.factor|pastmedcvdiscardiomyop.factor|pastmedcvdis.factor|pastmedchronlundisother.factor|pastmedchronlundiscystfib.factor|pastmedchronlundiscopd.factor|pastmedchronlundisasth.factor|pastmedchronlundis.factor|pastmedcancerrad.factor|pastmedcancerchemo.factor|pastmedcancer.factor|pastmedautoimm.factor|menses.factor|medhistav.factor|largefarm.factor|heightin|gender.factor|foodprep.factor|fluvaccine_date.factor|fluvaccine.factor|flock.factor|farm.factor|exposure_swine.factor|exposure_poultry.factor|exposure_birds.factor|exposure.factor|exposeother.factor|exposehuman.factor|enrolling_site.factor|enrolldate|employed.factor|education.factor|edchrev_ab_ed.factor|cursympt_wheezing.factor|cursympt_stomachpain.factor|cursympt_sorethroat.factor|cursympt_sinuspain.factor|cursympt_shortnessbreath.factor|cursympt_shakingchills.factor|cursympt_rhinorrhea.factor|cursympt_other.factor|cursympt_incrsputum.factor|cursympt_headache.factor|cursympt_getoutofbed.factor|cursympt_fever.factor|cursympt_fatigue.factor|cursympt_diarrhea.factor|cursympt_coughsputum.factor|cursympt_cough.factor|cursympt_conjunctivitis.factor|cursympt_chestpain.factor|cursympt_chesthurt.factor|cursympt_bodyaches.factor|cursympt_appetite.factor|cursympt_0usea.factor|curmedsteroids.factor|curmedimmunosupsp___4.factor|curmedimmunosupsp___3.factor|curmedimmunosupsp___2.factor|curmedimmunosupsp___1.factor|curmedimmunosup.factor|breastfeeding.factor|bcmethod.factor|age|admit.factor|H3|H1|B"/>
<parameter key="use_except_expression" value="false"/>
<parameter key="value_type" value="attribute_value"/>
<parameter key="use_value_type_exception" value="false"/>
<parameter key="except_value_type" value="time"/>
<parameter key="block_type" value="attribute_block"/>
<parameter key="use_block_type_exception" value="false"/>
<parameter key="except_block_type" value="value_matrix_row_start"/>
<parameter key="invert_selection" value="false"/>
<parameter key="include_special_attributes" value="false"/>
</operator>
<operator activated="true" class="set_role" compatibility="8.2.000" expanded="true" height="82" name="Set Role" width="90" x="179" y="34">
<parameter key="attribute_name" value="GeneXpert"/>
<parameter key="target_role" value="label"/>
<list key="set_additional_roles"/>
</operator>
<operator activated="true" class="split_validation" compatibility="8.2.000" expanded="true" height="124" name="Validation" width="90" x="313" y="34">
<parameter key="create_complete_model" value="false"/>
<parameter key="split" value="relative"/>
<parameter key="split_ratio" value="0.7"/>
<parameter key="training_set_size" value="100"/>
<parameter key="test_set_size" value="-1"/>
<parameter key="sampling_type" value="automatic"/>
<parameter key="use_local_random_seed" value="false"/>
<parameter key="local_random_seed" value="1992"/>
<process expanded="true">
<operator activated="true" class="concurrency:parallel_random_forest" compatibility="8.2.000" expanded="true" height="103" name="Random Forest" width="90" x="112" y="34">
<parameter key="number_of_trees" value="10"/>
<parameter key="criterion" value="gain_ratio"/>
<parameter key="maximal_depth" value="20"/>
<parameter key="apply_pruning" value="true"/>
<parameter key="confidence" value="0.25"/>
<parameter key="apply_prepruning" value="true"/>
<parameter key="minimal_gain" value="0.1"/>
<parameter key="minimal_leaf_size" value="2"/>
<parameter key="minimal_size_for_split" value="4"/>
<parameter key="number_of_prepruning_alternatives" value="3"/>
<parameter key="random_splits" value="false"/>
<parameter key="guess_subset_ratio" value="true"/>
<parameter key="subset_ratio" value="0.2"/>
<parameter key="voting_strategy" value="confidence vote"/>
<parameter key="use_local_random_seed" value="false"/>
<parameter key="local_random_seed" value="1992"/>
<parameter key="enable_parallel_execution" value="true"/>
</operator>
<connect from_port="training" to_op="Random Forest" to_port="training set"/>
<connect from_op="Random Forest" from_port="model" to_port="model"/>
<portSpacing port="source_training" spacing="0"/>
<portSpacing port="sink_model" spacing="0"/>
<portSpacing port="sink_through 1" spacing="0"/>
</process>
<process expanded="true">
<operator activated="true" class="apply_model" compatibility="8.2.000" expanded="true" height="82" name="Apply Model (2)" width="90" x="45" y="34">
<list key="application_parameters"/>
<parameter key="create_view" value="false"/>
</operator>
<operator activated="true" class="performance" compatibility="8.2.000" expanded="true" height="82" name="Performance" width="90" x="179" y="34">
<parameter key="use_example_weights" value="true"/>
</operator>
<connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
<connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
<connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
<connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
<portSpacing port="source_model" spacing="0"/>
<portSpacing port="source_test set" spacing="0"/>
<portSpacing port="source_through 1" spacing="0"/>
<portSpacing port="sink_averagable 1" spacing="0"/>
<portSpacing port="sink_averagable 2" spacing="0"/>
</process>
</operator>
<operator activated="true" class="apply_model" compatibility="8.2.000" expanded="true" height="82" name="Apply Model" width="90" x="447" y="136">
<list key="application_parameters"/>
<parameter key="create_view" value="false"/>
</operator>
<connect from_op="Retrieve totalfludata20180511" from_port="output" to_op="Split Data" to_port="example set"/>
<connect from_op="Split Data" from_port="partition 1" to_op="Set Role" to_port="example set input"/>
<connect from_op="Split Data" from_port="partition 2" to_op="Select Attributes" to_port="example set input"/>
<connect from_op="Select Attributes" from_port="example set output" to_op="Apply Model" to_port="unlabelled data"/>
<connect from_op="Set Role" from_port="example set output" to_op="Validation" to_port="training"/>
<connect from_op="Validation" from_port="model" to_op="Apply Model" to_port="model"/>
<connect from_op="Validation" from_port="averagable 1" to_port="result 1"/>
<connect from_op="Apply Model" from_port="labelled data" to_port="result 2"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
<portSpacing port="sink_result 3" spacing="0"/>
</process>
</operator>
</process>
I have designed a random forest classification model with splitting the dataset into training and testing in a ratio of 0.8:0.2. I have validated the model. I got accuracy for the testing dataset. I want to check the over-fitting problem of my model. So, I want to compare accuracy for both training and testing data set. How to retrieve accuracy for both training and testing dataset from my model.
Tagged:
0
Best Answer
-
kypexin
RapidMiner Certified Analyst, Member Posts: 291 
Unicorn
Hi @Avichandra
In your process, add second PERFORMANCE operator and connect is with lab output of APPLY MODEL, this way you'll also get performance measure for 0.2 test split data.
1
Answers
Thank you very much! I got what I wanted to know.
How can I check performance for testing dataset from Auto Model process?