"Customized X-fold cross-validation"

_paul_

Hi,

I want to perform an X-fold cross-validation which however does not
operate on sets that are defined by RapidMiner's XValidation "sampling_type"
parameter but on sets which are constructed using a "marker" in the
examples provided by an ExampleSource operator.

To be more accurate, my input examples (pairs of feature vectors and
labels) used for classification contain an attribute that defines the
application this particular example was extracted from. Let's say the
examples come from three applications "A", "B", and "C" and each
example contains an attribute holding one of the three characters.

Based on this, I would like to perform a 3-fold cross-validation where in
a first run, examples from "A" are excluded and tested on examples from
"B" and "C" ...

Is there an operator for that in RapidMiner?

Regards,
Paul

land

Hi Paul,
there is a special variant of the XValidation called BatchXValidation, where it uses an attribute with the special role batch to define the splitting sets. I post a process below, making use of this operator.

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="target_function"	value="sum classification"/>
    </operator>
    <operator name="AttributeSubsetPreprocessing" class="AttributeSubsetPreprocessing" expanded="yes">
        <parameter key="condition_class"	value="attribute_name_filter"/>
        <parameter key="attribute_name_regex"	value="att1"/>
        <operator name="BinDiscretization" class="BinDiscretization">
            <parameter key="number_of_bins"	value="3"/>
            <parameter key="range_name_type"	value="short"/>
        </operator>
    </operator>
    <operator name="ChangeAttributeRole" class="ChangeAttributeRole">
        <parameter key="name"	value="att1"/>
        <parameter key="target_role"	value="batch"/>
    </operator>
    <operator name="BatchXValidation" class="BatchXValidation" expanded="yes">
        <operator name="DecisionTree (2)" class="DecisionTree" breakpoints="before,after">
        </operator>
        <operator name="TestChain (2)" class="OperatorChain" expanded="no">
            <operator name="ModelApplier (2)" class="ModelApplier">
                <list key="application_parameters">
                </list>
            </operator>
            <operator name="Performance (2)" class="Performance">
            </operator>
        </operator>
    </operator>
</operator>

Greetings,
  Sebastian

_paul_

Hi Sebastian,

sorry for the late answer. ;-)

I didn't really get the idea of your model. How does the BatchXValidation operator
work? I assume that it relies on the operator ChangeAttributeRole (also on
AttributeSubsetProcessing?), but it's not clear to me how the operators
communicate.

Let's say I've this example set:

att1;att2;att3;label
1; 2; A; YES
2; 2; A; NO
3; 4; B; YES
1,4; C; NO
2,4; C; NO
4,4; C; YES

and I would like to have a 3-fold cross-validation where in each
run of the validation I want to exclude the examples belonging
to the class (A,B,C) specified by attribute "att3".

Thus, the cross-validation would look something like:
1. step: Exclude examples from class A, learn model for examples
  from class B and C, and apply this model to examples from class A
2. step: Exclude B, learn for A and C, apply to B
3. step: Exlcude C, learn for A and B, apply to C

How can I model this type of validation?
And is there a way to figure out within the BatchXValidation operator
which examples are currently excluded (like att3=A in 1. step)?

Thank you.

Regards,
Paul

land

Hi Paul,
the BatchXValidation does not divide examples of the same batch over folds. Instead the batches are always completedly swapped into one fold.
So, if you define your attribute att3 as the batch attribute and set the number of validations of the BatchXValidation on the numbers of different values in att3, this should do the trick.
In the first round the first fold is removed, containing all As and learning will be carried out on the remaining folds. And so on...

I hope this clarifies it?

Greetings,
  Sebastian

_paul_

Hi Sebastian,

yes, the x-validation is now clear. Thank you.

Best,
Paul