The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Image Mining: Combine Global and Local Features Extraction

Legacy UserLegacy User Member Posts: 0 Newbie
edited November 2018 in Help
Dear all,

I would like to ask how one can combine the extraction of  global and local features in the same workflow.
My idea was:
-Multiple Color Image Opener
  -Global Feature Extractor from Single Image
  -Local Feature Extractor

But or of curse this is not possible, since MCIO has only one port.
I also tried to combine 2 MCIO operators and use a join(on FileName) but this is is not working. It says that there is no FileName attribute on the on the right side (that is the Local Feature Extractor).

The Local Feature Extractor itself has the following structure(taken from some post in the forum):
-Local Feature Extractor
  -Multiply
    -Point of Interest
    -Local Features

Thank you very much,
Elena

Answers

  • StaryVenaStaryVena Member Posts: 126 Contributor II
    Hello Elena,
    you can use Cartesian Product operator from Series extension to merge local features ExampleSet and global features ExampleSet. 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.008" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="generate_data" compatibility="5.3.008" expanded="true" height="60" name="Generate Data" width="90" x="246" y="30"/>
          <operator activated="true" class="generate_data" compatibility="5.3.008" expanded="true" height="60" name="Generate Data (2)" width="90" x="45" y="165">
            <parameter key="number_examples" value="1"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="5.3.008" expanded="true" height="76" name="Select Attributes" width="90" x="179" y="165">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="label"/>
            <parameter key="invert_selection" value="true"/>
            <parameter key="include_special_attributes" value="true"/>
          </operator>
          <operator activated="true" class="rename_by_generic_names" compatibility="5.3.008" expanded="true" height="76" name="Rename by Generic Names" width="90" x="313" y="165">
            <parameter key="generic_name_stem" value="btt"/>
          </operator>
          <operator activated="true" class="cartesian_product" compatibility="5.3.008" expanded="true" height="76" name="Cartesian" width="90" x="514" y="30"/>
          <connect from_op="Generate Data" from_port="output" to_op="Cartesian" to_port="left"/>
          <connect from_op="Generate Data (2)" from_port="output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Rename by Generic Names" to_port="example set input"/>
          <connect from_op="Rename by Generic Names" from_port="example set output" to_op="Cartesian" to_port="right"/>
          <connect from_op="Cartesian" from_port="join" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    All local features from one image will have same global features.

    Best,
    Vaclav
  • Legacy UserLegacy User Member Posts: 0 Newbie
    Hello,

    in the meantime I came up with this. 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.008" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="imageprocessing:multiple_color_image_opener" compatibility="1.3.003" expanded="true" height="60" name="MCIO(Local Features)" width="90" x="45" y="120">
            <list key="images">
              <parameter key="test" value="C:\Users\Elena\Dropbox\Master\Thesis\mcio"/>
            </list>
            <parameter key="assign_label" value="true"/>
            <process expanded="true">
              <operator activated="true" class="multiply" compatibility="5.3.008" expanded="true" height="94" name="Multiply" width="90" x="45" y="30"/>
              <operator activated="true" class="imageprocessing:poi_generator" compatibility="1.3.003" expanded="true" height="60" name="Point of interest generator" width="90" x="179" y="165">
                <list key="Points Table"/>
              </operator>
              <operator activated="true" class="imageprocessing:feature_extraction_operator2" compatibility="1.3.003" expanded="true" height="94" name="Local Feature Extractor with Points" width="90" x="402" y="30">
                <parameter key="include_position_in_img" value="false"/>
                <parameter key="window_width" value="15"/>
                <parameter key="window_height" value="15"/>
                <process expanded="true">
                  <operator activated="true" class="imageprocessing:color_to_grayscale" compatibility="1.3.003" expanded="true" height="60" name="Color to grayscale" width="90" x="45" y="30"/>
                  <operator activated="true" class="imageprocessing:line_haar_like_feature" compatibility="1.3.003" expanded="true" height="60" name="Line Haar-like feature" width="90" x="402" y="30"/>
                  <connect from_port="image 1" to_op="Color to grayscale" to_port="color image plus"/>
                  <connect from_op="Color to grayscale" from_port="grayscale image" to_op="Line Haar-like feature" to_port="grayscale image plus"/>
                  <connect from_op="Line Haar-like feature" from_port="feature" to_port="feature 1"/>
                  <portSpacing port="source_image 1" spacing="0"/>
                  <portSpacing port="source_image 2" spacing="0"/>
                  <portSpacing port="sink_feature 1" spacing="0"/>
                  <portSpacing port="sink_feature 2" spacing="0"/>
                </process>
              </operator>
              <connect from_port="color image plus" to_op="Multiply" to_port="input"/>
              <connect from_op="Multiply" from_port="output 1" to_op="Local Feature Extractor with Points" to_port="color image plus"/>
              <connect from_op="Multiply" from_port="output 2" to_op="Point of interest generator" to_port="img"/>
              <connect from_op="Point of interest generator" from_port="points of interest" to_op="Local Feature Extractor with Points" to_port="points"/>
              <connect from_op="Local Feature Extractor with Points" from_port="example set" to_port="Example set"/>
              <portSpacing port="source_color image plus" spacing="0"/>
              <portSpacing port="sink_Example set" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="imageprocessing:multiple_color_image_opener" compatibility="1.3.003" expanded="true" height="60" name="MCIO(Global Features)" width="90" x="45" y="30">
            <list key="images">
              <parameter key="city" value="C:\Users\Elena\Dropbox\Master\Thesis\mcio"/>
            </list>
            <parameter key="auto_adjust_contrast" value="true"/>
            <process expanded="true">
              <operator activated="true" class="imageprocessing:global_feature_extraction" compatibility="1.3.003" expanded="true" height="60" name="Global Feature Extractor from a Single Image" width="90" x="334" y="30">
                <process expanded="true">
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS(Hue)">
                    <parameter key="Color component" value="Hue"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS (Saturation)">
                    <parameter key="Color component" value="Saturation"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS (Value)">
                    <parameter key="Color component" value="Value"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:statistics" compatibility="1.3.003" expanded="true" name="Global statistics"/>
                  <operator activated="true" class="imageprocessing:create_color_image" compatibility="1.3.003" expanded="true" name="Create Color Image"/>
                  <operator activated="true" class="imageprocessing:histogram" compatibility="1.3.003" expanded="true" name="histogram(H)"/>
                  <connect from_port="color image plus 1" to_op="ClrToGS(Hue)" to_port="color image plus"/>
                  <connect from_port="color image plus 2" to_op="ClrToGS (Saturation)" to_port="color image plus"/>
                  <connect from_port="color image plus 3" to_op="ClrToGS (Value)" to_port="color image plus"/>
                  <connect from_port="color image plus 4" to_op="Global statistics" to_port="color image plus"/>
                  <connect from_op="ClrToGS(Hue)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus X"/>
                  <connect from_op="ClrToGS (Saturation)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus Y"/>
                  <connect from_op="ClrToGS (Value)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus Z"/>
                  <connect from_op="Global statistics" from_port="features" to_port="feature 2"/>
                  <connect from_op="Create Color Image" from_port="color image plus" to_op="histogram(H)" to_port="color image plus"/>
                  <connect from_op="histogram(H)" from_port="features" to_port="feature 1"/>
                  <portSpacing port="source_color image plus 1" spacing="0"/>
                  <portSpacing port="source_color image plus 2" spacing="0"/>
                  <portSpacing port="source_color image plus 3" spacing="0"/>
                  <portSpacing port="source_color image plus 4" spacing="0"/>
                  <portSpacing port="source_color image plus 5" spacing="0"/>
                  <portSpacing port="sink_feature 1" spacing="0"/>
                  <portSpacing port="sink_feature 2" spacing="0"/>
                  <portSpacing port="sink_feature 3" spacing="0"/>
                </process>
              </operator>
              <connect from_port="color image plus" to_op="Global Feature Extractor from a Single Image" to_port="color image plus"/>
              <connect from_op="Global Feature Extractor from a Single Image" from_port="example set" to_port="Example set"/>
              <portSpacing port="source_color image plus" spacing="0"/>
              <portSpacing port="sink_Example set" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="join" compatibility="5.3.008" expanded="true" height="76" name="Join" width="90" x="246" y="75">
            <parameter key="use_id_attribute_as_key" value="false"/>
            <list key="key_attributes">
              <parameter key="FileName" value="FileName"/>
            </list>
          </operator>
          <connect from_op="MCIO(Local Features)" from_port="example set" to_op="Join" to_port="right"/>
          <connect from_op="MCIO(Global Features)" from_port="example set" to_op="Join" to_port="left"/>
          <connect from_op="Join" from_port="join" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
  • Legacy UserLegacy User Member Posts: 0 Newbie
    Hi Vaclav,

    I managed to combine the IMMI, DFKI and a db table in the same process.
    I have no idea if this is a good practice, especially since I only used one table from the db for my test. However, this is my workflow and it is working. Your workflow with the Rename operator helped. However, in order to achieve the final result I had to use transpose twice since the DFKI operators store the path and not the FileName like IMMI, but from what I see there is no way to manipulate the entries of the example sets,only the attribute names can change. Therefore you can see the kind of ugly workaround with transpose-Rename-transpose. What do you think? Is it a good idea to have all feature extractions in one workflow (they will become more of course). Of course an easier-to-read option is saving the example sets and combine multiple retrieves I think.. 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.008" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="imagemining:read_images" compatibility="5.0.000" expanded="true" height="60" name="read_images" width="90" x="45" y="210">
            <parameter key="directory" value="C:\Users\Elena\Dropbox\Master\Thesis\mcio"/>
            <parameter key="label" value="test"/>
          </operator>
          <operator activated="true" class="imagemining:scalable_color" compatibility="5.0.000" expanded="true" height="76" name="Scalable Color" width="90" x="179" y="210"/>
          <operator activated="true" class="transpose" compatibility="5.3.008" expanded="true" height="76" name="Transpose" width="90" x="313" y="210"/>
          <operator activated="true" class="retrieve" compatibility="5.3.008" expanded="true" height="60" name="Retrieve testRepo" width="90" x="45" y="480">
            <parameter key="repository_entry" value="testRepo"/>
          </operator>
          <operator activated="true" class="rename_by_replacing" compatibility="5.3.008" expanded="true" height="76" name="Rename by Replacing" width="90" x="45" y="345">
            <parameter key="attribute_filter_type" value="regular_expression"/>
            <parameter key="regular_expression" value="^(?:[a-zA-Z]\:(\\|\/)|file\:\/\/|\\\\|\.(\/|\\))([^\\\/\:\*\?\&lt;\&gt;\&quot;\|]+(\\|\/){0,1})+$"/>
            <parameter key="replace_what" value="^(?:[a-zA-Z]\:(\\|\/)|file\:\/\/|\\\\|\.(\/|\\))([^\\\/\:\*\?\&lt;\&gt;\&quot;\|]+(\\|\/){0,1})+$"/>
            <parameter key="replace_by" value="$3.jpg"/>
          </operator>
          <operator activated="true" class="transpose" compatibility="5.3.008" expanded="true" height="76" name="Transpose (2)" width="90" x="179" y="345"/>
          <operator activated="true" class="imageprocessing:multiple_color_image_opener" compatibility="1.3.003" expanded="true" height="60" name="MCIO(Local Features)" width="90" x="45" y="120">
            <list key="images">
              <parameter key="city" value="C:\Users\Elena\Dropbox\Master\Thesis\mcio"/>
            </list>
            <parameter key="assign_label" value="true"/>
            <process expanded="true">
              <operator activated="true" class="multiply" compatibility="5.3.008" expanded="true" name="Multiply"/>
              <operator activated="true" class="imageprocessing:poi_generator" compatibility="1.3.003" expanded="true" name="Point of interest generator">
                <list key="Points Table"/>
              </operator>
              <operator activated="true" class="imageprocessing:feature_extraction_operator2" compatibility="1.3.003" expanded="true" name="Local Feature Extractor with Points">
                <parameter key="include_position_in_img" value="false"/>
                <parameter key="window_width" value="15"/>
                <parameter key="window_height" value="15"/>
                <process expanded="true">
                  <operator activated="true" class="imageprocessing:color_to_grayscale" compatibility="1.3.003" expanded="true" name="Color to grayscale"/>
                  <operator activated="true" class="imageprocessing:line_haar_like_feature" compatibility="1.3.003" expanded="true" name="Line Haar-like feature"/>
                  <connect from_port="image 1" to_op="Color to grayscale" to_port="color image plus"/>
                  <connect from_op="Color to grayscale" from_port="grayscale image" to_op="Line Haar-like feature" to_port="grayscale image plus"/>
                  <connect from_op="Line Haar-like feature" from_port="feature" to_port="feature 1"/>
                  <portSpacing port="source_image 1" spacing="0"/>
                  <portSpacing port="source_image 2" spacing="0"/>
                  <portSpacing port="sink_feature 1" spacing="0"/>
                  <portSpacing port="sink_feature 2" spacing="0"/>
                </process>
              </operator>
              <connect from_port="color image plus" to_op="Multiply" to_port="input"/>
              <connect from_op="Multiply" from_port="output 1" to_op="Local Feature Extractor with Points" to_port="color image plus"/>
              <connect from_op="Multiply" from_port="output 2" to_op="Point of interest generator" to_port="img"/>
              <connect from_op="Point of interest generator" from_port="points of interest" to_op="Local Feature Extractor with Points" to_port="points"/>
              <connect from_op="Local Feature Extractor with Points" from_port="example set" to_port="Example set"/>
              <portSpacing port="source_color image plus" spacing="0"/>
              <portSpacing port="sink_Example set" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="imageprocessing:multiple_color_image_opener" compatibility="1.3.003" expanded="true" height="60" name="MCIO(Global Features)" width="90" x="45" y="30">
            <list key="images">
              <parameter key="city" value="C:\Users\Elena\Dropbox\Master\Thesis\mcio"/>
            </list>
            <parameter key="auto_adjust_contrast" value="true"/>
            <process expanded="true">
              <operator activated="true" class="imageprocessing:global_feature_extraction" compatibility="1.3.003" expanded="true" name="Global Feature Extractor from a Single Image">
                <process expanded="true">
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS(Hue)">
                    <parameter key="Color component" value="Hue"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS (Saturation)">
                    <parameter key="Color component" value="Saturation"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:color_model_operator" compatibility="1.3.003" expanded="true" name="ClrToGS (Value)">
                    <parameter key="Color component" value="Value"/>
                  </operator>
                  <operator activated="true" class="imageprocessing:statistics" compatibility="1.3.003" expanded="true" name="Global statistics"/>
                  <operator activated="true" class="imageprocessing:create_color_image" compatibility="1.3.003" expanded="true" name="Create Color Image"/>
                  <operator activated="true" class="imageprocessing:histogram" compatibility="1.3.003" expanded="true" name="histogram(H)"/>
                  <connect from_port="color image plus 1" to_op="ClrToGS(Hue)" to_port="color image plus"/>
                  <connect from_port="color image plus 2" to_op="ClrToGS (Saturation)" to_port="color image plus"/>
                  <connect from_port="color image plus 3" to_op="ClrToGS (Value)" to_port="color image plus"/>
                  <connect from_port="color image plus 4" to_op="Global statistics" to_port="color image plus"/>
                  <connect from_op="ClrToGS(Hue)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus X"/>
                  <connect from_op="ClrToGS (Saturation)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus Y"/>
                  <connect from_op="ClrToGS (Value)" from_port="grayscale image plus" to_op="Create Color Image" to_port="grayscale image plus Z"/>
                  <connect from_op="Global statistics" from_port="features" to_port="feature 2"/>
                  <connect from_op="Create Color Image" from_port="color image plus" to_op="histogram(H)" to_port="color image plus"/>
                  <connect from_op="histogram(H)" from_port="features" to_port="feature 1"/>
                  <portSpacing port="source_color image plus 1" spacing="0"/>
                  <portSpacing port="source_color image plus 2" spacing="0"/>
                  <portSpacing port="source_color image plus 3" spacing="0"/>
                  <portSpacing port="source_color image plus 4" spacing="0"/>
                  <portSpacing port="source_color image plus 5" spacing="0"/>
                  <portSpacing port="sink_feature 1" spacing="0"/>
                  <portSpacing port="sink_feature 2" spacing="0"/>
                  <portSpacing port="sink_feature 3" spacing="0"/>
                </process>
              </operator>
              <connect from_port="color image plus" to_op="Global Feature Extractor from a Single Image" to_port="color image plus"/>
              <connect from_op="Global Feature Extractor from a Single Image" from_port="example set" to_port="Example set"/>
              <portSpacing port="source_color image plus" spacing="0"/>
              <portSpacing port="sink_Example set" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="join" compatibility="5.3.008" expanded="true" height="76" name="Join" width="90" x="179" y="75">
            <parameter key="use_id_attribute_as_key" value="false"/>
            <list key="key_attributes">
              <parameter key="FileName" value="FileName"/>
            </list>
          </operator>
          <operator activated="true" class="join" compatibility="5.3.008" expanded="true" height="76" name="Join (2)" width="90" x="447" y="165">
            <parameter key="use_id_attribute_as_key" value="false"/>
            <list key="key_attributes">
              <parameter key="FileName" value="id"/>
            </list>
          </operator>
          <operator activated="true" class="join" compatibility="5.3.008" expanded="true" height="76" name="Join (3)" width="90" x="581" y="255">
            <parameter key="use_id_attribute_as_key" value="false"/>
            <list key="key_attributes">
              <parameter key="FileName" value="file"/>
            </list>
          </operator>
          <connect from_op="read_images" from_port="images" to_op="Scalable Color" to_port="input"/>
          <connect from_op="Scalable Color" from_port="output" to_op="Transpose" to_port="example set input"/>
          <connect from_op="Transpose" from_port="example set output" to_op="Rename by Replacing" to_port="example set input"/>
          <connect from_op="Retrieve testRepo" from_port="output" to_op="Join (3)" to_port="right"/>
          <connect from_op="Rename by Replacing" from_port="example set output" to_op="Transpose (2)" to_port="example set input"/>
          <connect from_op="Transpose (2)" from_port="example set output" to_op="Join (2)" to_port="right"/>
          <connect from_op="MCIO(Local Features)" from_port="example set" to_op="Join" to_port="right"/>
          <connect from_op="MCIO(Global Features)" from_port="example set" to_op="Join" to_port="left"/>
          <connect from_op="Join" from_port="join" to_op="Join (2)" to_port="left"/>
          <connect from_op="Join (2)" from_port="join" to_op="Join (3)" to_port="left"/>
          <connect from_op="Join (3)" from_port="join" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
Sign In or Register to comment.