"Set MongoDB Instance with a macro, is it possible?"

christian_ploch

Hello everyone,

long time lurker, first time poster.
I make it fast:

I've got a process, which reads with help of the NoSQL Connectors Plugin from my MongoDB. 
So far I was able to set a macro, which says what Collection should be read.

Is it also possible to set a macro, which sets the MongoDB "Instance" (e.g. mongodb://localhost:27017/{%db_name})?

I tried several methods, but it's not working here. Anyone an idea? Thanks!

kayman

You cannot really set the database name itself with a macro (to my awareness), but the branch operator could be one option. Below example picks the right database based on a macro value.

<?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="82" name="Set Macro" width="90" x="112" y="34">
        <parameter key="macro" value="database"/>
        <parameter key="value" value="mongo_1"/>
      </operator>
      <operator activated="true" class="branch" compatibility="9.1.000" expanded="true" height="82" name="choose" width="90" x="246" y="34">
        <parameter key="condition_type" value="expression"/>
        <parameter key="expression" value="equals(%{database}, &quot;mongo_1&quot;)"/>
        <parameter key="io_object" value="AHCDescribedCluster"/>
        <parameter key="return_inner_output" value="false"/>
        <process expanded="true">
          <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB" width="90" x="179" y="34">
            <parameter key="mongodb_instance" value="mongo_1"/>
            <parameter key="collection" value="coll1"/>
            <parameter key="sort_documents" value="false"/>
            <parameter key="limit_results" value="false"/>
            <parameter key="skip" value="0"/>
          </operator>
          <portSpacing port="source_condition" spacing="0"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_input 1" spacing="0"/>
          <portSpacing port="sink_input 2" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB (2)" width="90" x="179" y="34">
            <parameter key="mongodb_instance" value="mongo_2"/>
            <parameter key="collection" value="coll1"/>
            <parameter key="sort_documents" value="false"/>
            <parameter key="limit_results" value="false"/>
            <parameter key="skip" value="0"/>
          </operator>
          <portSpacing port="source_condition" spacing="0"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_input 1" spacing="0"/>
          <portSpacing port="sink_input 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Set Macro" from_port="through 1" to_op="choose" to_port="condition"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

Another, slightly more complex option, is to use the execute mongoDB command operator, where your command contains the 'macronised' collection name.

There are a few issues, one is that the command doesn't like linebreaks which makes it a bit difficult to generate, but this can be fixed as in below example. The example is using an aggregate, but it would probably work fine with a more standard command also.

If full flexibility is really needed you could try python instead

<?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="68" name="Set Macro" width="90" x="45" y="34">
        <parameter key="macro" value="mycoll"/>
        <parameter key="value" value="some_collection"/>
      </operator>
      <operator activated="true" class="text:create_document" compatibility="8.1.000" expanded="true" height="68" name="Create Document (4)" width="90" x="179" y="34">
        <parameter key="text" value="{aggregate: &quot;%{mycoll}&quot;,pipeline: [&#10;{$match: { &quot;Something&quot;: &quot;something&quot; } },&#10;&#10;{$project:{&#10;...&#10;}}&#10;],cursor: {},allowDiskUse: true}"/>
        <parameter key="add label" value="false"/>
        <parameter key="label_type" value="nominal"/>
      </operator>
      <operator activated="true" class="text:replace_tokens" compatibility="8.1.000" expanded="true" height="68" name="Replace Tokens (8)" width="90" x="313" y="34">
        <list key="replace_dictionary">
          <parameter key="\n" value=" "/>
        </list>
      </operator>
      <operator activated="true" class="nosql:mongodb_command" compatibility="9.0.000" expanded="true" height="68" name="Execute MongoDB Command (5)" width="90" x="447" y="34">
        <parameter key="mongodb_instance" value="my_database"/>
      </operator>
      <connect from_op="Create Document (4)" from_port="output" to_op="Replace Tokens (8)" to_port="document"/>
      <connect from_op="Replace Tokens (8)" from_port="document" to_op="Execute MongoDB Command (5)" to_port="command"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

christian_ploch

Yeah, I suspected that, it is not possible, hoped i overlooked something.
But I will give a try with your solutions @kayman

appreciate it! Thanks!

MartinLiebig

@christian_ploch
i think it's technically possible. you can hack the xml and just add %{macroName} at the right setting. Not a nice work around though.

+ We are working on it and the problem will be fixed "soon".

BR,
Martin

christian_ploch

@mschmitz Yeah, already tried it with the process itself and also with the configurable-mongodb_instance.xml.
If I play with the latter it throws me "Invalid JSON/BSON Object" (when I add the macro in the xml into the connection string). While the process gives me the "collection is not available".

Thanks anyway