How do you keep an overview of interacting processes?

denis_wolfrath

We are building our systems granularly, meaning instead of one big process, we're having many different sub processes that interact with each other.

During development and maintenance we often encounter the question

Which other processes are calling the one we are currently altering?

This is especially important when changing the process interfaces.

To my knowledge there is currently no possibility to check such process connections.

Therefore we build a small solution that…

…reads all processes in a given Folder.

…extracts all calling and called processes as well as their connection.

…creates a RapidMiner process showing the processes as operators and their respective connections.

Requirements

• Studio 7.4 or higher (9.3 tested)
• The “Text Processing” extension

Limitations

• Only works for local folders/repositories, not on the server. You need to copy/download the code from the server.

The following processes can be executed in order.

The code is attached to this post.

PROCESS 0100

Overview

Generate operator and connection information based on the internal process calls of a RM process folder.

Context

os_slash - System specific slash used to create paths

rm_process_folder - Folder containing all processes that shall be included in the overview

PROCESS 0200

Overview

Take the list of operators and connections created by process 0100_collect_process_information and create a RapidMiner process to visualize the process interfaces.

Open the newly created process and right click on free space -> Process Layout -> Arrange Operators

Context

ouput_rm_process_filename - The RapidMiner process created is stored under this name

Result

Although the result (new process as well as the connection table) suits our needs, we wanted a more visually appealing view of our system. Therefore we decided to try the tool gephi.

PROCESS 0300

Overview

Take the list of operators and connections created by process 0100_collect_process_information and create output nodes and edges for the visualisation tool gephi.

https://gephi.org/

Context

out_nodes_filename - The notes file is stored under this name

out_edges_filename - The edges file is stored under this name

Result

Some work with the settings of gephi resulted in the following picture.

The thickness of the links/arrows represents the number of process calls between the processes.

Going back to the initial question

Does anyone else have the need for such an overview and if so what is your current solution? I'd be interested in other ways to tackle this issue.

Is it planned to implement such feature within RapidMiner Studio in the future?

huayu

confused

IngoRM

First of all: this is really a cool idea.  To the best of my knowledge, there is currently no such feature on RapidMiner's roadmap so I really appreciate that you shared this with the community.  I often find myself using strict name schemas for my processes to make sure I understand where I use what but to be honest, things typically still get confusing for projects with LOTS of processes and dependencies.  This is where your solution would come in.

Thanks again,
Ingo

denis_wolfrath

We do exactly the same.

As you might have already noticed all our processes are numbered and there is a system to it. Said system always falls appart near the end of a project.

Kind regards
 Denis