"Execute Python" operator takes forever to run

Goh · May 2020

I tried to implement the chapter 10 RapidMiner file from the classic RapidMiner textbook, 2nd edition by Vijay Kotu and Bala Deshpande. Dr Kotu is a member of the Advisory Board at RapidMiner.
(http://www.introdatascience.com/download.html)

Yes, I have installed the Execute Python operator from Marketplace. However, this operator takes forever to run. After waiting for more than 15 minutes, it still could not execute these codes. (However, I do not encounter this drag when using Jupyter).

Any idea if the Execute Python operator is not working?

The Python code is below, as copied from the textbook:

******************************************

from __future__ import print_function

import os

import numpy as np

import pandas as pd

import keras

from keras.datasets import mnist

from keras import backend as K

### Loads and returns MNIST data set in Pandas format

def rm_main():

# input image dimensions

img_rows, img_cols, ch_no = 28, 28, 1

num_classes = 10

# the data, shuffled and split between train and test sets

(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':

input_shape = (ch_no, img_rows, img_cols)

else:

input_shape = (img_rows, img_cols, ch_no)

x_train = x_train.reshape(x_train.shape[0], img_rows*img_cols*ch_no).astype('float32')

x_test = x_test.reshape(x_test.shape[0], img_rows*img_cols*ch_no).astype('float32')

x_train /= 255

x_test /= 255

# convert image vectors to data frames

df_train = pd.concat([pd.DataFrame(data={'y': y_train}), pd.DataFrame.from_dict(x_train)], axis=1)

setattr(df_train, "rm_metadata", {})

df_train.rm_metadata['y'] = ("nominal","label")

df_test = pd.concat([pd.DataFrame(data={'y': y_test}), pd.DataFrame.from_dict(x_test)], axis=1)

setattr(df_test, "rm_metadata", {})

df_test.rm_metadata['y'] = ("nominal","label")

# Prepare shape info

shape_data = np.array([['', 'rows', 'cols', 'ch', 'shape'],

['', img_rows, img_cols, ch_no, str(input_shape)]])

shape_result = pd.DataFrame(data=shape_data[1:,1:],

index=shape_data[1:,0],

columns=shape_data[0,1:])

setattr(shape_result, "rm_metadata", {})

shape_result.rm_metadata['rows'] = ("integer",None)

shape_result.rm_metadata['cols'] = ("integer",None)

shape_result.rm_metadata['ch'] = ("integer",None)

shape_result.rm_metadata['shape'] = ("text",None)

# Return results

return df_train, df_test, shape_result

hughesfleming68 · May 2020

Does your process finish or does "take forever to run" mean that it does not finish at all? Have you tried the execute python operator without any additional code? It is much easier to post your process as these errors are quick to identify if we can see them first hand without having to read through code. As for the execution of the code itself, you would have to compare the execution time against the console. Python is being called in a separate thread so if you look in performance monitor, you will see the jvm and python working. There will be performance discrepancies if you don't have enough memory or your cpu is saturated.

Goh · May 2020

The Execute Python operator did not finish running even if I waited more than 15 minutes. For my sanity, I had to exit the software and relaunch again.

As requested, the XML is here. Hope you could spot the issues from this textbook example. Thank you.

==============================

<?xml version="1.0" encoding="UTF-8"?><process version="9.6.000">

</context>

</operator>

</list>

</operator>

</list>

</operator>

</enumeration>

</enumeration>

</operator>

</operator>

</operator>

</operator>

</operator>

</operator>

</operator>

</operator>

</operator>

</process>

</operator>

</operator>

</operator>

</operator>

</process>

</operator>

</process>

Image: https://us.v-cdn.net/6030995/uploads/editor/3y/odvbu1wwtyyj.jpg

Abhi0693 · January 2022

Hello Team, could you please tell what was the solution provided in this case?

Our team is also facing similar issue where execute Python operator keeps on spinning but doesn't finish ; no errors.
Any help is appreciated.

Thanks,
Abhishek Bourai

MartinLiebig · January 2022

Hi @Abhi0693 ,

it is almost certain that your script just takes this long. Any chance you can add print statements into the script to write the progress to the log file?

Best,

Martin

Abhi0693 · January 2022

Thanks for response @mschmitz. Apologies for the delay in response.
This block keeps spinning as shown below.
Since i am new to Rapidminer , could you please through a light on the - package manager and conda environment. Do i need to create a separate environment in Anaconda?

Image: https://us.v-cdn.net/6030995/uploads/editor/md/likhk9dke9p2.png

Here is my Python script - some Python and Numpy operations

import pandas as pd

import numpy as np

def rm_main(data):

data['Time_diff']=(data['Timestamp']-data['Timestamp'].shift())

data['Time_diff']=data['Time_diff'].dt.total_seconds()

data['Time_diff_binary'] = np.where(data['Time_diff']!=900, 1, 0)

data['CUMSUM'] = data['Time_diff_binary'].cumsum()

cols=['Start_Timestamp','End_Timestamp','Anomaly_Type']

cols_add = data.columns[data.columns.str.startswith('IS_')].tolist()

new_cols = cols + cols_add

# new_df=pd.DataFrame(columns=['Start_Timestamp','End_Timestamp','Anomaly_Type','IS_1','IS_2','IS_3','IS_4'])

new_df=pd.DataFrame(columns=new_cols)

for idx, grp in data.groupby('CUMSUM'):

st_time = grp['Timestamp'].iloc[0]

et_time = grp['Timestamp'].iloc[-1]

anomaly = grp['Anomaly'].iloc[0]

selected_idx = grp['E_Distance'].idxmax()

dict_temp = {'Start_Timestamp' : st_time, 'End_Timestamp' : et_time, 'Anomaly_Type':anomaly}

for i in range(1,len(cols_add)+1):

dict_temp['IS_'+str(i)] = grp.loc[selected_idx]['IS_'+str(i)]

new_df = new_df.append(dict_temp, ignore_index=True)

# new_df = new_df.append({'Start_Timestamp' : st_time, 'End_Timestamp' : et_time, 'Anomaly_Type':anomaly,

# 'IS_1' : grp.loc[selected_idx]['IS_1'], 'IS_2' : grp.loc[selected_idx]['IS_2'],

# 'IS_3' : grp.loc[selected_idx]['IS_3'], 'IS_4' : grp.loc[selected_idx]['IS_4']},ignore_index = True)

new_df['Continuity']=np.where(new_df['Start_Timestamp']==new_df['End_Timestamp'],0,1)

return new_df

Howdy, Stranger!

Quick Links

Categories

Altair RapidMiner Community

GET HELP. LEARN BEST PRACTICES. NETWORK WITH YOUR PEERS.

"Execute Python" operator takes forever to run

Answers