GridSearchCV, Pipeline and Functional Model

Question

Problem with predict function of Functional Model

I am trying to combine, nested-cross validation and pipeline with my Functional Model. This is the code:

• binaryModel = is hereby a Functional ANN

grid = dict(ann__n_neurons=[2], ann__num_hidden=[2], ann__used_optimizer=["adam"], 
                          ann__l1_reg=[0.0], ann__l2_reg=[0.0], ann__learning_rate=[0.01],
                          ann__dropout_rate=[0.0])
X, y = prepare_dataset("", short, bin_categorical, "",
                       continous_to_binary, target)

cv_outer = KFold(n_splits=10, shuffle=True, random_state=1) #outer cross-validatio 10 times, to test model 
# enumerate splits
outer_results = list()
i=0
for train_ix, test_ix in cv_outer.split(X):
    print("Outer-Split: ",i)
    i+=1
    # split data
    X_train, X_test = X.iloc[train_ix], X.iloc[test_ix]
    y_train, y_test = y[train_ix], y[test_ix]
    # configure the cross-validation procedure
    cv_inner = KFold(n_splits=3, shuffle=True, random_state=1) #inner cross-validation 3 times, to configure model
    
    # define the model
    ann = KerasClassifier(build_fn=binaryModel, input_shape=X_train.shape[1],
                          batch_size=32,
                          epochs=10, validation_split=0.2)

    # define search
    pipe = Pipeline(steps=[('scaler', StandardScaler()), ('ann', ann)])
    
    # define the grid search 
    cv = GridSearchCV(
        pipe, grid, n_jobs=1, cv=cv_inner,refit=True)
    # execute search 
    cv.fit(X_train, y_train, ann__verbose=0)
    
    print('Best score and parameter combination = ')
    print(cv.best_score_)    
    print(cv.best_params_)
    print(cv.best_estimator_)
    y_predicted = cv.predict(X_test)

Output:

Best score and parameter combination = 
0.8449265360832214
{'ann__dropout_rate': 0.0, 'ann__l1_reg': 0.0, 'ann__l2_reg': 0.0, 'ann__learning_rate': 0.01, 'ann__n_neurons': 2, 'ann__num_hidden': 2, 'ann__used_optimizer': 'adam'}
Pipeline(steps=[('scaler', StandardScaler()),
                ('ann',
                 <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object at 0x7efef01ffd30>)])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-28-3f1c5b78794d> in <module>
     61     print(cv.best_params_)
     62     print(cv.best_estimator_[1])
---> 63     y_predicted = cv.predict(X_test)
 AttributeError: 'Functional' object has no attribute 'predict_classes'

How to make predictions with the final best model?

• my outcome is a Pipeline but i cant use the predict function, why?
• i want to use the predict function to evaluate each fold (accuracy, sensitivity and so on...)

Answer 1

The problem is that a Keras Functional API model doesn't have a 'predict_classes' attribute, which is what sklearn's GridSearchCV uses to perform the predictions, only Sequential Keras models have it. I have been running into the same problem, what I would suggest trying is to implement your own GridSearchCV, or try out https://github.com/autonomio/talos which seems promising, though I have not tried it myself yet.