GridSearchCV for learning rate

Question

I'm trying to use GridSearchCV to find the best parameters in a CNN but when I try to find the best combination for learning rate and batch size the code doesn't work (it works if I use instead of learning rate, epochs). Any idea why it is not working?

# NEURAL NETWORK
# ======================================================================================================================
# region Now we build and train the CNN=================================================================================
vgg16_model = keras.applications.vgg16.VGG16()  # We import VGG16 model to copy f rom it the structure


def create_model():
    model = Sequential()  # We create our model with Sequential
    for layer in vgg16_model.layers:  # For each layer of VGG16 we add the same layer to our model
        model.add(layer)

    model.layers.pop()  # We remove the last layer to change it to what we need
    for layers in model.layers:  # We make the layers comming from VGG16 not trainables
        layers.trainable = False

    model.add(Dense(2, activation='softmax'))  # We add the last layer to have only 2 outputs: Cracked, Uncracked
    opt = Adam(lr=lrn_rate)
    model.compile(optimizer = opt, loss='categorical_crossentropy', metrics=['accuracy'])

    return model


model = KerasClassifier(build_fn=create_model,epochs=2, verbose=0)


# ====================================================================================
# define the grid search parameters
batch_size = [16, 32]
# epochs = [2,5]
lr=[0.1,0.2]
param_grid = dict(batch_size=batch_size, learn_rate=lr)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1, cv=3)

X, Y = train_batches.next()    # Batch of images to be analyzed
#grid_result = grid.fit_generator(train_imgs, train_labels )

grid_result = grid.fit(X,Y)

The error I get is 'ValueError: learn_rate is not a legal parameter' but I did it just as in an example I found that worked for the epochs but not working for learning rate.

Answer 1

try: def create_model(lrn_rate):

Answer 2

The other answer is correct but not explaining. You need to provide the learning rate in create_model() function, thus your fixed function would look like this:

def create_model(lrn_rate):
    model = Sequential()  # We create our model with Sequential
    for layer in vgg16_model.layers:  # For each layer of VGG16 we add the same layer to our model
        model.add(layer)

    model.layers.pop()  # We remove the last layer to change it to what we need
    for layers in model.layers:  # We make the layers comming from VGG16 not trainables
        layers.trainable = False

    model.add(Dense(2, activation='softmax'))  # We add the last layer to have only 2 outputs: Cracked, Uncracked
    opt = Adam(lr=lrn_rate)
    model.compile(optimizer = opt, loss='categorical_crossentropy', metrics=['accuracy'])

    return model

Althought, this is very constraining. Are you sure you never want to test other optimizers? If you want to test other optimizers, this might come handy:

def create_model(lrn_rate, optimizer_="Adam"):
    model = Sequential()  # We create our model with Sequential
    for layer in vgg16_model.layers:  # For each layer of VGG16 we add the same layer to our model
        model.add(layer)

    model.layers.pop()  # We remove the last layer to change it to what we need
    for layers in model.layers:  # We make the layers comming from VGG16 not trainables
        layers.trainable = False

    model.add(Dense(2, activation='softmax'))  # We add the last layer to have only 2 outputs: Cracked, Uncracked
    if optimizer=="Adam":
        opt = Adam(lr=lrn_rate)
    elif optimizer=="SGD":
        opt = keras.optimizers.SGD(learning_rate=lrn_rate)
    else opt = keras.optimizers.Adadelta(learning_rate=lrn_rate) # need some default 
    # just an example of optimizers you might want to try
    model.compile(optimizer = opt, loss='categorical_crossentropy', metrics=['accuracy'])

    return model