Keras callback functions like ReduceLROnPlateau, ModelCheckpoint, EarlyStopping throwing NoneType Attribute Errors

Question

I'm using Keras (with TensorFlow back-end) to train a model using train_on_batch(), evaluate() and using callback functions on_epoch_end(). This is part of an original code present in the following link: https://www.kaggle.com/code/kmader/pretrained-inceptionv3-for-bone-age

Instead of using the given fit_generator(), I am trying to implement it's primitive version since fit_generator() is not working due to the error described in the following link: https://github.com/keras-team/keras/issues/8595

Given below is the kernel for the callback functions list-

from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau
weight_path="/kaggle/working/{}_weights.best.hdf5".format('bone_age')

checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, 
                             save_best_only=True, mode='min', save_weights_only = True)


reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)
early = EarlyStopping(monitor="val_loss", 
                      mode="min", 
                      patience=5) # probably needs to be more patient, but kaggle time is limited
callbacks_list = [reduceLROnPlat, checkpoint, early]

And the main training kernel-

epochs = 2
steps_per_epoch = 10

for epoch in range(epochs):
    print("Epoch {}/{}".format(epoch + 1, epochs))

    # Training phase
    for step in range(steps_per_epoch):
        # Get the next batch of training data and labels
        batch_X, batch_Y = next(train_gen)

        # Train on batch
        loss = bone_age_model.train_on_batch(batch_X, batch_Y)

        print("Step {" + str(step + 1) + "}/{" + str(steps_per_epoch) + "}

    # Validation phase
    val_loss, accuracy = bone_age_model.evaluate(test_X, test_Y)
    print("Validation Loss: " + str(round(val_loss, 4)))

#     Apply callbacks
    for callback in callbacks_list:
        callback.on_epoch_end(epoch, logs={'val_loss': val_loss})

(NOTE: I am running the code on Kaggle, neither on Google Colab nor on my local PC)

I noticed the following error:

AttributeError                            Traceback (most recent call last)
<ipython-input-56-28ff6d63cd15> in <module>()
     27 #     Apply callbacks
     28     for callback in callbacks_list:
---> 29         callback.on_epoch_end(epoch, logs={'val_loss': val_loss})  # Call the callback at the end of each epoch
     30 
     31 

/opt/conda/lib/python3.6/site-packages/Keras-2.1.3-py3.6.egg/keras/callbacks.py in on_epoch_end(self, epoch, logs)
    914     def on_epoch_end(self, epoch, logs=None):
    915         logs = logs or {}
--> 916         logs['lr'] = K.get_value(self.model.optimizer.lr)
    917         current = logs.get(self.monitor)
    918         if current is None:

AttributeError: 'NoneType' object has no attribute 'optimizer'

I tried to use the callback functions one by one, i.e., only having a single callback function in callbacks_list, but all callback functions cause an error.

1. callbacks_list = [reduceLROnPlat] gives the above stated error.
2. callbacks_list = [early] gives

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-28ff6d63cd15> in <module>()
     27 #     Apply callbacks
     28     for callback in callbacks_list:
---> 29         callback.on_epoch_end(epoch, logs={'val_loss': val_loss})  # Call the callback at the end of each epoch
     30 
     31 

/opt/conda/lib/python3.6/site-packages/Keras-2.1.3-py3.6.egg/keras/callbacks.py in on_epoch_end(self, epoch, logs)
    498             )
    499             return
--> 500         if self.monitor_op(current - self.min_delta, self.best):
    501             self.best = current
    502             self.wait = 0

AttributeError: 'EarlyStopping' object has no attribute 'best'

3. callbacks_list = [checkpoint] gives

AttributeError                            Traceback (most recent call last)
<ipython-input-38-28ff6d63cd15> in <module>()
     27 #     Apply callbacks
     28     for callback in callbacks_list:
---> 29         callback.on_epoch_end(epoch, logs={'val_loss': val_loss})  # Call the callback at the end of each epoch
     30 
     31 

/opt/conda/lib/python3.6/site-packages/Keras-2.1.3-py3.6.egg/keras/callbacks.py in on_epoch_end(self, epoch, logs)
    414                         self.best = current
    415                         if self.save_weights_only:
--> 416                             self.model.save_weights(filepath, overwrite=True)
    417                         else:
    418                             self.model.save(filepath, overwrite=True)

AttributeError: 'NoneType' object has no attribute 'save_weights'

I am a little new to this, and have studied it theoretically a little bit. Can someone please explain me whether I am using the functions wrong or is there an error in the keras modules (since I have seen many posts claiming old versions have errors in the module)? Since all modules threw some kind of error, I believe there is a problem in my implementation.

PS: I don't know whether this is relevant or not, but I am using pretrained models of InceptionV3 and Resnet.

Answer 1

Indeed, the callback cannot save the model weights without access to the model weights : )

When you use keras fit function it creates a callback list and passess the model to it the code of which follows:

if not isinstance(callbacks, callbacks_module.CallbackList):
    callbacks = callbacks_module.CallbackList(
        callbacks,
        add_history=True,
        add_progbar=verbose != 0,
        verbose=verbose,
        epochs=epochs,
        steps=epoch_iterator.num_batches,
        model=self,
    )

which itself sets the model for every callback passed to it the code of which again follows:

def set_model(self, model):
    super().set_model(model)
    if self._history:
        model.history = self._history
    for callback in self.callbacks:
        callback.set_model(model)

So in order to use the callbacks manually you need to set the models before calling them. In your example before the training loop you can add the following lines of code to fix the issue:

for callback in callback_list:
    callback.set_model(bone_age_model)

Then your checkpoint have access to model and for example your modelcheckpoint has access to weights in order to save them.

Remark: You can use CallbackList class in order to unify your callbacks without the need to make the loop every time see keras documentation for example use.