FailedPreconditionError: Error while reading resource variable *** from Container

Question

I am seeing following error on running model.fit with the horovod callbacks. If I skip the callbacks model.fit runs fine. Note: I am using horovod.tensorflow.keras package and my model is based on tensorflow.keras (I am not direclty using keras package but from the tensorflow)

FailedPreconditionError: Error while reading resource variable conv1d/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv1d/kernel/N10tensorflow3VarE does not exist.
     [[{{node conv1d/conv1d/ExpandDims_1/ReadVariableOp}}]]

Callbacks are as following

callbacks = [
    # Horovod: broadcast initial variable states from rank 0 to all other processes.
    # This is necessary to ensure consistent initialization of all workers when
    # training is started with random weights or restored from a checkpoint.
    hvd.callbacks.BroadcastGlobalVariablesCallback(0),
    TensorBoard(log_dir='boardlogs/{}'.format(datetime.datetime.now())) #report logs to tensorboard
]
# Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.
if hvd.rank() == 0:
    horovod_callbacks.append(tf.keras.callbacks.ModelCheckpoint('.horovod-cps/checkpoint-{epoch}.h5')


history = model.fit(X, y, epochs=500, batch_size=64, callbacks=callbacks, verbose=1 if hvd.rank() == 0 else 0)

Environment: Framework: tensorflow.keras Tensorflow Version 1.13.1 Keras Version 2.2.4-tf Horovod version: horovod==0.17.0.post1 Python version: 3.6