a mismatch between the current graph and the graph

Question

I am trying to train encoder decoder model with multispectral images having 9 channels but the code that i am running is downloading pretrained resnet101 weights which is trained on 3 channel images. Input Given by me:

net_input = tf.placeholder(tf.float32,shape=[None,None,None,9])
net_output = tf.placeholder(tf.float32,shape=[None,None,None,num_classes])

code for getting pretrained weights for Resnet101:

if args.model == "ResNet101" or args.model == "ALL":
      subprocess.check_output(['wget','http://download.tensorflow.org/models/resnet_v2_101_2017_04_14.tar.gz', "-P", "models"])
try:
    subprocess.check_output(['tar', '-xvf', 'models/resnet_v2_101_2017_04_14.tar.gz', "-C", "models"])
    subprocess.check_output(['rm', 'models/resnet_v2_101_2017_04_14.tar.gz'])
except Exception as e:
    print(e)
    pass

error that i am getting is:

error:

  Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [7,7,9,64] rhs 
  shape= [7,7,3,64]

what can be the solution here?

Answer 1

If you do not want to change the channels of input from 9 to 3, you need to change ResNet architecture input and second layer from 3 to 9 channels and add the final layers for inference. Notice, you will have to train it again.

Here is a full code, as an example, you just have to change channels to 9:

import matplotlib.pyplot as plt
import numpy as np
import os
import tensorflow as tf

_URL = 'https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip'
path_to_zip = tf.keras.utils.get_file('cats_and_dogs.zip', origin=_URL, extract=True)
PATH = os.path.join(os.path.dirname(path_to_zip), 'cats_and_dogs_filtered')

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'validation')

BATCH_SIZE = 32
IMG_SIZE = (160, 160)

train_dataset = tf.keras.utils.image_dataset_from_directory(train_dir,
                                                            shuffle=True,
                                                            batch_size=BATCH_SIZE,
                                                            image_size=IMG_SIZE)

validation_dataset = tf.keras.utils.image_dataset_from_directory(validation_dir,
                                                                 shuffle=True,
                                                                 batch_size=BATCH_SIZE,
                                                                 image_size=IMG_SIZE)


class_names = train_dataset.class_names

val_batches = tf.data.experimental.cardinality(validation_dataset)
test_dataset = validation_dataset.take(val_batches // 5)
validation_dataset = validation_dataset.skip(val_batches // 5)

AUTOTUNE = tf.data.AUTOTUNE

train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)
validation_dataset = validation_dataset.prefetch(buffer_size=AUTOTUNE)
test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)

preprocess_input = tf.keras.applications.ResNet50.preprocess_input

# Create the base model from the pre-trained model ResNet50

############### HERE YOU CHANGE TO 9 CHANNELS ###############

channels=3 

IMG_SHAPE = IMG_SIZE + (channels,)
base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,
                                               include_top=True,
                                               weights='imagenet')

image_batch, label_batch = next(iter(train_dataset))
feature_batch = base_model(image_batch)
print(feature_batch.shape)

base_model.summary()

## HERE IS WHERE THE MAGIC HAPPENS

base_model_config = base_model.get_config()

#### HERE THE CHANNELS WILL BE ALTERED TO 9 ####

base_model_config['layers'][0]["config"]["batch_input_shape"]=(None, 160, 160, channels)
base_model_config['layers'][1]["config"]["padding"]=((channels,channels), (channels,channels))

#######################################################

model=tf.keras.Model().from_config(base_model_config)

model.summary()

### HERE YOU ADD THE FINAL LAYERS FOR INFERENCE

inputs = tf.keras.Input(shape=(160, 160, channels))
x = base_model(inputs, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.2)(x)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)

base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=base_learning_rate),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.fit(x=train_dataset, epochs=2)

If there is any shape mismatch in the middle of the way, you know how to change it: base_model_config['layers'][x]....=........

Answer 2

There are a few resources about this idea, such as a blog post about transferring a ResNet on RGB data to multi-channel images here, and a relevant Colab Notebook. Below is a working example based on those resources:

import numpy as np
import tensorflow as tf


def tile_kernels(kernel, out_channels, batch_dim=-2):
    mean_1d = np.mean(kernel, axis=batch_dim).reshape(kernel[:, :, -1:, :].shape)
    tiled = np.tile(mean_1d, (out_channels, 1))
    return tiled


def reshape_model_input(model_orig, custom_model, input_channels):
    conf = custom_model.get_config()

    layer_to_modify = conf["layers"][2]["config"]["name"]

    layer_names = [conf['layers'][x]['name'] for x in range(len(conf['layers']))]

    for layer in model_orig.layers:
        if layer.name in layer_names:
            if layer.get_weights() != []:
                target_layer = custom_model.get_layer(layer.name)

                if layer.name == layer_to_modify:
                    kernels, biases = layer.get_weights()

                    kernels_extra_channels = np.concatenate((kernels,
                                                            tile_kernels(kernels, input_channels - 3)),
                                                           axis=-2)

                    target_layer.set_weights([kernels_extra_channels, biases])

                else:
                    target_layer.set_weights(layer.get_weights())


if __name__ == "__main__":
    from tensorflow.keras.applications import ResNet50V2
    resnet50 = ResNet50V2(weights='imagenet', include_top=False)  # load resnet50 here - can be done differently
    config = resnet50.get_config()
    img_height = ...
    img_width = ...
    input_channels = 7
    config["layers"][0]["config"]["batch_input_shape"] = (None, img_height, img_width, input_channels) # change the batch input shape to handle the different channel dimensions
    custom_resnet = tf.keras.models.Model.from_config(config)
    reshape_model_input(resnet50, custom_resnet, input_channels) # modify the custom model by reference
    custom_resnet(np.zeros((1, img_width, img_height, input_channels))) # just verifying that predicting with the new shape works in the custom model

This process just iterates over each layer in the original model and sets the corresponding weights in the custom model. To produce the additional n 3 x 3 channels (in your case, n = 4, as you want 7 total channels) for the input, the mean is taken across the 3 RGB dimensions then replicated (as can be seen in the tile_kernels function). Another aggregation function could be used, such as the max, min, median, etc. If you don't want any of the weights from the original model (as in, not pretraining but just require the architecture), just modifying the original model's configuration and creating a new model from it will create a randomly initialized model:

    resnet50 = ...
    config = resnet50.get_config()
    img_height = ...
    img_width = ...
    input_channels = ...
    config["layers"][0]["config"]["batch_input_shape"] = (None, img_height, img_width, input_channels)
    custom_resnet = tf.keras.models.Model.from_config(config)