How to create image pairs for Siamese network using keras imagedatagenerator

Question

I want to create the positive and negative image pairs to train a Siamese network. My siamese network looks like following

def ResNet_model():
    
    baseModel = ResNet50(weights="imagenet", include_top=False,input_tensor=Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3)))
    for layer in baseModel.layers[:165]:
        layer.trainable = False
    
    headModel = baseModel.output
    headModel = GlobalAveragePooling2D()(headModel)
    model = Model(inputs=baseModel.input, outputs=headModel)
    
    return model


featureExtractor = ResNet_model()
imgA = Input(shape=(224, 224, 3))
imgB = Input(shape=(224, 224, 3))

view1_branch = featureExtractor(imgA)
view2_branch = featureExtractor(imgB)

all_features = Concatenate()([view1_branch, view2_branch]) # Lambda(euclidean_distance)([view1_branch, view2_branch]) # #Concatenate()([view1_branch, view2_branch]) 
hybridModel = Dense(500, activation="relu")(all_features)
hybridModel = Dropout(.3)(hybridModel)
hybridModel = Dense(500, activation="relu")(hybridModel)
hybridModel = Dense(500, activation="relu")(hybridModel)
hybridModel = Dense(500, activation="relu")(hybridModel)
hybridModel = Dropout(.25)(hybridModel)
hybridModel = Dense(500, activation="relu")(hybridModel)
hybridModel = Dense(500, activation="relu")(hybridModel)
hybridModel = Dense(10, activation="softmax")(hybridModel)
final_model = Model(inputs=[imgA,imgB], outputs=hybridModel,name="final_output") 

My folder structure is like following:

  |-- class_folder_a
  |-- img_1
  |-- img_2
  |-- img_3

  |-- class_folder_b
  |-- img_1
  |-- img_2
  |-- img_3 

So far I found some code here and here where all the images are in the same folder. How do i create image pairs ( positive: where both images belong to same class , negative: images belong to different class ) for folder structure like i mentioned. Any help would be appreciated .

Answer 1

You can try one or both of the following options:

1. Python generator Write your own python generator with the yield instead of return paradigm.

def data_generator(class_dir_a, class_dir_b, batchsize):
    while True:
      #load images from both directories 

            yield x_a, x_b, y

Read more about generators in Python in this tutorial.

In tensorflow-2, model.fit() accepts a python generator. It used to be that you had to call model.fit_generator().

2. Keras Generator

Folow this tutorial on how to build your own custom data generator by inheiting from tf.keras.utils.Sequence.

Just follow all the steps. When you get to the def __get_data(self) function: Adapt to your Siamese network by doing something like:

    def __get_data(self, batches):
        # Generates data containing batch_size samples

        path_batch_a = batches[self.X_col_a['path']]
        path_batch_b = batches[self.X_col_b['path']]
        


        X_batch_a = np.asarray([self.__get_input(x, y, self.input_size) for x, y in zip(path_batch_a)])

        X_batch_b = np.asarray([self.__get_input(x, y, self.input_size) for x, y in zip(path_batch_b)])

        y0_batch = ...

        return tuple(X_batch_a, X_batch_b), tuple([y0_batch..])

Hope this puts you on your way to a working generator.