Image augmentation in Keras mixed functional model

Question

I've created a mixed model in Keras, creating weights for metadata and image data and then combining them for the classification. Here's the model:

Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 80, 120, 3)] 0                                            
__________________________________________________________________________________________________
xception (Functional)           (None, 3, 4, 2048)   20861480    input_5[0][0]                    
__________________________________________________________________________________________________
input_4 (InputLayer)            [(None, 10)]         0                                            
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 3, 4, 8)      409608      xception[0][0]                   
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 4)            44          input_4[0][0]                    
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8)            0           conv2d_9[0][0]                   
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 12)           0           dense_3[0][0]                    
                                                                 global_average_pooling2d_1[0][0] 
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 4)            52          concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 1)            5           dense_4[0][0]                    
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________

I decided to augment the images due to imbalance. I used the following ImageDataGenerator:

aug = ImageDataGenerator(rescale=1/255.,
                        rotation_range=180,
                        height_shift_range=0.2,
                        width_shift_range=0.2,
                        brightness_range=[0.5,1.5],
                        channel_shift_range=100.0,
                        horizontal_flip=True,
                        vertical_flip=True,
                        shear_range=45.0)

I then compiled and attempted to train the model using ImageDataGenerator().flow():

epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)

history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)

This gives me an error:

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>

I've tried multiple versions of the code, but I'm just not familiar enough with the backend to correctly diagnose the problem. Can anyone help me with this?

---

Model code and MRE

Model code

LEARNING_RATE = 0.001

# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))

# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)

# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)

# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)


# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)

# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()

meta_train MRE

       age_approx  Unknown  female  male  head/neck  lower extremity  \
11655          45        0       0     1          0                0   
24502          60        0       0     1          0                1   
2524           50        0       1     0          0                1   
13894          60        0       1     0          0                0   
29325          45        0       1     0          0                1   

       oral/genital  palms/soles  torso  upper extremity  
11655             0            0      1                0  
24502             0            0      0                0  
2524              0            0      0                0  
13894             0            0      1                0  
29325             0            0      0                0 

img_train MRE

Array too large, see code here.

y_train.shape

(23188, 1)

Answer 1

first of all if you generate the following arrays your models works with no error;

import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001

# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))

# Model 1
meta_layer1 = tf.keras.layers.Dense(4, activation='relu')(meta_inputs)

# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)

# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4, activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1, activation='sigmoid')(merged_dense_layer)


# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)

# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()

Let's generate arrays as below;

y_array = np.zeros((23188, 1))
train_array = np.zeros((23188, 80, 120,3))
meta_array = np.zeros((23188, 10))

Now test your model;

model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)

as you can see it runs

>>> model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
 47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00

Now the problem is in your inputs:

(1) first convert all of your meta data in an array let's say it has batch of 1000 as a result it has to have a shape of meta_train.shape is (1000,10)

(2) your img_train.shape is (1000,80,120,3)

(3) your y_train.shape is (1000,1)

here 1000 can be 23188 as well. But let's assume you 1000 images, y_train (target) and 1000 meta data.

As you want to augment your image train you have to be careful here. Use as below;

import tensorflow as tf
import numpy as np
import time

here I created empty array as example but your origal data has to be in array as shapes of below, except instead of 1000, it has to be number of images you have.

y_train = np.zeros((1000, 1))
img_train = np.zeros((1000, 80, 120,3))
meta_train= np.zeros((1000, 10))

aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,
                        rotation_range=180,
                        height_shift_range=0.2,
                        width_shift_range=0.2,
                        brightness_range=[0.5,1.5],
                        channel_shift_range=100.0,
                        horizontal_flip=True,
                        vertical_flip=True,
                        shear_range=45.0)

aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false, you have to concatenate at each loop the meta_train and y_train for each image, so do not shuffle the images!
imagenerator = aug.flow(img_train, batch_size = img_train.shape[0], shuffle=False, sample_weight=None, seed=123, save_to_dir=None, save_prefix="", save_format="png", subset=None)

new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy()       # concatenate y_train at each loop to new_y


batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
    print(batches, time.strftime("%Y:%m%d-%H:%M:%S"))
    batches += 1
    img_train = np.concatenate((img_train, x_batch), axis = 0)
    new_meta = np.concatenate((new_meta, meta_train), axis = 0) # concatenate corresponding meta data
    new_y = np.concatenate((new_y, y_train), axis = 0) #concatenate corresponding label/target data!
    if batches >= 5:
        break

model.fit(x = [new_meta, img_train], y = new_y, epochs = 1, batch_size = 32)

NOTE: you can shuffle your data simply as below before using them in model.fit

idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)

img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]

Answer 2

This is not a direct answer, but will help to solve the question. There is missing information to solve your problem.

BUT first: the most obvious problem is; you provide meta_train which is a pandas data frame. Convert it to an array. Please try this first.

Second, if you still have problem, then model.fit may not handle a list [meta_train, flow], then you may have to find a way to provide two inputs to be handled by your model.

For that, you should provide the followings as MRE in order to reproduce your problem.

(1) Your code for the model even though one may generate same model based on the summary provided.

(2) the shape of y_train.