Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset

Question

I am trying to train an image classifier on an unbalanced training set. In order to cope with the class imbalance, I want either to weight the classes or the individual samples. Weighting the classes does not seem to work. And somehow for my setup I was not able to find a way to specify the samples weights. Below you can read how I load and encode the training data and the two approaches that I tried.

Training data loading and encoding

My training data is stored in a directory structure where each image is place in the subfolder corresponding to its class (I have 32 classes in total). Since the training data is too big too all load at once into memory I make use of image_dataset_from_directory and by that describe the data in a TF Dataset:

train_ds = keras.preprocessing.image_dataset_from_directory (training_data_dir,
                                                             batch_size=batch_size,
                                                             image_size=img_size,
                                                             label_mode='categorical')

I use label_mode 'categorical', so that the labels are described as a one-hot encoded vector.

I then prefetch the data:

train_ds = train_ds.prefetch(buffer_size=buffer_size)

Approach 1: specifying class weights

In this approach I try to specify the class weights of the classes via the class_weight argument of fit:

model.fit(
    train_ds, epochs=epochs, callbacks=callbacks, validation_data=val_ds,
    class_weight=class_weights
)

For each class we compute weight which are inversely proportional to the number of training samples for that class. This is done as follows (this is done before the train_ds.prefetch() call described above):

class_num_training_samples = {}
for f in train_ds.file_paths:
    class_name = f.split('/')[-2]
    if class_name in class_num_training_samples:
        class_num_training_samples[class_name] += 1
    else:
        class_num_training_samples[class_name] = 1
max_class_samples = max(class_num_training_samples.values())
class_weights = {}
for i in range(0, len(train_ds.class_names)):
    class_weights[i] = max_class_samples/class_num_training_samples[train_ds.class_names[i]]

What I am not sure about is whether this solution works, because the keras documentation does not specify the keys for the class_weights dictionary in case the labels are one-hot encoded. I tried training the network this way but found out that the weights did not have a real influence on the resulting network: when I looked at the distribution of predicted classes for each individual class then I could recognize the distribution of the overall training set, where for each class the prediction of the dominant classes is most likely. Running the same training without any class weight specified led to similar results. So I suspect that the weights don't seem to have an influence in my case.

Is this because specifying class weights does not work for one-hot encoded labels, or is this because I am probably doing something else wrong (in the code I did not show here)?

Approach 2: specifying sample weight

As an attempt to come up with a different (in my opinion less elegant) solution I wanted to specify the individual sample weights via the sample_weight argument of the fit method. However from the documentation I find:

[...] This argument is not supported when x is a dataset, generator, or keras.utils.Sequence instance, instead provide the sample_weights as the third element of x.

Which is indeed the case in my setup where train_ds is a dataset. Now I really having trouble finding documentation from which I can derive how I can modify train_ds, such that it has a third element with the weight. I thought using the map method of a dataset can be useful, but the solution I came up with is apparently not valid:

train_ds = train_ds.map(lambda img, label: (img, label, class_weights[np.argmax(label)]))

Does anyone have a solution that may work in combination with a dataset loaded by image_dataset_from_directory?

Answer 1

I found myself wondering the same thing today and here's the source code relevant to the question:

        if y.shape.rank >= 2:
            y_classes = tf.__internal__.smart_cond.smart_cond(
                backend.shape(y)[-1] > 1,
                lambda: backend.argmax(y, axis=-1),
                lambda: tf.cast(tf.round(tf.squeeze(y, axis=-1)), tf.int64),
            )

So it appears that Keras handles one-hot encoded targets by default.