Error: Incompatible Dimension Shapes during Training of Multi-Output Sequence Model (Keras/Python)

Question

The problem involves incompatible dimensions during the training of a multi-output sequence model in Keras/Python. Specifically, the issue arises due to shape mismatch between the expected and predicted output dimensions. This mismatch leads to a ValueError during the training process.

Here is the error: ValueError: Shapes (None, 10) and (None, 10, 47) are incompatible

X_train, X_test, y_l_train, y_l_test = train_test_split(X, y_l, test_size=0.2, random_state=42)
_, _, y_em_train, y_em_test = train_test_split(X, y_em, test_size=0.2, random_state=42)
_, _, y_f_train, y_f_test = train_test_split(X, y_f, test_size=0.2, random_state=42)

token_encoder = LabelEncoder()
token_encoder.fit(df['P_tokens'].explode())

label_encoder_l = LabelEncoder()
label_encoder_em = LabelEncoder()
label_encoder_f = LabelEncoder()

label_encoder_l.fit(df['l_tokens'].explode())
label_encoder_em.fit(df['em_tokens'].explode())
label_encoder_f.fit(df['f_tokens'].explode())

def encode_sequence(seq, encoder):
    return [encoder.transform([token])[0] for token in seq]

X_train_encoded = [encode_sequence(seq, token_encoder) for seq in X_train]
X_test_encoded = [encode_sequence(seq, token_encoder) for seq in X_test]

max_sequence_length = 10

X_train_padded = pad_sequences(X_train_encoded, maxlen=max_sequence_length, padding='post')
X_test_padded = pad_sequences(X_test_encoded, maxlen=max_sequence_length, padding='post')

def encode_and_pad_labels(labels, encoder):
    encoded_labels = [encode_sequence(seq, encoder) for seq in labels]
    padded_labels = pad_sequences(encoded_labels, maxlen=max_sequence_length, padding='post')
    return padded_labels

y_l_train_padded = encode_and_pad_labels(y_l_train, label_encoder_l)
y_em_train_padded = encode_and_pad_labels(y_em_train, label_encoder_em)
y_f_train_padded = encode_and_pad_labels(y_f_train, label_encoder_f)

y_l_test_padded = encode_and_pad_labels(y_l_test, label_encoder_l)
y_em_test_padded = encode_and_pad_labels(y_em_test, label_encoder_em)
y_f_test_padded = encode_and_pad_labels(y_f_test, label_encoder_f)

model = Model(inputs=input_layer, outputs=[output_l, output_em, output_f])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(X_train_padded, [y_l_train_padded, y_em_train_padded, y_f_train_padded],
          validation_data=(X_test_padded, [y_l_test_padded, y_em_test_padded, y_f_test_padded]),
          epochs=10, batch_size=32)

I attempted to adjust the dimensions of the training data to match those expected by the model. I expected the training to proceed without errors, but unfortunately, there's an ongoing mismatch between the shapes of the expected and predicted data, resulting in an error.

# to 2D
y_l_pred_2d = predictions[0].reshape(-1, len(label_encoder_l.classes_))
y_em_pred_2d = predictions[1].reshape(-1, len(label_encoder_em.classes_))
y_f_pred_2d = predictions[2].reshape(-1, len(label_encoder_f.classes_))

y_l_test_reshaped = y_l_test_padded.reshape(-1)
y_em_test_reshaped = y_em_test_padded.reshape(-1)
y_f_test_reshaped = y_f_test_padded.reshape(-1)

model.fit(X_train_padded, [y_l_train_padded.reshape(-1, 10), 
                           y_em_train_padded.reshape(-1, 10), 
                           y_f_train_padded.reshape(-1, 10)],
          validation_data=(X_test_padded, [y_l_test_padded.reshape(-1, 10), 
                                           y_em_test_padded.reshape(-1, 10), 
                                           y_f_test_padded.reshape(-1, 10)]),
          epochs=10, batch_size=32)

And same error