train row encoder and column encoder in Tensorflow

Question

I am trying to create a custom neural network that has 2 encoders and one decoder. The row encoder takes in the input of size eg: 30x40 and the column encoder is supposed to take the same data in transpose, making it 40x30. How do I make it happen?

Tensorflow gives me an error when I concatenate the two encoder inputs. Can this be done somehow? Should I use a different library or a different approach? Can this be done at all?

I know having data 40x40 for the same model works perfectly but I want it for non-square matrix.

Model:

from tensorflow.keras.layers import Input, Dense, Concatenate
from tensorflow.keras.models import Model
import tensorflow as tf


r = Input(shape=(data.shape[0],))
c = Input(shape=(data.shape[1],))

re = Dense(data.shape[0])(r)
ce = Dense(data.shape[1])(c)

e = Concatenate()([re, ce])

d = Dense(2, activation='sigmoid')(e)
o = Dense(data.shape[1], activation='linear')(d)

one_model = Model(inputs=[r, c], outputs=o)
one_model.compile(optimizer='adam', loss='mse')

history = one_model.fit([data, np.transpose(data)], data, epochs=50, batch_size=4, verbose=2)

Error:

Shapes of all inputs must match: values[0].shape = [30,40] != values[1].shape = [40,30]

ValueError: Data cardinality is ambiguous: x sizes: 30, 40 y sizes: 30 Make sure all arrays contain the same number of samples.

Answer 1

Just adding my thoughts may be it will help someone in future.

I solved this problem using padding. so the input array would 30x40 and 40x40. My goal was to Train two models with input 1x30 and 1x40. I cannot pass 2 different arrays hoping 1 decoder will find the relation between 2 different arrays. for every combination of encoder1 input and encoder2 input there must be 1 target output which which will be compared with the decoder output to compare and generate loss. Loss is internally used by the system to learn. Hence the inputs can be:

Encoder1 input = 40, 1, 40 
Encoder2 input = 40, 1, 30 --> 10, 1, 30 padded with zeros
Decoder output = 80, 1, output_shape

Or even better approach is:

Encoder1 input = 30 * 40, 1, 40 --> 30 * 40 is the product of rows and columns
Encoder2 input = 30 * 40, 1, 30
Decoder output = 2 * 30 * 40, 1, output_shape --> output shape in my case would be 1. i.e., 1x1 which represents each cell in the 30x40 matrix