Keras Single Input Multiple Outputs - why losses are so high comparing with Single Output?

Question

I think something wrong with my Keras multiple outputs coding, which causes a high loss comparing with the Sequential model. Please help me out which part of it is wrong.

import os, random, string, pandas, math, numpy
import tensorflow as tf
from tensorflow import keras

Training data:

feature_data = [] # common feature data
label_data = [] # for multiple outputs
single_data = [] # for single output
size = 10000
features = ['x1', 'x2']
labels = ['y1', 'y2']
for i in range(size):
    a = random.random()
    b = random.random()
    c = math.sin(a)
    d = math.cos(b)
    feature = [a, b]
    label = [c, d]
    feature_data.append(feature)
    label_data.append(label)
    single_data.append(c)

This is my single output model, which is working well: loss < 2e-05

single = keras.Sequential([
    keras.layers.Dense(2, input_shape=(2,), activation=tf.nn.softmax),
    keras.layers.Dense(4, activation=tf.nn.softmax),
    keras.layers.Dense(1)])
optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
single.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
single.fit(x=feature_data, y=single_data, epochs=100, batch_size=100)

This should be the identical multiple output model, but the losses are really high: 0.1

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    for label in labels:
        u = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

model = build_model()
model.fit(x=feature_data, y=label_data, epochs=100, batch_size=100)

I guess that something wrong with the input layer, or the label data format, but still have no idea how to fix it. Please help.

Answer 1

I was not paying attention.

Softmax is for picking one out of several possible outputs. You are doing continuous level outputs, not one-hot encoding.

I think your second model should work if you pick the right activation function at the right Dense layers.

Looking at your data, you are predicting sin of (0->1) and cos of (0->1). From the chart, these are both very flat lines within the range of sin & cos. This is such a small model that maybe tanh or sigmoid on any or all Dense layer should do the job.

Answer 2

[Ignore this] The second model is not the same as the first model: it uses a different Dense layer at the beginning.

This structure matches the first model:

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    d = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)
    for label in labels:
        u = d(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

The structure matches the first model because it uses the same dense layer (the exact same weights) to interpret the input in both chains.

Try plotting the loss history from your second network. You might see dual oscillating values: one gets good, the other gets bad, and they trade places.