Does the number of hidden nodes in the fully connected layer has to be equal to the number of output categories?

Question

I tried the tutorial for image recognition in R using the MXNet package https://www.r-bloggers.com/image-recognition-tutorial-in-r-using-deep-convolutional-neural-networks-mxnet-package . The aim of the tutorial is to recognize faces of 40 persons. The dataframe consists of 400 pictures (10 pictures per person). The CNN looks like this:

data <- mx.symbol.Variable('data')
# 1st convolutional layer
conv_1 <- mx.symbol.Convolution(data = data, kernel = c(5, 5), num_filter = 20)
tanh_1 <- mx.symbol.Activation(data = conv_1, act_type = "tanh")
pool_1 <- mx.symbol.Pooling(data = tanh_1, pool_type = "max", kernel =    c(2, 2), stride = c(2, 2))

# 2nd convolutional layer
conv_2 <- mx.symbol.Convolution(data = pool_1, kernel = c(5, 5), num_filter = 50)
tanh_2 <- mx.symbol.Activation(data = conv_2, act_type = "tanh")
pool_2 <- mx.symbol.Pooling(data=tanh_2, pool_type = "max", kernel = c(2, 2), stride = c(2, 2))

# 1st fully connected layer
flatten <- mx.symbol.Flatten(data = pool_2)
fc_1 <- mx.symbol.FullyConnected(data = flatten, num_hidden = 500)
tanh_3 <- mx.symbol.Activation(data = fc_1, act_type = "tanh")

# 2nd fully connected layer
fc_2 <- mx.symbol.FullyConnected(data = tanh_3, num_hidden = 40)
# Output. Softmax output since we'd like to get some probabilities.
NN_model <- mx.symbol.SoftmaxOutput(data = fc_2)

I used the same neuronal network for my own dataset, which consists of 1600 pictures of 5 persons. So I adjusted the number of nodes in the fully connected layer to 5.

fc_2 <- mx.symbol.FullyConnected(data = tanh_3, num_hidden = 5)

The results of the model are very bad, so I set ceteris paribus the number of nodes in the fully connected layer to 80 and got great results (accurancy: 100%).

fc_2 <- mx.symbol.FullyConnected(data = tanh_3, num_hidden = 80)

The model generates probabilities for 80 categories although I only got 5, but the accurancy is excellent. I don’t understand this event. I tried to add a third fully connected layer to get the right number of catgories:

# 2nd fully connected layer
fc_2 <- mx.symbol.FullyConnected(data = tanh_3, num_hidden = 80)
tanh_4 <- mx.symbol.Activation(data = fc_2, act_type = "tanh")
# 3rd fully connected layer
fc_3 <- mx.symbol.FullyConnected(data = tanh_4, num_hidden = 5)
# Output. Softmax output since we'd like to get some probabilities.
NN_model <- mx.symbol.SoftmaxOutput(data = fc_3)

But the results are very bad. I thought the number of nodes in the fully connected layer represents the number of output categories which the model should try to distinguish.

1. Is anyone possible to explain this event?
2. Does the number of hidden nodes in the fully connected layer has to be equal to the number of output categories?

Thanks for your help.

Answer 1

My model does work now:

# 1st fully connected layer
flatten <- mx.symbol.Flatten(data = pool_2)
fc_1 <- mx.symbol.FullyConnected(data = flatten, num_hidden = 500)
tanh_3 <- mx.symbol.Activation(data = fc_1, act_type = "tanh")

# 2nd fully connected layer
fc_2 <- mx.symbol.FullyConnected(data = tanh_3, num_hidden = 5)
# Output. Softmax output since we'd like to get some probabilities.
NN_model <- mx.symbol.SoftmaxOutput(data = fc_2)

I changed the label of the persons from:

Person1: 1 -> 0; Person2: 2 -> 1; Person2: 3 -> 2; Person2: 4 -> 3; Person5: 5 -> 4;

I dont know why it works, but it does. Thanks for help Hagay Lupesko.

Answer 2

You have significantly more parameters in your model than samples. This is typically very bad, and can cause over fitting.

Another approach you can take is taking a pre-trained model, and re-train the last layer only with your data (AKA transfer learning). Here's an MXNet tutorial for that: https://mxnet.incubator.apache.org/how_to/finetune.html