Why cleverhans pytorch tutorial using log_softmax instead of logits as output

Question

When generating adversarial examples, it is typically using logits as the output of the neural network, and then train the network with cross-entropy.

However, I found that the tutorial of cleverhans uses log softmax and then convert the pytorch model to a tensorflow model, and finally train the model.

https://github.com/tensorflow/cleverhans/blob/master/cleverhans_tutorials/mnist_tutorial_pytorch.py#L65

I am wondering if anyone has the idea about whether using logits instead of log_softmax will make any difference?

Answer 1

As you said, when we get logits from a neural network, we train it using CrossEntropyLoss. An alternative way is to compute the log_softmax and then train the network by minimizing the negative log-likelihood (NLLLoss).

Both approaches are basically the same if you are training a network for classification tasks. However, if you have a different objective function, you may find one of these two techniques, particularly useful in your scenario.

Reference

• CrossEntropyLoss
• NLLLoss