I've been trying to implement a custom loss function for a TF Estimator, but TensorFlow is returning NaN losses. This occurs even when setting the learning rate to low numbers (1e-10) or 0. When forcefully setting the gradient to 0, the neural network works, implying an issue with the gradients. I have already checked the dataset for NaN values, outliers, etc. In addition, I have attempted to remove any functions that could potentially interfere with the automatic differentiation, to no avail (e.g. map_fn). What else could be causing these problems?
- The loss function does not weight all of the neural network's predictions equally; some predictions are taken into account multiple times when generating the loss
- The loss function also requires features not run through the neural network
Error message:
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.