How to debug skflow code (tensorflow) gmm_ops.py?

95 Views Asked by At

Hi I am new to tensorflow. I want to debug Tensorflow (skflow) gmm_ops.py (Gaussian Mixture Model). I am getting ERROR:tensorflow:Model diverged with loss = NaN. How should I do it ? Is there any example?

    raise NanLossDuringTrainingError
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.
1

There are 1 best solutions below

2
Alexandre Passos On

Usually a NanLoss means something overflowed or underflowed during training. Things such as normalizing the examples or processing a subset of the data tend to help debugging what could have caused this.