I have a model which i have tried training on the gpu on my laptop and cpu on a seperate computer. On both, it runs fine and returns values for the loss. When i run it on a gpu on the computer (where the cpu was), nan values are returned for the loss.
Having checked through, I know there are no nans in my training dataset, I've tried making the learning rate (its an Adam optimizer) really small and large, and changing the batch size to be 1 and also be 64. In all cases, the same problem occurs.