GAN - Generator loss decreasing but Discriminator fake loss increase after a initial drop, why?

Question

I'm learning GAN and was trying to run the pix2pix GAN model on a custom dataset, my average generator loss per epoch and average Discriminator Fake and Real loss are as follows -

and

I just can't understand, how come my Generator loss decrease but discriminator fake image loss increase? From what I understood, it was supposed to go down like the generator. Can someone please help me understand the error I made or the training problem I'm facing?

Batch Size: 16
Epoch: 100
Learning Rate: 0.0008
L1 Lambda: 100
Optimizer : Gen - Adam ; Disc - SGD
BatchNORM used in Generator .

Answer 1

The losses in a GAN are hard to interpret and aren't what you think. The technical answer may be that the generator is overpowering the discriminator. The images are getting more realistic (which is all you really care about) so the generator's loss is improving, while the discriminator is doing the same quality job, but getting tougher data.

Measure your GAN using FID or IS, not loss.