I am training a model using tensorflow, somehow only the first two GPU were used heavily, occupying 10000 MiB and 9290 MiB, respectively.
I have eight GPUs, the six other GPUs were used but only 300 MB each. Does anyone know why the GPU usage is not evenly distributed across all GPUs?
I use the following code:
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = create_unet_model()
model.compile(...)
model.fit(...)
I looked it up, but no one could suggest any possible reason.