With the following code (uses tensorflow.contrib.nccl.all_sum
), I expected to see bytes being transferred over NVLINK.
In reality, I don't.
from tensorflow.contrib.nccl import all_sum
with tf.device('/gpu:0'):
a = tf.get_variable(
"a", initializer=tf.constant(1.0, shape=(args.dim, args.dim)))
with tf.device('/gpu:1'):
b = tf.get_variable(
"b", initializer=tf.constant(2.0, shape=(args.dim, args.dim)))
with tf.device('/gpu:0'):
summed_node = all_sum([a, b])
sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True,
log_device_placement=True))
init = tf.global_variables_initializer()
sess.run(init)
with tf.device('/gpu:0'):
summed = sess.run(summed_node)
My machine is an AWS instance of p3.8xlarge
. My understanding is, this configuration supports NVLINK.
The execution is fine but when I use nvidia-smi nvlink -g 0 -i 0
the link Tx/Rx counts are zero.