Tensorflow C API Selecting GPU

Question

I am using the Tensorflow C API to run models saved/frozen in python. We use to run these models on CPU but recently switched to GPU for performance. To interact with the C API we use a wrapper library called CPPFlow (https://github.com/serizba/cppflow). I recently updated this library so that we can pass in GPU Config options so that we can control GPU memory allocations. However we also now have systems with multiple GPUs which is causing some issues. It seems like I cant get Tensorflow to use the same GPU as our software does.

I use the visible_device_list parameter with the same GPU ID as our software. If I set our software to run on device 1 and Tensorflow to device 1, Tensorflow will pick device 2. If I set our software to use device 1 and Tensorflow to use device 2, both software use the same GPU.

How does Tensorflow order GPU devices and do I need to use another method to manually select the device? Every where I look suggests it can be done using the GPU Config options.

Answer 1

You can set Tensorflow GPU order by setting the environment variable CUDA_VISIBLE_DEVICES during execution. For more details, you can check it here

//Set TF to use GPU:1 and GPU:0 (in this order)     
setenv( "CUDA_VISIBLE_DEVICES", "1,0", 1 );

//Set TF to use only GPU:0 (in this order)     
setenv( "CUDA_VISIBLE_DEVICES", "0", 1 );

//Set TF to do not use GPUs     
setenv( "CUDA_VISIBLE_DEVICES", "-1", 1 );

Answer 2

One way to set the device is getting the hex string in python and then using the string in C API: For example, Sample 1:

gpu_options = tf.GPUOptions(allow_growth=True,visible_device_list='1')
config = tf.ConfigProto(gpu_options=gpu_options)
serialized = config.SerializeToString()
print(list(map(hex, serialized)))

Sample 2:

import tensorflow as tf
config = tf.compat.v1.ConfigProto(device_count={"CPU":1}, inter_op_parallelism_threads=1,intra_op_parallelism_threads=1)
ser = config.SerializeToString()
list(map(hex,ser))
Out[]: 
['0xa',
'0x7',
'0xa',
'0x3',
'0x43',
'0x50',
'0x55',
'0x10',
'0x1',
'0x10',
'0x1',
'0x28',
'0x1']

Use this string in C API as

uint8_t config[13] = {0xa, 0x7, 0xa, ... , 0x28, 0x1};
TF_SetConfig(opts, (void*)config, 13, status);

For more details:

https://github.com/tensorflow/tensorflow/issues/29217
https://github.com/cyberfire/tensorflow-mtcnn/issues/1
https://github.com/tensorflow/tensorflow/issues/27114