cuda-gdb sees only one least capable device from four CUDA-capable devices available

Question

There are four CUDA-capable devices available:

teslabot$ ./deviceQuery | grep -i "device [0-9]\|capability"
Device 0: "Tesla C2050 / C2070"
  CUDA Capability Major/Minor version number:    2.0
Device 1: "Tesla C2050 / C2070"
  CUDA Capability Major/Minor version number:    2.0
Device 2: "GeForce GTX 295"
  CUDA Capability Major/Minor version number:    1.3
Device 3: "GeForce GTX 295"
  CUDA Capability Major/Minor version number:    1.3

cuda-dbg sees only one of them:

teslabot$ cuda-gdb vector_add
NVIDIA (R) CUDA Debugger
4.0 release
Portions Copyright (C) 2007-2011 NVIDIA Corporation
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
[...]
(cuda-gdb) break vector_add_gpu
Breakpoint 1 at 0x400ddb: file vector_add.cu, line 7.
(cuda-gdb) run
[...]
(cuda-gdb) info cuda devices
  Dev Description SM Type SMs Warps/SM Lanes/Warp Max Regs/Lane Active SMs Mask
*   0       gt200   sm_13  30       32         32           128 0x00000001

I have checked that code build with -gencode arch=compute_20,code=sm_20 compiles without errors on said machine, and when compiled for sm_20 then using printf in CUDA kernel works correctly.

How can I make cuda-gdb see all devices (perhaps except one used for graphics... though in said case I am logging remotely via SSH), or at least one Tesla / sm_20 device?

---

When following advise in Michael Foukarakis response by setting CUDA_VISIBLE_DEVICES environment variable to contain only "0,1" i.e. make visible only Teslas, I get the following error after running info cuda devices:

(cuda-gdb) info cuda devices
fatal:  All CUDA devices are used for X11 and cannot be used while debugging. (error code = 24)

How to check which devices are used by X11 (X.Org), and how to make X Window System to use GeForce and not Tesla?

Answer 1

Can you make sure the CUDA_VISIBLE_DEVICES environment variable contains all the devices you want to be used, such as:

$ ./deviceQuery -noprompt | egrep "^Device"
Device 0: "Tesla C2050"
Device 1: "Tesla C1060"
Device 2: "Quadro FX 3800"

By setting the variable you can make only a subset of them visible to the runtime:

$ export CUDA_VISIBLE_DEVICES="0,2"
$ ./deviceQuery -noprompt | egrep "^Device"
Device 0: "Tesla C2050"
Device 1: "Quadro FX 3800"