Is there any way to discover at a given time how many process' are running on the GPU and possibly manage them (yield, resume, kill ... when necessary). What i want to do is while I run different programs, monitor each process activities on the GPU. I didn't see how that can be done. I saw that GPU context is the CPU equivalent of process however all the explanations I read on that was about influencing it inside a single program. Again, what I want is to trace all activity on the GPU (like with the nvidia-smi tool but with my own means). Can anyone help me with that? Is that even possible?
Get current CUDA contexts running on my GPU
727 Views Asked by Kanté At
1
A GPU Context is a driver level construct, the GPU itself doesn't know anything about contexts. All it knows is that it has a command queue, sent to it from the driver, that it will run through and execute. The various API's the GPU's support expose support for issuing commands, I don't think there are any commands for cancelling commands although there are commands for inspecting whether a command has been completed. Inspecting or manipulating commands in other contexts from my perspective would be a betrayal of the concept of a context.
RE inspecting what the GPU is doing, you could do this at the driver level by tracking the driver command queue, you are issuing and the hardware command queue sampling state, it is a lot of work which is why this task is usually left to GPU vendors to implement.