I am using NVENC to encode H.265 video frames. I am feeding it frames sequentially - and what I care about is ultra low latency encode (<6ms if possible), at a specific frame rate. I am generating the frames on the fly so I can't feed it any faster.
If the frame rate is high, then everything behaves as expected and the latency (total encode time) for the frames is very low. I.e. if I submit frames at 120FPS, my encode time is roughly 6ms. However, if I submit frames much more slowly, for instance 45 FPS, then the latency on my encode balloons to 15+ milliseconds.
What would be going on that would cause the latency of frames on NVENC to be reduced?
Why would submitting frames in rapid success cause reduction in latency?
I have often been suspicious that this could have to do with some sort of clock throttling on NVIDIA GPUs, as I had seen different behavior when using MSI Afterburner.
These systems have DCVS algorithms that modifies voltages based on current load dynamically. Higher rate input generation may trigger higher bandwidth voting. The same vote may result in encoder running at higher clock, too.
Besides, your suspicion may actually be the reason. Try multiple input frame-rate and you may be able to find the threshold where clock rate is boosted. Higher input (queue) frame rate, generally, corresponds to non-real time video encoding and clock boost for such use-cases isn't uncommon.
Of course, I'm assuming both experiments are performed under similar conditions for similar resolutions.