CUDA Kernel Execution Time does not change on larger array

Question

I am profiling a cuda application on different 1d input sizes. However NSIGHT profiler kernel execution times are similar for small vector sizes. As an example, there is not any difference between vector sizes 512 and 2048.Kernel execution time increases linearly for bigger vectors but no difference on smaller vectors like vector size 512 and 2048. Is it an expected result?

Answer 1

Let's suppose it takes 3 microseconds of execution time to launch a kernel of any size, and, after that overhead, 1 ns of execution time per point in your vector. Now let's ask, what is the percent difference in the execution of kernels of x and 2x points, when x is small (say 1024) and when x is large (say, 1048576)?

x = 1024:

execution_time(x) =  3000+1024 = 4024ns
execution_time(2x) = 3000+2048 = 5048ns
%difference = (5048-4024)/4024 * 100% = 25.45%

x = 1048576:

execution_time(x) =  3000+1048576 = 1051576ns
execution_time(2x) = 3000+2097152 = 2100152ns
%difference = (2100152-1051576)/1051576 * 100% = 99.71%

This demonstrates what to expect when making measurements of execution time (and changes in execution time) when the execution time is small compared to the fixed overhead, vs. when it is large compared to the fixed overhead.

In the small case, the execution time is "swamped" by the overhead. Doubling the "work" does not lead to a doubling in the execution time. In the large case, the overhead is insignificant compared to the execution time. Therefore in the large case we see approximately the expected result, that doubling the "work" (vector length) approximately doubles the execution time.

Note that the "fixed overhead" here may be composed of a number of items, the "kernel launch overhead" being just one. CUDA typically has other start-up fixed "overheads" associated with initialization, that also play a role.