How to create a benchmark with the same and unchanged input?

Question

I want the input to be fixed and the number of instructions to not change when I run a benchmark. How to create a benchmark with the same and unchanged input? My benchmark is oss-performance (oss-performance github). I want the following two to have the same output in terms of instruction count.

$ perf stat -e L1-icache-load-misses,instruc
tions hhvm perf.php --wordpress --hhvm=/usr/local/bin/hhvm  -c General -r 0

      110647492437      L1-icache-load-misses                                       
     2676929150620      instructions                                                

     316.817844079 seconds time elapsed

perf stat -e L1-icache-load-misses,instruc
tions hhvm perf.php --wordpress --hhvm=/usr/local/bin/hhvm  -c General -r 0

    110728668966      L1-icache-load-misses                                       
     2677096040536      instructions                                                

     303.617243520 seconds time elapsed

While now, as you can see, the output is not the same? Please help me, I need these outputs to be the same every time so that I can get a trace.

But I don't know what to do.

Please help me.

Thanks

Answer 1

Perhaps the program has some timing-dependent paths of execution, for example retries in clock_gettime if the kernel timestamp was in mid-update (if it uses a SeqLock). I don't think clock_gettime retries could account for everything, but if there's any multithreading within hhvm, it probably uses some locks. And those locks probably have spin-retry loops if they find the lock was not available, before falling back to a futex system call.

Perhaps it could help to taskset -c 1 perf stat ... or something to pin everything to one core, so only one thread could ever execute at once. But that doesn't stop an unlucky context switch from putting a thread to sleep while it holds a lock, leading to others failing to get a lock.

This might make it more granular, like if everything happens to schedule perfectly, you get the minimum, and if not you get some fixed number of extra instructions per time a thread went to sleep while holding a lock.