Can time.clock() be heavily affected by the state of the system?

Question

This is a rather general question:

I am having issues that the same operation measured by time.clock() takes longer now than it used to.

While I had very some very similar measurements

• 1954 s
• 1948 s
• 1948 s

One somewhat different measurement

• 1999 s

Another even more different

• 2207 s

It still seemed more or less ok, but for another one I get

• 2782 s

And now that I am repeating the measurements, it seems to get slower and slower.

I am not summing over the measurements after rounding or doing other weird manipulations.

Do you have some ideas whether this could be affected by how busy the server is, the clock speed or any other variable parameters? I was hoping that using time.clock() instead of time.time() would mostly sort these out...

The OS is Ubuntu 18.04.1 LTS.

The operations are run in separate screen sessions.

The operations do not involve hard-disk acccess.

The operations are mostly numpy operations that are not distributed. So this is actually mainly C code being executed.

EDIT: This might be relevant: The measurements in time.time() and time.clock() are very similar in any of the cases. That is time.time() measurements are always just slightly longer than time.clock(). So if I haven't missed something, the cause has almost exactly the same effect on time.clock() as on time.time().

EDIT: I do not think that my question has been answered. Another reason I could think of is that garbage collection contributes to CPU usage and is done more frequently when the RAM is full or going to be full.

Mainly, I am looking for an alternative measure that gives the same number for the same operations done. Operations meaning my algorithm executed with the same start state. Is there a simple way to count FLOPS or similar?

Answer 1

The issue seems to be related to Python and Ubuntu.

Try the following:

• Check if you have the latest stable build of the python version you're using Link 1

• Check the process list, also see which cpu core your python executable is running on.

• Check the thread priority state on the cpu Link 2 , Link 3

Note:

• Time may vary due to process switching, threading and other OS resource management and application code execution (this cannot be controlled)

Suggestions:

• it could be because of your systems build, try running your code on another machine or on a Virtual Machine.

Read Up:

• Link 4

• Link 5

Good Luck.

~ Dean Van Geunen

Answer 2

As a result of repeatedly running the same algorithm at different 'system states', I would summarize that the answer to the question is:

Yes, time.clock() can be heavily affected by the state of the system.

Of course, this holds all the more for time.time().

The general reasons could be that

1. The same Python code does not always result in the same commands being sent to the CPU - that is the commands depend not only on the code and the start state, but also on the system state (i.e. garbage collection)
2. The system might interfere with the commands sent from Python, resulting in additional CPU usage (i.e. by core switching) that is still counted by time.clock()

The divergence can be very large, in my case around 50%.

It is not clear which are the specific reasons, nor how much each of them contributes to the problem.

It is still to be tested whether timeit helps with some or all of the above points. However timeit is meant for benchmarking and might not be recommended to be used during normal processing. It turns off garbage collection and does not allow accessing return values of the timed function.