IPython %timeit what is loop and iteration in the options?

Question

I am wondering about the %timeit command in IPython

From the docs:

%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code

Options:

-n: execute the given statement times in a loop. If this value is not given, a fitting value is chosen.

-r: repeat the loop iteration times and take the best result. Default: 3

For example, if I write:

%timeit -n 250 -r 2 [i+1 for i in range(5000)]

So, -n 250 executes [i+1 for i in range(5000)] 250 times? Then what does -r 2?

Answer 1

It specifies the number of repeats, the number of repeats are used to determine the average. For example:

%timeit -n 250 a = 2
# 61.9 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 250 loops each)

%timeit -n 250 -r 2 a = 2
# 62.6 ns ± 0 ns per loop (mean ± std. dev. of 2 runs, 250 loops each)

The number of executions will be n * r but the statistic is based on the number of repeats (r) but the number of "loops" for each repeat is determined based on the number (n).

Basically you need a large enough n so the minimum of the number of loops is accurate "enough" to represent the fastest possible execution time, but you also need a large enough r to get accurate "statistics" on how trustworthy that "fastest possible execution time" measurement is (especially if you suspect that some caching could be happening).

For superficial timings you should always use an r of 3, 5 or 7 (in most cases that's large enough) and choose n as high as possible - but not too high, you probably want it to finish in a reasonable time :-)

Answer 2

timeit -n 250 <statement>

The statement will get executed 3 * 250 = 750 times (-r has a default value of 3)

timeit -n 250 -r 4 <statement>

The statement will get executed 4 * 250 = 1000 times

-r - how many times to repeat the timer (in the examples above, each time the timer is called with -n 250 which means 250 executions)

Answer 3

A more statistical way of explaining is as the bootstrapping estimation of the distribution of some statistics (specifically, its mean and standard deviation), in such context: "r" can be seen as the number of samples and "n" as the size of each sample.