How do I use line_profiler (from Robert Kern)?

Question

I have tried using the line_profiler module for getting a line-by-line profile over a Python file. This is what I've done so far:

1) Installed line_profiler from pypi by using the .exe file (I am on WinXP and Win7). Just clicked through the installation wizard.

2) Written a small piece of code (similar to what has been asked in another answered question here).

from line_profiler import LineProfiler
def do_stuff(numbers):
    print numbers

numbers = 2
profile = LineProfiler(do_stuff(numbers))
profile.print_stats()

3) Run the code from IDLE/PyScripter. I got only the time.

Timer unit: 4.17188e-10 s

How do I get full line-by-line profile over the code I execute? I have never used any advanced Python features like decorators, so it is hard for me to understand how shall I use the guidelines provided by several posts like here and here.

Answer 1

Just follow Dan Riti's example from the first link, but use your code. All you have to do after installing the line_profiler module is add a @profile decorator right before each function you wish to profile line-by-line and make sure each one is called at least once somewhere else in the code—so for your trivial example code that would be something like this:

example.py file:

@profile
def do_stuff(numbers):
    print numbers

numbers = 2
do_stuff(numbers)

Having done that, run your script via the kernprof.py^✶ that was installed in your C:\Python27\Scripts directory. Here's the (not very interesting) actual output from doing this in a Windows 7 command-line session:

> python "C:\Python27\Scripts\kernprof.py" -l -v example.py
2
Wrote profile results to example.py.lprof
Timer unit: 3.2079e-07 s

File: example.py
Function: do_stuff at line 2
Total time: 0.00185256 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile
     2                                           def do_stuff(numbers):
     3         1         5775   5775.0    100.0      print numbers

You likely need to adapt this last step—the running of your test script with kernprof.py instead of directly by the Python interpreter—in order to do the equivalent from within IDLE or PyScripter.

^✶Update

It appears that in line_profiler v1.0, the kernprof utility is distributed as an executable, not a .py script file as it was when I wrote the above. This means the following now needs to used to invoke it from the command-line:

> "C:\Python27\Scripts\kernprof.exe" -l -v example.py

Answer 2

If you're using PyCharm, you can also take a look at https://plugins.jetbrains.com/plugin/16536-line-profiler

It's a plugin I created that allows you to load and visualize line profiler results into the PyCharm editor.

Answer 3

load the line_profiler and numpy

%load_ext line_profiler
import numpy as np

define a function for example:

def take_sqr(array):
    sqr_ar = [np.sqrt(x) for x in array]
    return sqr_ar

use line_profiler to count the time as follows:

%lprun -f take_sqr take_sqr([1,2,3])

the output looks like this:

Timer unit: 1e-06 s
 
Total time: 6e-05 s File: <ipython-input-5-e50c1b05a473> Function:
take_sqr at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def take_sqr(array):
     2         4         59.0     14.8     98.3      sqr_ar = [np.sqrt(x) for x in array]
     3         1          1.0      1.0      1.7      return sqr_ar

Answer 4

Just an addition to @Lhenkel answer. This is a decorator for async functions

def async_profile(func):
    """line profiler for an async funciton"""
    from functools import wraps

    @wraps(func)
    async def wrapper(*args, **kwargs):
        from line_profiler import LineProfiler
        prof = LineProfiler()
        try:
            return await prof(func)(*args, **kwargs)
        finally:
            prof.print_stats()

    return wrapper

To use these decorators with methods read this answer

Answer 5

Found a good use to line_profiler using decorator i.e. @profile that worked for me:

def profile(func):
    from functools import wraps

    @wraps(func)
    def wrapper(*args, **kwargs):
        from line_profiler import LineProfiler
        prof = LineProfiler()
        try:
            return prof(func)(*args, **kwargs)
        finally:
            prof.print_stats()

    return wrapper

Credits to: pavelpatrin

Answer 6

This answer is a copy of my answer here for how to get line_profiler statistics from within a Python script (without using kernprof from the command line or having to add @profile decorators to functions and class methods). All answers (that I've seen) to similar line_profiler questions only describe using kernprof.

---

The line_profiler test cases (found on GitHub) have an example of how to generate profile data from within a Python script. You have to wrap the function that you want to profile and then call the wrapper passing any desired function arguments.

from line_profiler import LineProfiler
import random

def do_stuff(numbers):
    s = sum(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

Output:

Timer unit: 1e-06 s

Total time: 0.000649 s
File: <ipython-input-2-2e060b054fea>
Function: do_stuff at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                           def do_stuff(numbers):
     5         1           10     10.0      1.5      s = sum(numbers)
     6         1          186    186.0     28.7      l = [numbers[i]/43 for i in range(len(numbers))]
     7         1          453    453.0     69.8      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

Adding Additional Functions to Profile

Also, you can add additional functions to be profiled as well. For example, if you had a second called function and you only wrap the calling function, you'll only see the profile results from the calling function.

from line_profiler import LineProfiler
import random

def do_other_stuff(numbers):
    s = sum(numbers)

def do_stuff(numbers):
    do_other_stuff(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

The above would only produce the following profile output for the calling function:

Timer unit: 1e-06 s

Total time: 0.000773 s
File: <ipython-input-3-ec0394d0a501>
Function: do_stuff at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def do_stuff(numbers):
     8         1           11     11.0      1.4      do_other_stuff(numbers)
     9         1          236    236.0     30.5      l = [numbers[i]/43 for i in range(len(numbers))]
    10         1          526    526.0     68.0      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

In this case, you can add the additional called function to profile like this:

from line_profiler import LineProfiler
import random

def do_other_stuff(numbers):
    s = sum(numbers)

def do_stuff(numbers):
    do_other_stuff(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(do_other_stuff)   # add additional function to profile
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

Output:

Timer unit: 1e-06 s

Total time: 9e-06 s
File: <ipython-input-4-dae73707787c>
Function: do_other_stuff at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                           def do_other_stuff(numbers):
     5         1            9      9.0    100.0      s = sum(numbers)

Total time: 0.000694 s
File: <ipython-input-4-dae73707787c>
Function: do_stuff at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def do_stuff(numbers):
     8         1           12     12.0      1.7      do_other_stuff(numbers)
     9         1          208    208.0     30.0      l = [numbers[i]/43 for i in range(len(numbers))]
    10         1          474    474.0     68.3      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

NOTE: Adding functions to profile in this way does not require changes to the profiled code (i.e., no need to add @profile decorators).