GeneratorExit in Python generator

Question

I wrote a test program about Python generator. But I got an error that is not expected. And I don't know how to explain it. Let me show you the code:

def countdown(n):
    logging.debug("Counting down")
    while n > 0:
        try:
            yield n
        except GeneratorExit:
            logging.error("GeneratorExit")
        n -= 1


if __name__ == '__main__':
    c = countdown(10)
    logging.debug("value: %d", c.next())

I thought it should run without any problem. But the output is:

# ./test.py
[2015/06/16 04:10:49] DEBUG    - Counting down     
[2015/06/16 04:10:49] DEBUG    - value: 10 
[2015/06/16 04:10:49] ERROR    - GeneratorExit
Exception RuntimeError: 'generator ignored GeneratorExit' in <generator object countdown at 0x7f9934407640> ignored

Why is there an error at the last line. I don't know why I triggered the GeneratorExit exception. Is there something aobut generator I missed? I also typed the code into interactive python shell, and everything is OK. How can this happen?

Answer 1

When the generator object is garbage-collected at the end of your program, its close() method is called, and this raises the GeneratorExit exception inside the generator. Normally this is not caught and causes the generator to exit.

Since you catch the exception and proceed to yield another value, this causes a RuntimeError. If you catch the GeneratorExit exception you need to either reraise it, or exit the function without yielding anything else.

Answer 2

Suppose you have the following generator:

def gen():
    with open('important_file') as f:
        for line in f:
            yield line

and you next it once and throw it away:

g = gen()
next(g)
del g

The generator's control flow never leaves the with block, so the file doesn't get closed. To prevent this, when a generator is garbage-collected, Python calls its close method, which raises a GeneratorExit exception at the point from which the generator last yielded. This exception is intended to trigger any finally blocks or context manager __exit__s that didn't get a chance to run.

When you catch the GeneratorExit and keep going, Python sees that the generator didn't exit properly. Since that can indicate that resources weren't properly released, Python reports this as a RuntimeError.

Answer 3

Maybe It's because your's yield value n is inside try blocks which always returns new n reference which make the last n value is garbage collected automatically. It's also state in PEP 342 :

"Add support to ensure that close() is called when a generator iterator is garbage-collected"

"Allow yield to be used in try/finally blocks, since garbage collection or an explicit close() call would now allow the finally clause to execute."

Since close method in generator is equivalent to throw a GeneratorExit and catched by yours exception, then logging.error('GeneratorExit') expression is executed.

"RunTimeError" is raised because the generator is yield next value n(9), it's state in python documentation https://docs.python.org/3.6/reference/expressions.html#generator-iterator-methods :

"Raises a GeneratorExit at the point where the generator function was paused. If the generator function then exits gracefully, is already closed, or raises GeneratorExit (by not catching the exception), close returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit"

May be the code should like this :

#pygen.py

import sys
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s \
        %(levelname)s - %(message)s', datefmt='[%Y/%m/%d %H:%M:%S]')

def genwrapper(func):
    #makes gen wrapper
    #which automatically send(None) to generator
    def wrapper(n=None):
        f = func(n)
        f.send(None)
        return f
    return wrapper

@genwrapper
def countdown(n=None):
    logging.debug('Counting Down')
    while True:
        try:
            n = yield(n)
        except GeneratorExit as e:
            logging.error('GeneratorExit')
            raise e

if __name__ == '__main__':
    n = int(sys.argv[1])
    c = countdown() #avoid function call in loop block (avoid new reference to c)
     while n > 0:
         a = c.send(n)
         logging.debug('Value: %d', a)
         n -= 1

then in yours terminal :

guest@xxxxpc:~$ python pygen.py 5

will result :

[2018/12/13 16:50:45]         DEBUG - Counting Down
[2018/12/13 16:50:45]         DEBUG - Value: 5
[2018/12/13 16:50:45]         DEBUG - Value: 4
[2018/12/13 16:50:45]         DEBUG - Value: 3
[2018/12/13 16:50:45]         DEBUG - Value: 2
[2018/12/13 16:50:45]         DEBUG - Value: 1
[2018/12/13 16:50:45]         ERROR - GeneratorExit

Sorry for my bad english or my suggestion if not clear enough, thanks