How to understand `yield from` in python coroutine?

Question

The code come form Fluent Python 1st edtion,

I cannot understand the line while True: in grouper, delete that line raise a StopIteration error.

But I find a new version of grouper without while True: that works. Why group.send(None) need another loop in while True: (or another results[key] = yield from averager())?

My understanding is group.send(None) will stop yield from averager() and assign results[key] a value(Result(count, average)). That's all.

from collections import namedtuple

Result = namedtuple('Result', 'count average')


# the subgenerator
def averager():  # <1>
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield  # <2>
        if term is None:  # <3>
            break
        total += term
        count += 1
        average = total/count
    return Result(count, average)  # <4>


# the delegating generator
def grouper(results, key):  # <5>
    while True:  # <6>
        results[key] = yield from averager()  # <7>

# Another version works
#def grouper(results, key):
#    results[key] = yield from averager()
#    results[key] = yield from averager()

# the client code, a.k.a. the caller
def main(data):  # <8>
    results = {}
    for key, values in data.items():
        group = grouper(results, key)  # <9>
        next(group)  # <10>
        for value in values:
            group.send(value)  # <11>
        group.send(None)  # important! <12>

    # print(results)  # uncomment to debug
    report(results)


# output report
def report(results):
    for key, result in sorted(results.items()):
        group, unit = key.split(';')
        print('{:2} {:5} averaging {:.2f}{}'.format(
              result.count, group, result.average, unit))


data = {
    'girls;kg':
        [40.9, 38.5, 44.3, 42.2, 45.2, 41.7, 44.5, 38.0, 40.6, 44.5],
    'girls;m':
        [1.6, 1.51, 1.4, 1.3, 1.41, 1.39, 1.33, 1.46, 1.45, 1.43],
    'boys;kg':
        [39.0, 40.8, 43.2, 40.8, 43.1, 38.6, 41.4, 40.6, 36.3],
    'boys;m':
        [1.38, 1.5, 1.32, 1.25, 1.37, 1.48, 1.25, 1.49, 1.46],
}


if __name__ == '__main__':
    main(data)

Answer 1

This makes me remember how nice ascynio is, and why everybody should use it...

What is happening is best explained by walking through the operation of the iterators. This is the inner generator, simplified:

def averager():
    local_var
    while True:
        term = yield
        if term is None:
            break
        local_var = do_stuff(term)
    return local_var

This does two things. Firstly, it gets some data with yield (ugh, explaining that choice of words is just confusing) so long as that data isn't None. Then when it is None, it raises a StopIterationException with the value of local_var. (This is what returning from a generator does).

Here is the outer generator:

def grouper(results, key):
    while True:
        results[key] = yield from averager()

What this does is to expose the inner generator's yield up to the calling code, until the inner generator raises StopIterationException, which is silently captured (by the yield from statement) and assigned. Then it gets ready to do the same thing again.

Then we have the calling code:

def main(data):
    results = {}
    for key, values in data.items():
        group = grouper(results, key)
        next(group)
        for value in values:
            group.send(value)
        group.send(None)

What this does is:

• it iterates the outer generator exactly once
• this exposes the inner generator's yield, and it uses that (.send) to communicate with the inner generator.
• it 'ends' the inner generator by sending None, at which point the first yield from statement ends, and assigns the value passed up.
• at this point, the outer generator gets ready to send another value
• the loop moves on, and the generator is deleted by garbage collection.

what's with the while True: loop?

Consider this code, which also works for the outer generator:

def grouper(result, key):
    result[key] = yield from averager
    yield 7

The only important thing is that the generator should not be exhausted, so it doesn't pass an exception up the chain saying 'I have nothing left to iterate'.

P.S. confused? I was. I had to check this out, it's a while since I've tried to use generator based coros. They're scheduled for deletion---use asyncio, it's much nicer.