Does * Unpacking Use Memory on Generator Expression?

Question

Example for context:

Does calling * to unpack input put everything into memory? I'm hoping not but just want to confirm my understanding.

input = (x for x in ((1, 'abc'), (2, 'def'))) # generator expression
unzipped = zip(*input) # Does *input get completely unpacked or stay memory efficient?
first_items = next(unzipped)
print(first_items)
# >> (1, 2)

Answer 1

Unpacking eagerly unpacks the top level of the iterable in question, so in your case, yes, it will run the generator expression to completion before zip is actually invoked, then perform the equivalent of zip((1, 'abc'), (2, 'def')). If the iterables inside the generator were themselves lazy iterators though, zip won't preread them at all, which is usually the more important savings. For example, if input is defined with:

input = (open(name) for name in ('file1', 'file2'))

then while:

unzipped = zip(*input)

does eagerly open both files (so you may as well have used a listcomp; the genexpr didn't really save anything), it doesn't read a single line from either of them. When you then do:

first_items = next(unzipped)

it will read exactly one line from each, but it doesn't read the rest of the file until you ask for more items (technically, under the hood, file objects do block reads, so it will read more than just the line it returns, but that's an implementation detail; it won't slurp the whole of a 10 GB file just to give you the first line).

This is the nature of * unpacking; the receiving function needs to populate its arguments at the moment it is called. If you define:

def foo(a, b):
    print(b)
    print(a)

it would be very strange if a caller could do foo(*iterator), the iterator raises an exception when it produces the value for a, but you only see it when you do print(b) (at which point it has to advance the iterator twice to lazily populate b). No one would have the foggiest idea what went wrong. And literally every function would have to deal with the fact that simply loading its arguments (without doing anything with them) might raise an exception. Not pretty.

When it's reasonable to handle lazy iterators (it isn't for zip; the very first output would need to read from all the arguments anyway, so at best you'd delay the realization of the arguments from the moment of construction to the first time you extract a value from it, saving nothing unless you build a zip object and discard it unused), just accept the iterator directly. Or do both; itertools' chain allows both an eager:

for item in chain(iter1, iter2):

and a lazy:

for item in chain.from_iterable(iter_of_iters):

call techniqe, precisely because it didn't want to force people with an iter_of_iters to realize all of the iterators in memory before it chained a single value from the first one (which is what for item in chain(*iter_of_iters): would require).