“iter() returned non-iterator” for dynamically bound `next` method

Question

Why does this next method I dynamically bound to an instance of a class fail and return a non-iterator object?

from collections import Iterator
from collections import Iterable
from types import MethodType

def next(cls):
    if cls.start < cls.stop:
        cls.start += 1
        return cls.start
    else:
        raise StopIteration


class Foo(object):
    start, stop = 0, 5

    def __iter__(self):
        return self

if __name__ == "__main__":
    foo = Foo()
    setattr(foo, 'next', MethodType(next, foo, Foo))
    print hasattr(foo, "next")
    if isinstance(foo, Iterable):
        print "iterable"
    if isinstance(foo, Iterator):
        print "iterator"

    for i in foo:
        print i

Output:

iterable
True
TypeError: iter() returned non-iterator of type 'Foo'

It worked properly, when I did setattr(Foo, 'next', classmethod(next)).

Answer 1

for i in foo:
    print i

This is the code that failed, so let’s take a look at what happens internally there, diving into the source for some Python internals!

When for i in foo is compiled, the generated bytecode will contain the GET_ITER opcode which is responsible of converting foo into an iterable. GET_ITER causes a PyObject_GetIter call on the object which is the actual implementation that provides the iterable. So let’s take a look at what it does:

PyObject * PyObject_GetIter(PyObject *o)
{
    PyTypeObject *t = o->ob_type;
    getiterfunc f = NULL;
    if (PyType_HasFeature(t, Py_TPFLAGS_HAVE_ITER))
        f = t->tp_iter;
    if (f == NULL) {
        if (PySequence_Check(o))
            return PySeqIter_New(o);
        return type_error("'%.200s' object is not iterable", o);
    }
    else {
        PyObject *res = (*f)(o);
        if (res != NULL && !PyIter_Check(res)) {
            PyErr_Format(PyExc_TypeError,
                         "iter() returned non-iterator of type '%.100s'",
                         res->ob_type->tp_name);
            Py_DECREF(res);
            res = NULL;
        }
        return res;
    }
}

As you can see (if you understand some basic C at least), the underlying type is looked up from the object first (o->ob_type), and then its iter function is read (t->tp_iter).

Since you have implemented an __iter__ function on the type, this function does exist, so we get to the else case in the code above, which runs the iter function on the object o. The result is non-null, but we still get the “returned non-iterator” message, so the PyIter_Check(res) appears to have failed. So let’s take a look at what that does:

#define PyIter_Check(obj) \
    (PyType_HasFeature((obj)->ob_type, Py_TPFLAGS_HAVE_ITER) && \
    (obj)->ob_type->tp_iternext != NULL && \
    (obj)->ob_type->tp_iternext != &_PyObject_NextNotImplemented)

So this one essentially checks if the type (ob_type) of the passed object has a non-null next method (tp_iternext) which does not happen to be the not-implemented next function.

Check closely where that check happens though: On the result’s type, not the result itself. The foo object does have a next function, but its type Foo does not have one.

setattr(foo, 'next', MethodType(next, foo, Foo))

… or more explicitly …

foo.next = next.__get__(foo, Foo)

… does only set the bound next method on the instance but not on the type itself. So the above C code will fail consuming it as an iterable.

If you were to set your next function on the type instead, it would work fine:

foo = Foo()
Foo.next = next

for i in foo:
    print i

This is the reason why your attempt with classmethod worked: You set the function on the type instead of its concrete instance.