Running python manage.py test gives "Maximum Recursion Depth reached" error

Question

So I have a django project which was built on 1.6.5 and now I'm migrating it to 1.9.5. I successfully migrated it to 1.7.0 and then to 1.8.0. When doing so from 1.8.0 to 1.9.0, I had to replace SortedDict by collections.OrderedDict. Now I'm encountering this error when I do python manage.py test:

    File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 725, in values_list
    clone = self._values(*fields)
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 671, in _values
    clone = self._clone()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1059, in _clone
    query = self.query.clone()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/sql/query.py", line 298, in clone
    obj._annotations = self._annotations.copy() if self._annotations is not None else None
  File "/opt/python/python-2.7/lib64/python2.7/collections.py", line 194, in copy
    return self.__class__(self)
  File "/opt/python/python-2.7/lib64/python2.7/collections.py", line 57, in __init__
    self.__update(*args, **kwds)
  File "venv/mystuff/lib64/python2.7/abc.py", line 151, in __subclasscheck__
    if subclass in cls._abc_cache:
  File "venv/mystuff/lib64/python2.7/_weakrefset.py", line 72, in __contains__
    wr = ref(item)
RuntimeError: maximum recursion depth exceeded

The solutions on other similar questions ask to upgrade python to 2.7.5 but I'm already running it on 2.7.11.

EDIT: forum/models/base.py

def iterator(self):
    cache_key = self.model._generate_cache_key("QUERY:%s" % self._get_query_hash())
    on_cache_query_attr = self.model.value_to_list_on_cache_query()

    to_return = None
    to_cache = {}

    with_aggregates = len(self.query.aggregates) > 0
    key_list = self._fetch_from_query_cache(cache_key)

    if key_list is None:
        if not with_aggregates:
            values_list = [on_cache_query_attr]

            if len(self.query.extra):
                values_list += self.query.extra.keys()

            key_list = [v[0] for v in self.values_list(*values_list)] #Line 134
            to_cache[cache_key] = (datetime.datetime.now(), key_list)
        else:
            to_return = list(super(CachedQuerySet, self).iterator())
            to_cache[cache_key] = (datetime.datetime.now(), [
                (row.__dict__[on_cache_query_attr], dict([(k, row.__dict__[k]) for k in self.query.aggregates.keys()]))
                for row in to_return])
    elif with_aggregates:
        tmp = key_list
        key_list = [k[0] for k in tmp]
        with_aggregates = [k[1] for k in tmp]
        del tmp

    if (not to_return) and key_list:
        row_keys = [self.model.infer_cache_key({on_cache_query_attr: attr}) for attr in key_list]
        cached = cache.get_many(row_keys)

        to_return = [
            (ck in cached) and self.obj_from_datadict(cached[ck]) or ToFetch(force_unicode(key_list[i])) for i, ck in enumerate(row_keys)
        ]

        if len(cached) != len(row_keys):
            to_fetch = [unicode(tr) for tr in to_return if isinstance(tr, ToFetch)]

            fetched = dict([(force_unicode(r.__dict__[on_cache_query_attr]), r) for r in
                          models.query.QuerySet(self.model).filter(**{"%s__in" % on_cache_query_attr: to_fetch})])

            to_return = [(isinstance(tr, ToFetch) and fetched[unicode(tr)] or tr) for tr in to_return]
            to_cache.update(dict([(self.model.infer_cache_key({on_cache_query_attr: attr}), r._as_dict()) for attr, r in fetched.items()]))

        if with_aggregates:
            for i, r in enumerate(to_return):
                r.__dict__.update(with_aggregates[i])


    if len(to_cache):
        cache.set_many(to_cache, 60 * 60)

    if to_return:
        for row in to_return:
            if hasattr(row, 'leaf'):
                row = row.leaf

            row.reset_original_state()
            yield row

django/db/models/query.py:

def _fetch_all(self):
    if self._result_cache is None:
        self._result_cache = list(self.iterator()) #Line 1074
    if self._prefetch_related_lookups and not self._prefetch_done:
        self._prefetch_related_objects()

django/db/models/query.py

def __iter__(self):
    """
    The queryset iterator protocol uses three nested iterators in the
    default case:
        1. sql.compiler:execute_sql()
           - Returns 100 rows at time (constants.GET_ITERATOR_CHUNK_SIZE)
             using cursor.fetchmany(). This part is responsible for
             doing some column masking, and returning the rows in chunks.
        2. sql/compiler.results_iter()
           - Returns one row at time. At this point the rows are still just
             tuples. In some cases the return values are converted to
             Python values at this location.
        3. self.iterator()
           - Responsible for turning the rows into model objects.
    """
    self._fetch_all() #Line 258
    return iter(self._result_cache)

UPDATE: My Django Log is showing this:

/forum/settings/base.py TIME: 2017-06-27 06:49:53,410 MSG: base.py:value:65 Error retrieving setting from database (FORM_EMPTY_QUESTION_BODY): maximum recursion depth exceeded in cmp
/forum/settings/base.py TIME: 2017-06-27 06:49:53,444 MSG: base.py:value:65 Error retrieving setting from database (FORM_MIN_NUMBER_OF_TAGS): maximum recursion depth exceeded

Answer 1

So you call values_list() from your iterator, which clones the QuerySet, which iterates the QuerySet, which calls your iterator, which clones the Queryset...

The API changed there in this commit. It should provide you with ample information to reimplement your Queryset.

On a different note, it looks like Django has implemented query caching itself, so before refactoring you might take a look if your CachedQuerySet has become obsolete.