Custom Manager in Django Destroys Caching for prefetch_related

Question

If I do everything below without a custom manager, it all works as expected:

class Content(models.Model):
   name = models.CharField(max_length=200)
   def __str__(self):
      return self.name
   class Meta:
      app_label = 'game'

class Requirement(models.Model):
   content = models.ForeignKey(Content, on_delete=models.CASCADE, related_name = 'requirements')
   value = models.IntegerField(default=0)
   def __str__(self):
      return "{} requires value {}".format(self.content,self.value)
   class Meta:
      app_label = 'game'

def testPrefetchOrig():
   contents = Content.objects.filter(name__startswith = 'a').prefetch_related('requirements')
   for c in contents:
      for r in c.requirements.all():
         print r
         logging.warning(r)
   logging.warning("Done with query")

This prefetches the data once and never again:

DEBUG:django.db.backends:(0.001) SELECT "game_content"."id", "game_content"."name", "game_content"."deleted" FROM "game_content" WHERE "game_content"."name"::text LIKE 'a%'; args=(u'a%',)
DEBUG:django.db.backends:(0.001) SELECT "game_requirement"."id", "game_requirement"."content_id", "game_requirement"."value", "game_requirement"."deleted" FROM "game_requirement" WHERE "game_requirement"."content_id" IN (5, 6); args=(5, 6)
alphabet requires value 5
WARNING:root:alphabet requires value 5
alphabet requires value 3
WARNING:root:alphabet requires value 3
albatross requires value 1
WARNING:root:albatross requires value 1
albatross requires value 0
WARNING:root:albatross requires value 0
WARNING:root:Done with query

However, I want to use a custom manager to handle filtering of entries that are 'deleted' by setting a deleted flag.

class DeletedItemsQuerySet(models.query.QuerySet):
   def get(self, *args, **kwargs):
      kwargs['deleted']=False
      return models.query.QuerySet.get(self, *args, **kwargs)
   def all(self):
      return self.filterNoDeleted()
   def filterNoDeleted(self, *args, **kwargs):
      kwargs['deleted']=False
      return models.query.QuerySet.filter(self, *args, **kwargs)
   def getDeleted(self, *args, **kwargs):
      return models.query.QuerySet.get(self, *args, **kwargs)
   def filterDeleted(self, *args, **kwargs):
      return models.query.QuerySet.filter(self, *args, **kwargs)

class DeletedItemsManager(models.Manager.from_queryset(DeletedItemsQuerySet)):
   def all(self):
      return super(models.Manager,self).all().filterNoDeleted()

And then we modify our models to use this:

class Content(models.Model):
   name = models.CharField(max_length=200)
   objects = DeletedItemsManager()
   deleted = models.BooleanField(default=False)
   def __str__(self):
      return self.name
   class Meta:
      app_label = 'game'

class Requirement(models.Model):
   content = models.ForeignKey(Content, on_delete=models.CASCADE, related_name = 'requirements')
   value = models.IntegerField(default=0)
   deleted = models.BooleanField(default=False)
   objects = DeletedItemsManager()
   def __str__(self):
      return "{} requires value {}".format(self.content,self.value)
   class Meta:
      app_label = 'game'

def testPrefetchOrig():
   contents = Content.objects.filter(name__startswith = 'a').prefetch_related('requirements')
   for c in contents:
      for r in c.requirements.all():
         print r
         logging.warning(r)
   logging.warning("Done with query")

This prefetches the data, but still queries it:

DEBUG:django.db.backends:(0.001) SELECT "game_content"."id", "game_content"."name", "game_content"."deleted" FROM "game_content" WHERE "game_content"."name"::text LIKE 'a%'; args=(u'a%',)
DEBUG:django.db.backends:(0.001) SELECT "game_requirement"."id", "game_requirement"."content_id", "game_requirement"."value", "game_requirement"."deleted" FROM "game_requirement" WHERE "game_requirement"."content_id" IN (5, 6); args=(5, 6)
DEBUG:django.db.backends:(0.000) SELECT "game_requirement"."id", "game_requirement"."content_id", "game_requirement"."value", "game_requirement"."deleted" FROM "game_requirement" WHERE ("game_requirement"."content_id" = 5 AND "game_requirement"."deleted" = false); args=(5, False)
alphabet requires value 5
WARNING:root:alphabet requires value 5
alphabet requires value 3
WARNING:root:alphabet requires value 3
DEBUG:django.db.backends:(0.001) SELECT "game_requirement"."id", "game_requirement"."content_id", "game_requirement"."value", "game_requirement"."deleted" FROM "game_requirement" WHERE ("game_requirement"."content_id" = 6 AND "game_requirement"."deleted" = false); args=(6, False)
albatross requires value 1
WARNING:root:albatross requires value 1
albatross requires value 0
WARNING:root:albatross requires value 0
WARNING:root:Done with query

How do I use the custom manager and still have prefetch_related work?

Answer 1

Okay. The DeletedItemsManager adds an extra filterNoDeleted() method call onto every all() method call. That filter call destroys the cache, as per the note in https://docs.djangoproject.com/en/1.11/ref/models/querysets/

However, I really do need to filter out the deleted related objects.

I should be able to select all the required rows in the Requirement table, though I'm not sure how to set the right fields in the Queryset to reflect those results. However, if I only want to read the results, I can probably turn that into a dictionary.

Does anyone have a more elegant solution?

Note: There is a pull request for a filtered_relation addition to Django, but that doesn't yet work with prefetch_related. Perhaps in the future Django will support this use case.

Answer 2

def testPrefetchOrig():
contents = Content.objects.filter(name__startswith = 'a').prefetch_related(Prefetch('requirements', queryset=Requirement.objects.filterNoDeleted(),to_attr='undeletedRequirements'))
for c in contents:
    for r in c.undeletedRequirements:
        print r
print "Done with query"