Abstract classes and PyMongo; can't instantiate abstract class

Question

I created the empty abstract class AbstractStorage and inherited the Storage class from it:

import abc
import pymongo as mongo

host = mongo.MongoClient()

print(host.alive()) # True

class AbstractStorage(metaclass=abc.ABCMeta):
    pass

class Storage(AbstractStorage):
    dbh = host
    def __init__(self):
        print('__init__')

Storage()

I expected the output to be

True
__init__

however, the one I'm getting is

True
Traceback (most recent call last):
  File "/home/vaultah/run.py", line 16, in <module>
    Storage()
TypeError: Can't instantiate abstract class Storage with abstract methods dbh

The problem (apparently) goes away if I remove metaclass=abc.ABCMeta (so that AbstractStorage becomes an ordinary class) and/or if I set dbh to some other value.

What's going on here?

Answer 1

This isn't really a problem with ABCs, it's a problem with PyMongo. There is an issue about it here. It seems that pymongo overrides __getattr__ to return some sort of database class. This means that host.__isabstractmethod__ returns a Database object, which is true in a boolean context. This cause ABCMeta to believe that host is an abstract method:

>>> bool(host.__isabstractmethod__)
True

The workaround described in the issue report is to manually set host.__isabstractmethod__ = False on your object. The last comment on the issue suggests a fix has been put in for pymongo 3.0.

Answer 2

Short Version

mongo.MongoClient returns an object that appears to be (is?) an abstract method, which you then assign to the dbh field in Storage. This makes Storage an abstract class, so instantiating it raises a TypeError.

Note that I don't have pymongo, so I can't tell you anything more about MongoClient than how it gets treated by ABCMeta.

Long Version

The ABCMeta.__new__ method looks inside each field of the new class it's creating. Any field that itself has a True (or "true-like") __isabstractmethod__ field is considered an abstract method. If a class has any non-overridden abstract methods, the whole class is considered abstract, so any attempt to instantiate it is an error.

From an earlier version of the standard library's abc.py:

def __new__(mcls, name, bases, namespace):
    cls = super().__new__(mcls, name, bases, namespace)
    # Compute set of abstract method names
    abstracts = {name
                 for name, value in namespace.items()
                 if getattr(value, "__isabstractmethod__", False)}
    # ...
    cls.__abstractmethods__ = frozenset(abstracts)
    # ...

This is not mentioned in the abc.ABCMeta class docs, but a bit lower, under the @abc.abstractmethod decorator:

In order to correctly interoperate with the abstract base class machinery, the descriptor must identify itself as abstract using __isabstractmethod__. In general, this attribute should be True if any of the methods used to compose the descriptor are abstract.

Example

I created a bogus "abstract-looking" class with an __isabstractmethod__ attribute, and two supposedly-concrete subclasses of AbstractStorage. You'll see that one produces the exact error you're getting:

#!/usr/bin/env python3


import abc
# I don't have pymongo, so I have to fake it.  See CounterfeitAbstractMethod.
#import pymongo as mongo


class CounterfeitAbstractMethod():
    """
    This class appears to be an abstract method to the abc.ABCMeta.__new__
    method.

    Normally, finding an abstract method in a class's namespace means
    that class is also abstract, so instantiating that class is an
    error.

    If a class derived from abc.ABCMeta has an instance of
    CounterfeitAbstractMethod as a value anywhere in its namespace
    dictionary, any attempt to instantiate that class will raise a
    TypeError: Can't instantiate abstract class <classname> with
    abstract method <fieldname>.
    """
    __isabstractmethod__ = True


class AbstractStorage(metaclass=abc.ABCMeta):

    def __init__(self):
        """
        Do-nothing initializer that prints the name of the (sub)class
        being initialized.
        """
        print(self.__class__.__name__ + ".__init__ executing.")
        return


class ConcreteStorage(AbstractStorage):
    """
    A concrete class that also _appears_ concrete to abc.ABCMeta.  This
    class can be instantiated normally.
    """
    whatever = "Anything that doesn't appear to be an abstract method will do."


class BogusStorage(AbstractStorage):
    """
    This is (supposedly) a concrete class, but its whatever field appears
    to be an abstract method, making this whole class abstract ---
    abc.ABCMeta will refuse to construct any this class.
    """
    #whatever = mongo.MongoClient('localhost', 27017)
    whatever = CounterfeitAbstractMethod()


def main():
    """
    Print details of the ConcreteStorage and BogusStorage classes.
    """
    for cls in ConcreteStorage, BogusStorage:
        print(cls.__name__ + ":")
        print("    whatever field: " + str(cls.whatever))
        print("    abstract methods: " + str(cls.__abstractmethods__))
        print("    Instantiating...")
        print("    ", end="")
        # KABOOM!  Instantiating BogusStorage will raise a TypeError,
        # because it appears to be an _abstract_ class.
        instance = cls()
        print("    instance: " + str(instance))
        print()
    return


if "__main__" == __name__:
    main()

Running this produces:

$ ./storage.py
ConcreteStorage:
    whatever field: Anything that doesn't appear to be an abstract method will do.
    abstract methods: frozenset()
    Instantiating...
    ConcreteStorage.__init__ executing.
    instance: <__main__.ConcreteStorage object at 0x253afd0>

BogusStorage:
    whatever field: <__main__.CounterfeitAbstractMethod object at 0x253ad50>
    abstract methods: frozenset({'whatever'})
    Instantiating...
    Traceback (most recent call last):
  File "./storage.py", line 75, in <module>
    main()
  File "./storage.py", line 68, in main
    instance = cls()
TypeError: Can't instantiate abstract class BogusStorage with abstract methods whatever