Python shelve module type hints in PyCharm, "expected type xx, got Shelf[object] instead"

Question

I'm trying to understand why I'm getting IDE warnings about how I'm using the Python shelve module. All of the code here functions fine, I'm just getting IDE (PyCharm 2022.2.3 community edition) type warnings that I'd like to clean up.

A super basic test that exemplifies the issue:

import shelve

s = shelve.open("testshelf.db")

s["my_list"]: list = [1, 2, 3, 4]

print(len(s["my_list"]))

Here I get a warning trying to get len(s["my_list"]):

I don't have enough reputation to post images, but the warning highlights s["my_list"] and says:

Expected type 'Sized', got 'object' instead

I tried to get fancier with it based on another answer on here, but it just moved the issue:

import shelve
from typing import Dict

s = shelve.open("testshelf.db")

s["my_list"]: list = [1, 2, 3, 4]


def foo(s_: Dict[str, list]):
    print(len(s_["my_list"]))


foo(s)

Now I get a similar warning highlighting s in foo(s):

Expected type 'dict[str, list]', got 'Shelf[object]' instead

Again, both examples run fine and produce expected output, I'd just like to get rid of the IDE warnings if possible. I am still learning about type-hinting and this is my first use of Python shelve. It seems to be a very convenient and easy-to-use solution to a use case I have, but not if I have to suffer IDE warnings or disable them or something.

Answer 1

The issue here is that shelve.open returns a shelve.Shelf instance (specifically a shelve.DbfilenameShelf). That is a MutableMapping subtype.

A shelve.Shelf thus behaves essentially like a regular old dict, when it comes to type safety. Since the created Shelf can hold essentially any type of object, the inferred type is shelve.Shelf[Any]. And all values in a mapping are treated the same by a static type checker. See this related post here for more insight.

The type checker can't know that a specific value has the __len__ method implemented. It must treat them all as object. Not all objects have a concept of size. That is where the warning comes from.

As for solutions, there are a few options. Here are the most practical ones IMHO.

A) Intermediary variable

The simplest one in my opinion is to use an intermediary variable.

Write like this:

import shelve

s = shelve.open("testshelf.db")
my_list: list[int] = [1, 2, 3, 4]
# do stuff with `my_list`
s["my_list"] = my_list

Read like this: (declares the list type argument to be int)

import shelve
from typing import cast

s = shelve.open("testshelf.db")

my_list = cast(list[int], s["my_list"])
print(len(my_list))

Or alternatively like this: (list elements will be inferred as Any)

import shelve

s = shelve.open("testshelf.db")

my_list = s["my_list"]
assert isinstance(my_list, list)
print(len(my_list))

B) Declare items as Sized

If you know that all items in your shelf will implement the Sized protocol (i.e. have the __len__ method), you can cast it as such:

import shelve
from collections.abc import Sized
from typing import cast

s = cast(
    shelve.Shelf[Sized],
    shelve.open("testshelf.db"),
)

s["my_list"] = [1, 2, 3, 4]

print(len(s["my_list"]))

Obviously, if you know that all items will be not just "something Sized", but specifically lists of integers, you could cast it like this as well. Just use cast(shelve.Shelf[list[int]], ...) instead.

C) Fine-grained structural subtyping

This may be overkill, but if you know that all items in your shelf will share a certain protocol (/interface) beyond just __len__, you can explicitly define one in advance and the cast it accordingly.

Say for example that all your objects in the shelf will be some generic containers with elements of type T and have the __len__ and __iter__ methods, as well as a remove method. You handle it like so:

import shelve
from collections.abc import Iterator
from typing import Protocol, TypeVar, cast

T = TypeVar("T")

class ShelfItem(Protocol[T]):
    def __len__(self) -> int:
        ...

    def __iter__(self) -> Iterator[T]:
        ...

    def remove(self, element: T) -> None:
        ...

s = cast(
    shelve.Shelf[ShelfItem[int]],
    shelve.open("testshelf.db"),
)
...

---

References

• typing.cast
• Collections ABC
• typing.Protocol