How to annotate wrapper for Optional[int]? Difficulty: when it's a parameter

Question

I can't seem to find an acceptable annotation for a variable that receives a value that is either: a) function returning int or b) None. The wrinkle is that the function returns a value that is an optional keyword parameter of the parent function, so it was previously declared as Optional[int]. However, the runtime assignment guarantees that the function will never return None.

If I remove the annotation, mypy accepts it as golden. But I'd prefer to use some (acceptable) annotation. In for a penny, in for a pound...

My code:

from typing import Optional, Callable


def myfun(p1: Optional[int] = None):
    # mypy complains about this
    dyn_p1:Optional[Callable[[], int]] = (lambda: p1) if p1 else None

    # ... but has no problem with this
    # dyn_p1 = (lambda: p1) if p1 else None

    otherFun(dyn_p1)


# I expect the parameter annotation here to be checked at the point of invocation above.
def otherFun(dyn_p1: Optional[Callable[[], int]]):
    pass

Here's the mypy error:

PS $ mypy .\prompt_toolkit\shortcuts\test1.py
prompt_toolkit\shortcuts\test1.py:6: error: Incompatible types in assignment (expression has type "Optional[Callable[[], Optional[int]]]", variable has type "Optional[Callable[[], int]]")
prompt_toolkit\shortcuts\test1.py:6: error: Incompatible return value type (got "Optional[int]", expected "int")
Found 2 errors in 1 file (checked 1 source file)

# comment out the first dyn_p1 and uncomment the second, run again:
PS $ mypy .\prompt_toolkit\shortcuts\test1.py
Success: no issues found in 1 source file

Answer 1

I (kinda) don't think the issue here is your lambda. I think it's actually p1.

Suppose you had instead

def myfun(p1: int = 5):
    dyn_p1:Optional[Callable[[], int]] = (lambda: p1) if p1 > 0 else None

After all, the lambda returns p1. p1 is an integer. The lambda returns an integer. The lambda is Callable[[], int]. Except when it's None, and Optional makes everything happy.

In your case, p1 isn't an int. It's an Optional[int]. So MyPy correctly infers that it's a Callable[[], Optional[int]]. That disagrees with the type hint, so it squawks.

Now you might say "But MyPy should be able to do code flow analysis, and know that the lambda only gets asigned in the case where p1 isn't None."

Which... is true. And MyPy can do some code flow analysis. For example, it's fine with this:

def test(p1: Optional[int]) -> None:
    p2: Union[int, str] = p1 if p1 is not None else "Fail"

I suspect the catch here is that lambdas are late bound. That is, although you've checked that your p1 is not None at the point where you assign it, it's still entirely typing legal to set it back to None after you create dyn_p1.

If you had the code

def myfun(p1: Optional[int] = None):
    dyn_p1:Optional[Callable[[], int]] = (lambda: p1) if p1 else None
    p1 = None
    return dyn_p1()
myfun(5)

even though p1 starts as an int, the result would be None because that's what p1 is at the point the lambda is executed.

MyPy doesn't analyse control flow to ensure this variable never gets changed to something type-legal. Instead it warns you that easy-to-forget Python rules mean that might not be as predictable as it looks at first sight.

One thing you could do is introduce another variable which is annotated as int. Because it would not be legal to change the new variable to None, MyPy can stopy worrying.

def myfun(p1: Optional[int] = None):
    if p1 is not None:
        p1_: int = p1
        dyn_p1:Optional[Callable[[], int]] = (lambda: p1_)
    else:
        dyn_p1 = None
    otherFun(dyn_p1)

---

Rather ironically, in your example of it works if you don't check, MyPy doesn't notice the late binding problem. It's not marking it as Any or anything; it actually infers the type you wanted to tell it!

def test(p1: Optional[int] = 5) -> int:
    dyn_p1 = (lambda: p1) if p1 else None
    reveal_type(dyn_p1)
    p1 = None
    reveal_type(dyn_p1)
    return dyn_p1() if dyn_p1 is not None else 0

This type checks fine. The revealed type is 'Union[def () -> builtins.int, None]' both times (which is MyPy spelling of Optional[Callable[[], int]]). But if you actually call it, it sneaks through an unauthorised None.

Answer 2

It simply needs to be changed as follows

from typing import Optional, Callable


def myfun(p1: Optional[int] = None):
    dyn_p1: Optional[Callable[[], Optional[int]]] = (lambda: p1) if p1 else None
    otherFun(dyn_p1)


def otherFun(dyn_p1: Optional[Callable[[], Optional[int]]]):
    pass

This is because you've clearly defined p1 to be an optional int, and therefore if p1 is to be returned by the lambda, then dyn_p1 must be able to return an optional int, not just an int.

Now running mypy

(venv) ➜  pythonProject mypy okay.py
Success: no issues found in 1 source file