functools.partial support

[berdario]

I haven't been able to find another issue for this, weird that this hasn't been reported already.

from typing import *
from functools import partial

def inc(a:int) -> int:
    return a+1

def foo(f: Callable[[], int]) -> int:
    return f()

foo(partial(inc, 1))

This will fail with

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[], int]

a similar code by using a dyadic add instead of inc will yield:

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[int], int]

(so, partial apparently doesn't currently retain any type information on the non-applied inputs... I guess this'll make the fix non trivial)

[JukkaL]

The error message is generated because a class with a __call__ method (in this case, partial) isn't considered a subtype of Callable, even though it should be. See #797.

Also, partial only retains information about the return type, since the type system can't represent a more precise type.

[berdario]

the type system can't represent a more precise type.

You mean that the partial class doesn't represent it at runtime, right?
I think mypy could special case partial, and coerce it to the appropriate Callable type transparently

[JukkaL]

I meant that mypy would have to special case partial, since we can't write a good enough stub using PEP 484 features only. Mypy already does some special casing, but for this I'd rather have a more general mypy plugin/extension system instead of even more ad-hoc special case logic in the type checker.

[gvanrossum]

Maybe this could be supported by #3299

[JukkaL]

Yes, this looks like a good candidate for function plugins. Increasing priority since this is may be pretty easy to implement now. Just supporting positional arguments for partial should cover the majority of uses and would probably be good enough.

[ilevkivskyi]

This particular case and probably other basic use cases are already supported by protocols (see tests in PR #3132), simply because partial has __call__ and therefore is a structural subtype of Callable.

[JukkaL]

Protocols won't help with inferring the argument types of the result, though?

[ilevkivskyi]

Protocols won't help with inferring the argument types of the result, though?

Yes, I didn't do any special-casing, partial in typeshed is only generic in one type _T which is the original return type. Maybe with the new support for flexible Callable this might be improved, but probably a plugin will still be required if we want precise types.

[ilevkivskyi]

The original example is now fixed by #3132. This issue can be kept open to track better support of partial using a plugin.

[astrojuanlu]

Not sure if this has been fixed entirely or perhaps it's a different issue:

(py34) juanlu@centauri /tmp $ cat test.py 
from functools import reduce, partial

def sum_two(a, b, c=0):
  return a + b + c

f_0 = partial(sum_two, c=0)
print(reduce(f_0, [1, 2, 3], 10))
(py34) juanlu@centauri /tmp $ mypy --version
mypy 0.550
(py34) juanlu@centauri /tmp $ mypy --ignore-missing-imports --check-untyped-defs test.py 
test.py:7: error: No overload variant of "reduce" matches argument types [functools.partial[Any], builtins.list[builtins.int*], builtins.int]

[nicktimko]

I think this crops up also when trying to use a partial for a defaultdict factory:

from collections import defaultdict
from functools import partial
from typing import cast, Callable, Mapping

# error: No overload variant of "defaultdict" matches argument types [functools.partial[builtins.int*]]
x: Mapping[str, int] = defaultdict(partial(int, 10))

# (ok, but feels like it shouldn't be needed)
y: Mapping[str, int] = defaultdict(cast(Callable, partial(int, 10)))

[JukkaL]

python/typeshed#2878 changed how partial works. Now mypy can often preserve the argument types, but it loses keyword argument names. It may still be worth it to special case partial using a plugin.

[wkschwartz]

+1 for supporting partial's instance attributes: func, args, and keywords.

[raphCode]

I stumbled across this:

from typing import reveal_type
from functools import partial

class A:
    pass

p = partial(A)

reveal_type(p)  # Revealed type is "functools.partial[A]"
reveal_type(p.func)  # Revealed type is "def (*Any, **Any) -> A"
print(issubclass(p.func, A))
# error: Argument 1 to "issubclass" has incompatible type "Callable[..., A]"; expected "type"  [arg-type]

I don't know if this is considered a mypy bug or even related to partials, but the code runs fine, and issubclass returns True as expected.

[NeilGirdhar]

I don't know if this is considered a mypy bug or even related to partials, but the code runs fine, and issubclass returns True as expected.

It's because in the typeshed, you have

class partial(Generic[_T]):
    func: Callable[..., _T]

So, the class is transformed into a callable on assignment. MyPy is doing the right thing here.

[raphCode]

I see.
Is this the desired behavior? The typeshed does not seem to reflect what partial does at runtime in this case.

[NeilGirdhar]

Is this the desired behavior?

Yes, assigning to a member of a class can lose type information. Just like tuple(list(('a', 'b'))) gives you tuple[str, ...] and not tuple[str, str].

This issue is about partial matching callable.

[Dreamsorcerer]

Should this be a separate issue?

A partial of a type is not recognised as a type:

def foo(a: Type[int], **kwargs):
    x = a(1)

foo(functools.partial(int, base=10))  # Argument 1 to "foo" has incompatible type "partial[int]"; expected "Type[int]"

Admittedly, this could also be not type-safe:

foo(functools.partial(int, 5))

Not sure if there's a better way to handle this. My actual use case is to accept any subclass of an abstract type, but some implementations may need positional arguments and some not, so I use partial to include the positional arguments needed before passing into the function.

[erictraut]

@Dreamsorcerer, I don't think your example above should type check. In other words, I think mypy is doing the right thing because partial[int] is not type compatible with type[int]. For example, if you attempt to pass a base keyword argument to partial[int], you will receive an error because base was already provided. You can make your code type safe by changing the definition of foo to the following, which type checks without error:

def foo(a: Callable[..., int], **kwargs): ...

[Dreamsorcerer]

Right, but then we lose all typing for arguments.

My actual use case is:

def init_app(session_storage: Type[aiohttp_session.AbstractStorage]):
    ...
    storage = session_storage(cookie_name="SESSION", max_age=3600, secure=True)

# First use case
init_app(aiohttp_session.SimpleCookieStorage)
# Second use case, which needs a positional key parameter
init_app(functools.partial(EncryptedCookieStorage, Fernet(config["fernet_key"]))

Would be nice not to lose typing on the kwargs (and I get an explicit Any error too).

[Dreamsorcerer]

Although, that's interesting that I can do init_app(EncryptedCookieStorage), which then type checks, but would fail at runtime...

So, actually, maybe the problem is that it allows a subclass of the abstract type to require an argument which the abstract class does not require...
https://github.com/aio-libs/aiohttp-session/blob/master/aiohttp_session/cookie_storage.py#L18

[NeilGirdhar]

So, actually, maybe the problem is that it allows a subclass of the abstract type to require an argument which the abstract class does not require...

In Python, constructors don't obey LSP. That's why you shouldn't annotate the function with type[...]: there's no way to know how with which parameters to construct it. You should annotate with Callable.

Right, but then we lose all typing for arguments.

You can type the arguments by providing them to Callable or by creating a protocol.