Literals overlap when defaults given

[llchan]

It appears that Literals with default values lead to overlapping signatures. In the following example, the defaults are necessary since this argument follows another defaulted argument.

• Are you reporting a bug, or opening a feature request? bug
• Please insert below the code you are checking with mypy.

  @overload
  def foo(x: int = 0, y: Literal[False] = False) -> int: ...
  @overload
  def foo(x: int = 0, y: Literal[True] = True) -> str: ...
  @overload
  def foo(x: int = 0, y: bool = False) -> Union[int, str]: ...

• What is the actual behavior/output?

  Overloaded function signatures 1 and 2 overlap with incompatible return types
  

• What is the behavior/output you expect?
  No error
• What are the versions of mypy and Python you are using?
  mypy 0.680+dev.4e0a1583aeb00b248e187054980771f1897a1d31 (current master)
  CPython 3.7.1

[gvanrossum]

But mypy is right. When you call foo() it won’t know whether the return is int or str.

[llchan]

In the foo() case, I agree it could be either return type because there is no literal False nor literal True so it would match signature 3. However, as soon as there's a literal it should be unambiguous, I think. Am I'm overlooking something?

[gvanrossum]

The point is that without arguments the defaults take over.

[llchan]

I guess maybe let me rephrase the question: how do I specify a literal keyword argument in the middle of other keyword arguments? It seems that we may need a sentinel value so we can signal to the overload to only match that signature if the keyword is explicitly present as a literal.

@overload
def foo(x: int = 0, y: Literal[False] = REQUIRED) -> int: ...
@overload
def foo(x: int = 0, y: Literal[True] = REQUIRED) -> str: ...
@overload
def foo(x: int = 0, y: bool = False) -> Union[int, str]: ...

The desired behavior:

reveal_type(foo())          # Union[int, str]
reveal_type(foo(0, False))  # int
reveal_type(foo(0, True))   # str
reveal_type(foo(y=False))   # int
reveal_type(foo(y=True))    # str
reveal_type(foo(y=z))       # Union[int, str]

[llchan]

I should mention that I'm adding types to code that must continue to support PY2 for now (I know, I know, but migrating takes time, and there are many downstream users), so making these keyword-only with the star is not an option.

[gvanrossum]

You don't have to provide a default to make something a keyword argument.

def foo(x: int, y: Literal[True]) -> ...

can still be called as

foo(x=42, y=True)

With overloading I would recommend overloading all variants with mandatory arguments, e.g.

@overload
def foo(x: int, y: Literal[True]) -> str: ...
@overload
def foo(x: int, y: Literal[False]) -> int: ...
@overload
def foo(x: int) -> int: ...

[llchan]

My bad, my terminology was a bit imprecise. I meant a literal after one or more defaulted arguments. The x argument has a default that forces us to stick a default on the y argument. I'd like for us to be able to have a way to say "match this only when y has a literal binding", but we need some placeholder to put as the default.

[gvanrossum]

So make two sets of overloads -- one set with x: int and one set without it. Then in each overload x is mandatory if it is present at all, and you don't need a default for y.

[llchan]

This toy example only has one defaulted argument before the literal, but the actual function has many other defaulted arguments before, which would require 2N 2^(N +1) overloads manually spelled out. I'd argue that the sentinel value solves this more elegantly and conveys the intent in a more readable manner.

If this is a wont-fix due to limited dev resources, that's understandable, I just want to make sure I'm getting the proposal across.

EDIT: was only thinking positional calls. it would need at least 2^(N+1) overloads, and also this strategy doesnt work if any other defaulted arguments have a matching type (e.g. if x were a bool in the example above).

[JukkaL]

It looks like this is a real problem, and mypy could probably support this use case without requiring a huge number of overloads. I suspect that the same issue will at least occasionally come up when using literal types in library stubs. Reopening the issue, though I can't estimate when the mypy team might have resources to fix this.

[DevilXD]

I don't get this whole issue, overload typings shouldn't include default arguments - what if someone specifies different defaults across multiple overloads?

The overload decorator specification itself says that an unannotated function declaration should immediately follow the overloaded variants, hence why this should be correct:

@overload
def foo(x: int, y: Literal[False]) -> int:
    ...

@overload
def foo(x: int, y: Literal[True]) -> str:
    ...

def foo(x=0, y=False):
    ...

The only problem is that MyPy doesn't seem to look at the actual function declaration (last one) for the defaults (yet?).

[llchan]

@DevilXD yes, your last sentence is essentially the crux of this problem. Without a mechanism to propagate defaults into the overloads, we need to include defaulted-ness in the overload signatures. However, this leads to "incompatible return types" issues, and furthermore we can't specify literals without defaults after defaulted arguments. The only way to solve this currently is to manually define each overload for signatures with/without those arguments set (roughly exponential). For example, in your snippet above, if there's a foo(y=True) call, you'll get a "No overload variant" error since there isn't an explicitly defined overload for that. It would require a def foo(*, y: Literal[True]) -> str: overload to be defined.

[DevilXD]

Eh, I'd say that's stupid then. You shouldn't need to define all possible variants in this case, those two should be enough. I commented on this because it seemed like everyone was trying to put the defaults into overloads themselves, which doesn't sound like a good idea to me.

I found this issue due to me having a similar problem, here's my example:

@overload
async def get_players(
    self, player_ids: List[int], *, return_private: Literal[False] = False
) -> List[Player]:
    ...

@overload
async def get_players(
    self, player_ids: List[int], *, return_private: Literal[True]
) -> List[Union[Player, PartialPlayer]]:
    ...

async def get_players(self, player_ids, *, return_private=False):
    ...

Without that = False in the first overload, MyPy doesn't seem to match .get_player([1234]) case and reports Any via reveal_type. When I add the default there, it works correctly, but then I figured the default should be included in the second overload too, and that's where it didn't match the Literal and reported an error. Adding a default of True there just feels wrong, mostly because it'd be incorrect and wouldn't match the final implementation, so I just left it without one. The final implementation is really where the information about defaults should come from, not the overloads.

[llchan]

I agree with you in principle. The overloads shouldn't need to specify defaults (or more strongly put, maybe they shouldnt specify defaults), since the defaults are mostly* just a fancy mechanism to automatically bind values to omitted parameters. I'd maybe even take your idea a step further and say that a omitted parameter like this that defaults to True/False should map to the the corresponding literal case, rather than the bool case that I believe it would map to now.

However, there may be nuances that we are overlooking. Furthermore that ship may have already sailed and it may require a PEP to change/augment the way overloads work. I'll defer to a maintainer to chime in.

[Michael0x2a]

The reason why the defaults need to be specified in the overload variant list is due to the intersection of the following factors:

1. Whether or not a param is optional is a crucial part of the function type signature.

   For example, def (x: int, y: int = ...) -> str is a subtype of def (x: int) -> str, but def (x: int, y: int) -> str is not.

2. To make overloads as maximally useful as possible, we want them to completely and fully override/replace the implementation signature.

   This makes it possible to do things like have the implementation for any overload have the signaturedef (*args: object, **kwargs: object) -> Any without mypy complaining or getting in the way -- or in stub files, just omit the implementation entirely.

   And if we do want the implementation signature to "count" in some way, it's easy enough to just add another overload variant duplicating it.

The consequence of these two factors is that we end up needing to encode whether an argument is positional-only, keyword-only, optional, mandatory, etc... in the overload variant signatures.

---

Regarding the async def get_players example -- it's true that having to omit the = True default in the second overload variant can seem a little weird and asymmetrical. However, that asymmetry is actually an accurate reflection of the runtime behavior, and I think it's reasonable to encode that bias directly in the type.

If the objection is that it feels redundant to have to actually repeat the default value, you can use ... ellipsis as a placeholder instead.

@overload
def foo(x: Literal[False] = ...) -> int: ...
@overload
def foo(x: Literal[True]) -> str: ...
def foo(x: bool = False): pass

(Mypy understands the overloads disappear at runtime and so well let ... serve as a placeholder, even though it isn't of type Literal[False].)

---

Regarding the original posted issue -- it seems the crux of the problem there is really that Python 2 doesn't support keyword-only arguments, and you sort of need them to express certain overloads in a clean and compact way?

Unfortunately, I'm not really sure if what mypy can do here. We can't do the same kind of trick we did to support positional-only arguments in older versions of Python (treat any params starting with two underscores as positional only), and given that Python 2 is EOL'd, I'm not sure whether it's worth investing the time into finding a solution.

One workaround might be to move any complicated overloads into a dedicated file with a stub file. Stub files can use Python 3 syntax (even if it's supposed to be hints for Python 2 code), which would sort of sidestep the problem. This solution is admittedly pretty suboptimal though: for example, you'd lose the ability for mypy to type check the implementation.

---

Regarding Jukka's comment: I do think it's worth auditing typeshed to see places where we've had to add # type: ignore comments to overloads and check to see if mypy is being too strict/if it's worth special-casing some edge cases. But IMO this seems tangential to the original issue.

[tibbe]

There also seems to be a related issue with passing keyword arguments with defaults from one overloaded function to another:

from typing import Literal, overload

@overload
def f(*, arg: Literal[True] = ...) -> None:
    ...

@overload
def f(*, arg: Literal[False]) -> None:
    ...

def f(*, arg: bool = True) -> None:
    ...

@overload
def g(*, arg: Literal[True] = ...) -> None:
    ...

@overload
def g(*, arg: Literal[False]) -> None:
    ...

def g(*, arg: bool = True) -> None:
    # No overload variant of "f" matches argument type "bool"  [call-overload] mypy(error)
    # Possible overload variants:mypy(note)
    #     def f(*, arg: Literal[True] = ...) -> None mypy(note)
    #     def f(*, arg: Literal[False]) -> None mypy(note)
    return f(arg=arg)

Is there a way to make this work today?

[DevilXD]

@tibbe I believe that's an entirely different issue: #14764

[ilevkivskyi]

This was fixed recently, as we decided to relax mypy's behavior w.r.t. overlapping overloads where it was technically correct but (highly) impractical.