Fix crashes and fails in forward references #3952

ilevkivskyi · 2017-09-13T09:56:29Z

Fixes #3340
Fixes #3419
Fixes #3674
Fixes #3685
Fixes #3799
Fixes #3836
Fixes #3881
Fixes #867
Fixes #2241
Fixes #2399
Fixes #1701
Fixes #3016
Fixes #3054
Fixes #2762
Fixes #3575
Fixes #3990

Currently, forward references don't work with anything apart from classes, for example this doesn't work:

x: A
A = NamedTuple('A', [('x', int)])

The same situation is with TypedDicts, NewTypes, and type aliases. The root problem is that these synthetic types are neither detected in first pass, nor fixed in third pass. In certain cases this can lead to crashes (first six issues above are various crash scenarios). I fix these crashes by applying some additional patches after third pass. Here is the summary of the PR:

New simple wrapper type ForwardRef with only one field link is introduced (with updates to type visitors)
When an unknown type is found in second pass, the corresponding UnboundType is wrapped in ForwardRef, it is given a "second chance" in third pass.
After third pass I record the "suspicious" nodes, where forward references and synthetic types have been encountered and append patches (callbacks) to fix them after third pass. Patches use the new visitor TypeReplacer (which is the core of this PR).

Here are two problems that I encountered:

Third pass (both in semanal.py and in typeanal.py) was more "shallow" than the second one, some visitor methods were literally pass. It was necessary to update these to match the "depth" of the second pass.
Now third pass has a link to second pass analyzer, self.sem, same as for first pass. It would be nice to refactor the passes since all three share some code/functionality, there is already Refactor and document semantic analysis passes #3459 to track this.

NOTE: self-referential types are still not properly supported, but now we give a reasonable error for this, not a crash, and they still can be used to certain extent, for example:

class MyNamedTuple(NamedTuple):
    parent: 'MyNamedTuple'

def get_parent(nt: MyNamedTuple) -> MyNamedTuple:
    return nt.parent

x: MyNamedTuple
reveal_type(x.parent)
reveal_type(x[0])

results in

main:2: error: Recursive types not fully supported yet, nested types replaced with "Any"
main:9: error: Revealed type is 'Tuple[Any, fallback=__main__.MyNamedTuple]
main:10: error: Revealed type is 'Tuple[Any, fallback=__main__.MyNamedTuple]

Proper support of recursive types would be much harder (mostly due to (de-)serialization that would require something similar to type_ref), there is a separate issue for this #731.

@JukkaL sorry it took a bit longer than expected because of CPython sprint last week.

… one

ilevkivskyi · 2017-09-13T09:58:14Z

An important comment, for some reason forward references between files still don't work properly (but at least they don't crash).

ilevkivskyi · 2017-09-13T10:10:10Z

This is ready for review, but I marked it WIP since I have found one more crash, will fix it shortly.

…; Better error in case something is still missing

ilevkivskyi · 2017-09-13T13:55:52Z

OK, I have fixed some remaining problems and TODO's, also added more tests.

JukkaL · 2017-09-14T16:16:27Z

Thank you very much for implementing this! Based on a quick pass, it looks like this fixes many long-standing issues in a clean way. Forward reference handling will be also be useful for fine-grained incremental checking, which we are planning to continue working on later this year. Special thanks for writing many test cases.

I'll try to do a full review by mid next week. I'll also run this against internal Dropbox codebases to see if there are regressions (or performance issues).

Also, there's now a merge conflict. Can you fix it?

ilevkivskyi · 2017-09-14T17:00:07Z

Also, there's now a merge conflict. Can you fix it?

Fixed.

Special thanks for writing many test cases.

I am glad you like the tests. But this is a subtle area, if you have ideas for more tests, then I will be grateful (I am sure this PR does not fix all issues with forward references, for example across files as I mentioned above, but this could be a good start).

JukkaL · 2017-09-19T16:50:59Z

I ran this against internal Dropbox repos and found one minor difference. Previously this code didn't generate an error:

from typing import AnyStr, Dict

def f():   # Note no annotation
    x = {}  # type: Dict[str, AnyStr]

Now it generates this error:

t.py:4: error: Invalid type "typing.AnyStr"

Also I there seems to be no significant performance impact, which is good.

I'll continue tomorrow with a more detailed review.

…shes

ilevkivskyi · 2017-09-20T23:22:30Z

@JukkaL I started addressing your comments, I will try to finish before the end of this week.

While making changes you requested, I remembered something important I found before: third pass calls is_subtype and is_same_types. I think this is not the right place to do this, and may be dangerous. Currently I overcome this problem with a hack, but I think we should move these calls to a later stage (either callbacks or type checker) in this PR or in a separate PR.

…iases

…ly lookup functions

ethanhs · 2017-09-22T00:19:14Z

Looks like travis flaked. I restarted that job.

ilevkivskyi · 2017-09-22T23:31:53Z

@JukkaL I think I have now addressed all your comments. This is now ready for further review.

JukkaL

Thanks for the updates! This seems pretty close to ready. I left a bunch of minor comments.

JukkaL · 2017-09-25T16:28:16Z

test-data/unit/check-namedtuple.test

+class G(Generic[T]):
+    x: T
+
+yb: G[int]


Shouldn't this be rejected, since int is not compatible with M?

JukkaL · 2017-09-25T16:29:34Z

test-data/unit/check-namedtuple.test

+yb: G[int]
+yg: G[M]
+z: int = G[M]().x.x
+z = G[M]().x[0]


reveal_type would be better here as well.

JukkaL · 2017-09-25T16:29:53Z

test-data/unit/check-statements.test

+lst: List[N]
+
+for i in lst: # type: N
+    a: int = i.x


Again, prefer reveal_type.

JukkaL · 2017-09-25T16:30:26Z

test-data/unit/check-statements.test

+cm: ContextManager[N]
+
+with cm as g:  # type: N
+    a: str = g['x']


reveal_type is better here as well.

JukkaL · 2017-09-25T16:30:43Z

test-data/unit/check-typeddict.test

+class G(Generic[T]):
+    x: T
+
+yb: G[int]


Similar to above, shouldn't this be rejected?

This is rejected, here and above. I see the problem, it looks like you reviewed only a commit range, not all changes in PR.

I will anyway go though all comments this evening. Thanks for review!

JukkaL · 2017-09-25T16:32:30Z

mypy/semanal.py

@@ -4275,6 +4291,15 @@ def perform_transform(self, node: Union[Node, SymbolTableNode],
                    new_bases.append(alt_base)
            node.bases = new_bases

+    def transform_types(self, lvalue: Lvalue, transform: Callable[[Type], Type]) -> None:


The name of the method would be more informative as transform_types_in_lvalue or similar.

JukkaL · 2017-09-25T16:43:11Z

mypy/typeanal.py

+                bound = tvar.upper_bound
+                if isinstance(bound, ForwardRef):
+                    bound = bound.link
+                if isinstance(bound, Instance) and bound.type.replaced:


I wonder if this code is almost duplicated somewhere else. If so, it would be better to have only one implementation in a utility function/method.

I will minimize duplication (but I can't use exactly the code from visitor in semanal.py it is too soon here).

JukkaL · 2017-09-25T16:43:58Z

test-data/unit/check-namedtuple.test

@@ -440,7 +440,7 @@ T = TypeVar('T', bound='M')
 class G(Generic[T]):
    x: T

-yb: G[int]
+yb: G[int] # E: Type argument "builtins.int" of "G" must be a subtype of "Tuple[builtins.int, fallback=__main__.M]"
 yg: G[M]
 z: int = G[M]().x.x


Using reveal_type would be more reliable as it would catch unwanted Any types.

JukkaL · 2017-09-25T16:45:38Z

mypy/semanal.py

@@ -4349,8 +4355,14 @@ def analyze_types(self, types: List[Type], node: Node) -> None:
        # Similar to above but for nodes with multiple types.
        indicator = {}  # type: Dict[str, bool]
        for type in types:
-            analyzer = TypeAnalyserPass3(self.fail, self.options, self.is_typeshed_file,
-                                         self.sem, indicator)
+            analyzer = TypeAnalyserPass3(self.sem.lookup_qualified,


Could you combine the two places where we generate TypeAnalyserPass3 as there seems to be duplication?

JukkaL · 2017-09-26T15:30:19Z

mypy/types.py

+        x: A
+        A = TypedDict('A', {'x': int})
+
+    To avoid false positives and crashes in such situations, we first wrap the second


Should this be 'the first occurrence ...'?

Yes, will fix this docstring.

JukkaL · 2017-09-26T15:42:43Z

Most of my comments from the last round are shown as outdated because I screwed up things a little. I think that they should mostly be still relevant though.

ilevkivskyi · 2017-09-26T15:48:22Z

Most of my comments from the last round are shown as outdated because I screwed up things a little. I think that they should mostly be still relevant though.

By last round you mean few minutes ago ago or few days ago? I see almost everything as outdated, I assume you mean few minutes ago, I received an e-mail from GitHub, so that I can see these comments.

ilevkivskyi · 2017-09-27T00:56:40Z

@JukkaL I think I now implemented all your recent comments. In addition I fixed one more minor crash on MRO being None for NewTypes and improved tests. This includes:

Added more reveal_types here and there.
Used shorted and/or less distracting names.
Fixed formatting to make tests more compact.

JukkaL · 2017-09-27T17:59:31Z

Thanks for the updates! Glad to see so many bugs fixed.

ilevkivskyi added 20 commits September 1, 2017 00:00

Add basic tests, more details will be added when they will not crash

45e5931

Correct tests

cb4caa5

Implement ForwardRef type, wrap UnboundType, pass SecondPass to third…

1cdc980

… one

Add ForwardRefRemover

260ef02

Add elimination patches

a58a217

Fix replacement logic; fix newtype error formatting

950a022

Fix third pass (need to go deeper)

411b24d

Implement syntethic replacer

b9b8528

Need to go deeper (as usual)

48d6de4

Fix postponed fallback join

ec45441

Simplify some code and add annotations

ac32ed4

Simplify traversal logic; add loads of tests

3fb3019

Take care about one more special case; add few tests and dcostrings

f9b1320

Unify visitors

cf014b8

Add some more comments and docstrings

665236b

Add recursive type warnings

9a318aa

Fix lint

757fbd9

Also clean-up bases; add more tests and allow some previously skipped

4502ce2

One more TypedDict test

3b39d40

Add another simple self-referrential NamedTuple test

c8b28fe

ilevkivskyi changed the title ~~Fix crashes and fails in forward references~~ [WIP] Fix crashes and fails in forward references Sep 13, 2017

Fix type_override; add tests for recursive aliases; fix Callable TODO…

9f92b0f

…; Better error in case something is still missing

Merge branch 'master' into fix-synthetic-crashes

9779103

Merge remote-tracking branch 'upstream/master' into fix-synthetic-cra…

b914bdb

…shes

Add support for generic types with forward references

13c7176

Prohibit forward refs to type vars and subscripted forward refs to al…

79b10d6

…iases

ilevkivskyi mentioned this pull request Sep 21, 2017

Do not call is_subtype and is_same_type in third pass of semantic analysis #3977

Closed

Refactor code to avoid passing semantic analyzer to type analyzer, on…

321a809

…ly lookup functions

ilevkivskyi added 2 commits September 23, 2017 00:47

Address the rest of the review comments

076c909

Improve two tests

c1a63ec

ethanhs mentioned this pull request Sep 23, 2017

Complex Forward-reference NamedTuples cause crash #3990

Closed

Add one more test as suggested in python#3990

97e6f47

gvanrossum mentioned this pull request Sep 25, 2017

Release 0.530 planning #4009

Closed

5 tasks

JukkaL reviewed Sep 26, 2017

View reviewed changes

ilevkivskyi mentioned this pull request Sep 26, 2017

Minor updates to protocol semantics #3996

Merged

ilevkivskyi added 3 commits September 27, 2017 00:13

Address latest review comments

8f52654

Improve tests; Fix one more crash on NewType MRO

6edd078

Fix formatting in tests

514b8bd

JukkaL merged commit a611b11 into python:master Sep 27, 2017

ilevkivskyi deleted the fix-synthetic-crashes branch September 27, 2017 20:17

This was referenced Oct 10, 2017

Refactoring: Make the state of type forward references explicit #4092

Merged

Have multiple passes by SemanticAnalyzerPass3 to resolve more forward references. #4095

Closed

ilinum mentioned this pull request Nov 8, 2017

Add --warn-unused-strictness-exceptions flag #4225

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix crashes and fails in forward references #3952

Fix crashes and fails in forward references #3952

ilevkivskyi commented Sep 13, 2017 •

edited

Loading

ilevkivskyi commented Sep 13, 2017 •

edited

Loading

ilevkivskyi commented Sep 13, 2017

ilevkivskyi commented Sep 13, 2017

JukkaL commented Sep 14, 2017

ilevkivskyi commented Sep 14, 2017

JukkaL commented Sep 19, 2017

ilevkivskyi commented Sep 20, 2017

ethanhs commented Sep 22, 2017

ilevkivskyi commented Sep 22, 2017

JukkaL left a comment

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

ilevkivskyi Sep 26, 2017

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

ilevkivskyi Sep 26, 2017

JukkaL Sep 25, 2017

JukkaL Sep 25, 2017

JukkaL Sep 26, 2017

ilevkivskyi Sep 26, 2017

JukkaL commented Sep 26, 2017

ilevkivskyi commented Sep 26, 2017

ilevkivskyi commented Sep 27, 2017

JukkaL commented Sep 27, 2017

Fix crashes and fails in forward references #3952

Fix crashes and fails in forward references #3952

Conversation

ilevkivskyi commented Sep 13, 2017 • edited Loading

ilevkivskyi commented Sep 13, 2017 • edited Loading

ilevkivskyi commented Sep 13, 2017

ilevkivskyi commented Sep 13, 2017

JukkaL commented Sep 14, 2017

ilevkivskyi commented Sep 14, 2017

JukkaL commented Sep 19, 2017

ilevkivskyi commented Sep 20, 2017

ethanhs commented Sep 22, 2017

ilevkivskyi commented Sep 22, 2017

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JukkaL commented Sep 26, 2017

ilevkivskyi commented Sep 26, 2017

ilevkivskyi commented Sep 27, 2017

JukkaL commented Sep 27, 2017

ilevkivskyi commented Sep 13, 2017 •

edited

Loading

ilevkivskyi commented Sep 13, 2017 •

edited

Loading