Fix mypy cache of WithAnnotation types

[kalekseev]

• Refs mypy crashes with TypeError: Object of type AnyType is not JSON serializable #700

This is a result of my research of that issue, the fix works for my projects but introduces other problems.

The fix is based on assumption that metadata may consist only of primitives, the fastest way to serialize dict of object is to use pickle, so I just pickle/unpickle metadata on set/get.
After that fix another problem was found, mypy doesn't support lookups of names with generics:

mypy/lookup.py", line 47, in lookup_fully_qualified
    assert key in names, "Cannot find component %r for %r" % (key, name)
AssertionError: Cannot find component 'WithAnnotations[project' for "django_stubs_ext.WithAnnotations[project.Model, TypedDict({'remind_after': Any, 'remind_before': Any})]"

The reason is https://github.com/python/mypy/blob/d1d431630d967923a9562b4dff436a509c2acb8b/mypy/lookup.py#L31 splits by '.' inside generic [...]. See how it complaints "Cannot find component 'WithAnnotations[project" while it should search for WithAnnotations[project.Model, TypedDict({'remind_after': Any, 'remind_before': Any})].
That problem was solved by replacing dots with __ but I believe that will prevent users to write their own WithAnnotations typings...

Having said that I don't expect that PR to be merged in current state.

[sobolevn]

Thanks a lot for the research! I will also try to see what we can do here.

[syastrov]

It seems like my hack with making type names include [ and . is causing issues. I think we can try to use _ instead and live with the fact that error messages don't 100% match the type the user must use.
Otherwise, we could try to customize the "repr" used in error messages, using some hack. Similar to:
#681

Unfortunately I don't have time to look at this at the moment.

[AustinScola]

Is there any update here or a plan to fix this? The WithAnnotation causes mypy to crash so often that I have resorted to clearing the entire cache first which results in really slow type checking. I would be willing to fix bug if someone could point me in the right direction.

[kalekseev]

It seems like my hack with making type names include [ and . is causing issues. I think we can try to use _ instead and live with the fact that error messages don't 100% match the type the user must use.

That was implemented in that PR, I rebased it.

[sobolevn]

Looks like a solid hack 😆
Thanks, let's try this out.

[AustinScola]

I'm still having problems with the caching of WithAnnotations on version 1.10.1

[AustinScola]

Here is a slightly redacted trace if it helps:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/austin.s/src/xxx/.venvs/type_check/lib/python3.9/site-packages/mypy/__main__.py", line 34, in <module>
    console_entry()
  File "/Users/austin.s/src/xxx/.venvs/type_check/lib/python3.9/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 96, in main
  File "mypy/main.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 256, in _build
  File "mypy/build.py", line 2727, in dispatch
  File "mypy/build.py", line 3068, in process_graph
  File "mypy/build.py", line 3146, in process_fresh_modules
  File "mypy/build.py", line 2006, in fix_cross_refs
  File "mypy/fixup.py", line 25, in fixup_module
  File "mypy/fixup.py", line 92, in visit_symbol_table
  File "mypy/nodes.py", line 3058, in accept
  File "mypy/fixup.py", line 141, in visit_type_alias
  File "mypy/types.py", line 1065, in accept
  File "mypy/fixup.py", line 155, in visit_instance
  File "mypy/fixup.py", line 272, in lookup_fully_qualified_typeinfo
  File "mypy/lookup.py", line 47, in lookup_fully_qualified
AssertionError: Cannot find component "WithAnnotations[xxx__models__Report, TypedDict('xxx" for "django_stubs_ext.WithAnnotations[xxx__models__Report, TypedDict('xxx.api.report.RelatedInfo', {'response_count': builtins.int, 'viewed': builtins.bool, 'preview': Union[builtins.str, None]})]"

[ashkop]

I'm also still having this issue with on version 1.13.0