Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Static checking with mypy #621

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

WIP: Static checking with mypy #621

wants to merge 2 commits into from

Conversation

ghisvail
Copy link
Collaborator

Work in progress

See methodology here.

@ghisvail
Copy link
Collaborator Author

ghisvail commented Feb 27, 2023 •
edited
Loading

First run:

$ python -m mypy pydra
pydra/utils/messenger.py:185: error: Skipping analyzing "pyld": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/helpers.py:5: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/core.py:13: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/core.py:161: error: Argument 1 to "Path" has incompatible type "Union[str, File]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/core.py:162: error: Argument 1 to "Path" has incompatible type "Union[str, File]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/core.py:169: error: Argument after ** must be a mapping, not "Union[str, File, Dict[Any, Any], None]"  [arg-type]
pydra/engine/core.py:174: error: Need type annotation for "inp_lf" (hint: "inp_lf: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:179: error: Need type annotation for "_inner_cont_dim" (hint: "_inner_cont_dim: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:180: error: Need type annotation for "_output" (hint: "_output: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:181: error: Need type annotation for "_result" (hint: "_result: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:185: error: Incompatible types in assignment (expression has type "Dict[<nothing>, <nothing>]", variable has type "None")  [assignment]
pydra/engine/core.py:903: error: Argument 1 to "_sanitize_input_spec" has incompatible type "Union[List[str], SpecInfo, None]"; expected "Union[SpecInfo, List[str]]"  [arg-type]
pydra/engine/core.py:926: error: Need type annotation for "name2obj" (hint: "name2obj: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/workers.py:860: error: Cannot find implementation or library stub for module named "dask.distributed"  [import]
pydra/engine/task.py:44: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/task.py:141: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, Any]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/task.py:141: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/task.py:141: note: Consider using "Sequence" instead, which is covariant
pydra/engine/task.py:143: error: Item "BaseSpec" of "Union[SpecInfo, BaseSpec]" has no attribute "fields"  [union-attr]
pydra/engine/task.py:184: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, Any]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/task.py:184: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/task.py:184: note: Consider using "Sequence" instead, which is covariant
pydra/engine/task.py:297: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/task.py:594: error: Need type annotation for "bindings" (hint: "bindings: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/__init__.py:26: error: Skipping analyzing "etelemetry": module is installed, but missing library stubs or py.typed marker  [import]
pydra/__init__.py:26: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
pydra/utils/tests/test_messenger.py:52: error: Skipping analyzing "pyld": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/boutiques.py:95: error: Skipping analyzing "boutiques.searcher": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/utils.py:16: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]
pydra/engine/tests/utils.py:164: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/utils.py:164: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/utils.py:197: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:199: error: Incompatible return value type (got "Path", expected "File")  [return-value]
pydra/engine/tests/utils.py:199: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:205: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:207: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:217: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:219: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:221: error: Incompatible types in assignment (expression has type "int", target has type "Path")  [assignment]
pydra/engine/tests/utils.py:227: error: No overload variant of "open" matches argument type "File"  [call-overload]
pydra/engine/tests/utils.py:227: note: Possible overload variants:
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['r+', '+r', 'rt+', 'r+t', '+rt', 'tr+', 't+r', '+tr', 'w+', '+w', 'wt+', 'w+t', '+wt', 'tw+', 't+w', '+tw', 'a+', '+a', 'at+', 'a+t', '+at', 'ta+', 't+a', '+ta', 'x+', '+x', 'xt+', 'x+t', '+xt', 'tx+', 't+x', '+tx'], Literal['w', 'wt', 'tw', 'a', 'at', 'ta', 'x', 'xt', 'tx'], Literal['r', 'rt', 'tr', 'U', 'rU', 'Ur', 'rtU', 'rUt', 'Urt', 'trU', 'tUr', 'Utr']] = ..., buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> TextIOWrapper
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: Literal[0], encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> FileIO
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedRandom
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedWriter
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedReader
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: int = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BinaryIO
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: str, buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> IO[Any]
pydra/engine/tests/utils.py:236: error: No overload variant of "open" matches argument type "File"  [call-overload]
pydra/engine/tests/utils.py:236: note: Possible overload variants:
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['r+', '+r', 'rt+', 'r+t', '+rt', 'tr+', 't+r', '+tr', 'w+', '+w', 'wt+', 'w+t', '+wt', 'tw+', 't+w', '+tw', 'a+', '+a', 'at+', 'a+t', '+at', 'ta+', 't+a', '+ta', 'x+', '+x', 'xt+', 'x+t', '+xt', 'tx+', 't+x', '+tx'], Literal['w', 'wt', 'tw', 'a', 'at', 'ta', 'x', 'xt', 'tx'], Literal['r', 'rt', 'tr', 'U', 'rU', 'Ur', 'rtU', 'rUt', 'Urt', 'trU', 'tUr', 'Utr']] = ..., buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> TextIOWrapper
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: Literal[0], encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> FileIO
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedRandom
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedWriter
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedReader
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: int = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BinaryIO
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: str, buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> IO[Any]
pydra/engine/tests/test_tasks_files.py:20: error: No overload variant of "listdir" matches argument type "Directory"  [call-overload]
pydra/engine/tests/test_tasks_files.py:20: note: Possible overload variants:
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: Optional[Union[str, PathLike[str]]] = ...) -> List[str]
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: Union[bytes, PathLike[bytes]]) -> List[bytes]
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: int) -> List[str]
pydra/engine/tests/test_tasks_files.py:45: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/test_tasks_files.py:45: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/test_tasks_files.py:46: error: Argument 1 to "load" has incompatible type "File"; expected "Union[str, bytes, PathLike[Any], _SupportsReadSeek[bytes]]"  [arg-type]
pydra/engine/tests/test_tasks_files.py:56: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/test_tasks_files.py:56: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/test_tasks_files.py:57: error: Argument 1 to "load" has incompatible type "File"; expected "Union[str, bytes, PathLike[Any], _SupportsReadSeek[bytes]]"  [arg-type]
pydra/engine/tests/test_singularity.py:13: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]
pydra/engine/tests/test_profiles.py:6: error: Skipping analyzing "pympler": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_helpers_file.py:258: error: Need type annotation for "MOUNT_OUTPUTS"  [var-annotated]
pydra/engine/tests/test_task.py:5: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_task.py:326: error: No overload variant of "sum" matches argument type "Union[float, int]"  [call-overload]
pydra/engine/tests/test_task.py:326: note: Possible overload variants:
pydra/engine/tests/test_task.py:326: note:     def sum(Iterable[bool], /, start: int = ...) -> int
pydra/engine/tests/test_task.py:326: note:     def [_SupportsSumNoDefaultT <: _SupportsSumWithNoDefaultGiven] sum(Iterable[_SupportsSumNoDefaultT], /) -> Union[_SupportsSumNoDefaultT, Literal[0]]
pydra/engine/tests/test_task.py:326: note:     def [_AddableT1 <: SupportsAdd[Any, Any], _AddableT2 <: SupportsAdd[Any, Any]] sum(Iterable[_AddableT1], /, start: _AddableT2) -> Union[_AddableT1, _AddableT2]
pydra/engine/tests/test_task.py:339: error: Argument 1 to "sum" has incompatible type "Union[str, int]"; expected "Iterable[bool]"  [arg-type]
pydra/engine/tests/test_task.py:379: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_task.py:394: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_task.py:410: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_shelltask_inputspec.py:2172: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, type, Dict[str, object]]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/tests/test_shelltask_inputspec.py:2172: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/tests/test_shelltask_inputspec.py:2172: note: Consider using "Sequence" instead, which is covariant
pydra/engine/tests/test_shelltask_inputspec.py:2173: error: Need type annotation for "task_output_fields" (hint: "task_output_fields: List[<type>] = ...")  [var-annotated]
pydra/engine/tests/test_shelltask_inputspec.py:2175: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/tests/test_nipype1_convert.py:18: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/tests/test_helpers.py:8: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_boutiques.py:15: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]

@codecov
Copy link

codecov bot commented Feb 27, 2023 •
edited
Loading

Codecov Report

Base: 81.17% // Head: 80.49% // Decreases project coverage by -0.69% ⚠️

Coverage data is based on head (9bcd0dc) compared to base (b5fe4c0).
Patch has no changes to coverable lines.

❗ Current head 9bcd0dc differs from pull request most recent head d4dc196. Consider uploading reports for the commit d4dc196 to get more accurate results

Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #621      +/-   ##
==========================================
- Coverage   81.17%   80.49%   -0.69%     
==========================================
  Files          20       20              
  Lines        4393     4393              
  Branches     1264        0    -1264     
==========================================
- Hits         3566     3536      -30     
- Misses        823      857      +34     
+ Partials        4        0       -4
Flag Coverage Δ
unittests 80.49% <ø> (-0.69%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pydra/engine/task.py 88.07% <0.00%> (-5.51%) ⬇️
pydra/engine/helpers.py 84.39% <0.00%> (-1.66%) ⬇️
pydra/engine/helpers_file.py 84.89% <0.00%> (-1.21%) ⬇️
pydra/engine/core.py 92.96% <0.00%> (-0.16%) ⬇️
pydra/engine/workers.py 18.76% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@effigies effigies mentioned this pull request May 3, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ghisvail