Skip to content

Deprecate MapReduceMixin and implements its methods in BaseChecker#6383

Merged
DanielNoord merged 7 commits intopylint-dev:mainfrom
DanielNoord:map_reduce
Apr 19, 2022
Merged

Deprecate MapReduceMixin and implements its methods in BaseChecker#6383
DanielNoord merged 7 commits intopylint-dev:mainfrom
DanielNoord:map_reduce

Conversation

@DanielNoord
Copy link
Copy Markdown
Collaborator

  • Write a good description on what the PR does.
  • If you used multiple emails or multiple names when contributing, add your mails
    and preferred name in script/.contributors_aliases.json

Type of Changes

Type
✨ New feature
🔨 Refactoring

Description

Kinda high priority since main is broken due to two typing PRs apparently depending on each other but passing on their own.

main complains about checker.get_map_data() and checker.reduce_map_data in parallel.py. Instead of adding ignores there I think mypy is actually right and warns of a design flaw. There is no need to separate these MixIns (ideally we would deprecate the MixIn as there is little overhead of making BaseChecker a MapReduceMixin and just having the methods do nothing.
That's why I propose this solution.

@DanielNoord DanielNoord added Maintenance Discussion or action around maintaining pylint or the dev workflow High priority Issue with more than 10 reactions labels Apr 18, 2022
@DanielNoord DanielNoord added this to the 2.14.0 milestone Apr 18, 2022
@coveralls
Copy link
Copy Markdown

coveralls commented Apr 18, 2022

Pull Request Test Coverage Report for Build 2190395707

  • 17 of 18 (94.44%) changed or added relevant lines in 4 files are covered.
  • 11 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.003%) to 95.148%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pylint/checkers/base_checker.py 3 4 75.0%
Files with Coverage Reduction New Missed Lines %
pylint/checkers/stdlib.py 11 94.53%
Totals Coverage Status
Change from base Build 2189917137: -0.003%
Covered Lines: 15785
Relevant Lines: 16590

💛 - Coveralls

Copy link
Copy Markdown
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this imply that BaseChecker is able to perform map reduce when it's not ?

@DanielNoord
Copy link
Copy Markdown
Collaborator Author

Won't this imply that BaseChecker is able to perform map reduce when it's not ?

Yes and no I guess. It is able to perform it in so far as that it has a reduce_map_data method. However, that simply does nothing. The way the MixIn works is that get_map_data should return something (Any) which then gets appended to a list specific to that checker. When reducing all data the respective reduce_map_data method is then called to reduce the specific from of data that get_map_data returns.
So, you will never be able to get this to work with only implementing get_map_data and I think you will always catch this when writing a test for this.


The issue with the current implementation is that there is a dictionary mapping checker to data. Only checkers that implement get_map_data are added to this dict. However, when we interact with this dict later on mypy has no way of knowing that implementing get_map_data is a requirement for having an entry in this dictionary and just considers all entries to be BaseChecker, and then complains.

Comment thread pylint/lint/parallel.py
@DanielNoord DanielNoord removed the High priority Issue with more than 10 reactions label Apr 19, 2022
Copy link
Copy Markdown
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM the final result is pretty nice and reasonable 👍

Copy link
Copy Markdown
Member

@jacobtylerwalls jacobtylerwalls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some cleanups.

Comment thread ChangeLog Outdated
Comment thread doc/how_tos/custom_checkers.rst Outdated
Comment thread doc/how_tos/custom_checkers.rst Outdated
Comment thread doc/whatsnew/2.14.rst Outdated
Comment thread doc/how_tos/custom_checkers.rst Outdated
Co-authored-by: Jacob Walls <jacobtylerwalls@gmail.com>
@DanielNoord
Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review guys! Agree with @Pierre-Sassoulas!

Copy link
Copy Markdown
Member

@jacobtylerwalls jacobtylerwalls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Terrific, thank you!

@jacobtylerwalls
Copy link
Copy Markdown
Member

(Maybe a better PR title, though... now that BaseChecker isn't a MapReduceMixin.)

@DanielNoord DanielNoord merged commit 5ee5cf7 into pylint-dev:main Apr 19, 2022
@DanielNoord DanielNoord deleted the map_reduce branch April 19, 2022 15:38
@DanielNoord DanielNoord changed the title Make BaseChecker a MapReduceMixin Deprecate MapReduceMixin and implements its methods in BaseChecker Apr 19, 2022
omarandlorraine pushed a commit to omarandlorraine/pylint that referenced this pull request Apr 19, 2022
…ker`` (pylint-dev#6383)

Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>
Co-authored-by: Jacob Walls <jacobtylerwalls@gmail.com>
@doublethefish
Copy link
Copy Markdown
Contributor

I completely missed this change, but it's great to see. When I added the Mixin object the intent was to have as little impact over the core code as possible, it's always been clear that the right intemediary place for it was in the base-checker - longer term the map-reduce functionality should completely go in favour of a fully parallelised astroid or moved into astroid
or something.

I think mypy is actually right and warns of a design flaw

Mypy is definitely correct that there's an issue, but it's not a warning about a design flaw. Mypy is just showing that there's a lack of type checking and/or explicit error handling around whether the mix-in has been applied or not. After all python isn't C++ and is a dynamic language which allows for that sort of thing more easily :) IIRC the design implicitly ensures that the mix-in is applied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Maintenance Discussion or action around maintaining pylint or the dev workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants