Skip to content

[Typing][A-100] Add type annotations for python/paddle/io/dataloader/worker.py #65645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SigureMo merged 5 commits into from
Jul 4, 2024
Merged

[Typing][A-100] Add type annotations for python/paddle/io/dataloader/worker.py #65645

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a67069b
[Typing][A-100] Add type annotations for python/paddle/io/dataloader/…
ooooo-create Jul 2, 2024
d3aec45
add # type: ignore
ooooo-create Jul 3, 2024
69ba4cf
fix
ooooo-create Jul 3, 2024
9aa8fd6
update
ooooo-create Jul 3, 2024
81a2d70
Merge branch 'develop' into typing/A100
SigureMo Jul 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions python/paddle/io/dataloader/worker.py
View file
Open in desktop
Copy link
Member

@SigureMo SigureMo Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

2024-07-03 00:26:39 ++ python type_checking.py --debug
2024-07-03 00:26:39 ----------------Check results--------------------
2024-07-03 00:26:39 paddle.io.get_worker_info:1
2024-07-03 00:26:39 <string>:18:21: error: "WorkerInfo" has no attribute "num_workers"  [attr-defined]
2024-07-03 00:26:39 <string>:19:25: error: "WorkerInfo" has no attribute "id"  [attr-defined]
2024-07-03 00:26:39 Found 2 errors in 1 file (checked 1 source file)
2024-07-03 00:26:39 
2024-07-03 00:26:39 
2024-07-03 00:26:39 >>> Mistakes found in type checking!
2024-07-03 00:26:39 >>> Please recheck the type annotations. Run `tools/type_checking.py` to check the typing issues:
2024-07-03 00:26:39 > python tools/type_checking.py paddle.io.get_worker_info
2024-07-03 00:26:39 ----------------End of the Check--------------------
2024-07-03 00:26:39 ----------------End of the Check--------------------
2024-07-03 00:26:39 ++ type_checking_error=1

示例代码还是挂了，因为 WorkerInfo 完全是动态属性的

我尝试将 WorkerInfo 变为泛型，但是 Python 尚无一种泛型参数能够承载这样的需求，因此貌似并不能解决这个

现在还是添加下 # type: ignore 吧

Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import annotations

import os
import queue
Expand Down Expand Up @@ -76,7 +77,7 @@ def is_alive(self):
_worker_info = None


def get_worker_info():
def get_worker_info() -> WorkerInfo | None:
"""
Get DataLoader worker process information function, this function is
used to split data copy in worker process for IterableDataset
Expand Down Expand Up @@ -117,8 +118,8 @@ def get_worker_info():
... else:
... per_worker = int(
... math.ceil((self.end - self.start) / float(
... worker_info.num_workers)))
... worker_id = worker_info.id
... worker_info.num_workers))) # type: ignore[attr-defined]
... worker_id = worker_info.id # type: ignore[attr-defined]
... iter_start = self.start + worker_id * per_worker
... iter_end = min(iter_start + per_worker, self.end)
...
Expand Down