forked from NVIDIA/NeMo
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix tarred dataset len when num shards is not divisible by workers (N…
…VIDIA#4553) * fix tarred dataset len when num shards is not divisible by workers Signed-off-by: Iztok Lebar Bajec <[email protected]> * update error reporting on invalid `shard_strategy` * update NLP/PC tarred dataset docstring * add `shard_strategy` to NLP/PC `@dataclass` * update NLP/PC tarred dataset docstring * add `shard_strategy` to NLP/PC docs * revert test with Dataloader retruning the actual data length * make dataloader return actual num of samples, set `limit_train_baches` on `setup_*` * update `shard_strategy` docstrings Signed-off-by: Iztok Lebar Bajec <[email protected]> * update `tarred_dataset` documentation Signed-off-by: Iztok Lebar Bajec <[email protected]> * fix style * update documentation Signed-off-by: Iztok Lebar Bajec <[email protected]> * updated docstrings Signed-off-by: Iztok Lebar Bajec <[email protected]> Co-authored-by: PeganovAnton <[email protected]> Signed-off-by: Hainan Xu <[email protected]>
- Loading branch information
1 parent
ad0adf5
commit ce96af5
Showing
14 changed files
with
395 additions
and
104 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.