Add jupyter notebook tutorial for single node mulilingual dataset - #30
Merged
Conversation
nicoleeeluo
marked this pull request as ready for review
April 11, 2024 13:06
Contributor
|
@ryantwolf / @Maghoumi can we review and get this merged? |
Contributor
|
pinging again, @Maghoumi can you help review this - this will help teams trying to create multilingual datasets with Curator! |
Contributor
|
Thanks for opening the PR @nicoleeeluo All our PR's require commits to be verified (signed) and signed off. |
Contributor
Author
|
@ayushdg Thanks for reminding! |
ryantwolf
reviewed
May 15, 2024
ryantwolf
left a comment
ryantwolf
left a comment
Contributor
There was a problem hiding this comment.
Thanks so much for creating this tutorial! It's quite extensive which is super helpful. There are a couple of changes I'd appreciate if you could make, please let me know if you have any concerns or disagree with my requests. Thanks again.
| "def pre_imports():\n", | ||
| " import cudf \n", | ||
| "\n", | ||
| "def load_dataset(input_data_dir, file_type='jsonl'):\n", |
Contributor
There was a problem hiding this comment.
Do you think you could use the DocumentDataset.read_json and DocumentDataset.read_parquet methods we have added recently? Let me know if something about them would prevent you from using them.
| "source": [ | ||
| "## 2.Language separation and unicode fixing\n", | ||
| "\n", | ||
| "**Note**: In order to be run on interactive python. Please comment `from.code import *` and the related imports in `./nemo_curator/filters/__init__.py`" |
Contributor
There was a problem hiding this comment.
Can you try this again with the latest version of curator? And please let us know what errors you get if you get any.
Contributor
Author
There was a problem hiding this comment.
This is fixed. I will amend the notebook accordingly
| "TH wikipedia data do have `id` field, but the `id` field contains number only. It will be better if we unified the `id` field and transform it to the format of `<prefix>_<id>`. In this way, when handling multiple dataset, we will able to know which document from which dataset has been removed. This `id` will be useful when we are running deduplication and heuristic filtering. The function we will be using is `AddID()`. Arguments for this function include:\n", | ||
| "- `id_field`: fields will be added to input .json file. If the key already exists in the .jsonl, it's value will be replaced.\n", | ||
| "- `id_prefix`: prefix used in ID. Default is 'doc-id'\n", | ||
| "- `start_index`: starting index in ID. Default is 0" |
Contributor
There was a problem hiding this comment.
This recently changed. The default is now None and the id is considered "unordered" by default to improve speed.
Contributor
Author
There was a problem hiding this comment.
Fixed the description. In the code section, I keep the start_index = 0 for easier reference
| " 2. Fuzzy deduplication\n", | ||
| "4. Heuristic filtering\n", | ||
| "\n", | ||
| "What is not included:\n", |
Contributor
There was a problem hiding this comment.
Probably also worth mentioning that this also doesn't include
- Distributed data classification with PyTorch models
- Personal identifiable information (PII) redaction
Contributor
Author
|
@ryantwolf Hi Ryan, I have pushed a new version to include the fixes you mentioned. |
ryantwolf
reviewed
May 17, 2024
ryantwolf
left a comment
ryantwolf
left a comment
Contributor
There was a problem hiding this comment.
Perfect! I have one last comment about the docker image you refer too, but other than that it looks great!
Though, as Ayush mentioned all our PR's require commits to be verified (signed) and signed off. Quoting from him earlier:
To achieve this you need to commit with the -sS flags. (more info in the Contributing Guide.
More details on signing commits can be found on the guide: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits
Also it looks like the style guide is failing for the config file you made. You should be able to fix it by running pip install pre-commit && pre-commit install && pre-commit run --all. Thanks again!
| " Password: <Your NGC Key>\n", | ||
| "- Get NeMo NeMo Framework Training Container\n", | ||
| " ```bash\n", | ||
| " docker pull nvcr.io/ea-bignlp/ga-participants/nemofw-training:24.01\n" |
Contributor
There was a problem hiding this comment.
I'd either change this to the dev container or the 24.05 tag (which hasn't been released yet).
docker pull nvcr.io/nvidia/nemo:dev.framework
docker pull nvcr.io/nvidia/nemo:24.05.framework
Since the other container versions don't have the latest version of NeMo Curator that the tutorial uses.
Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Fix metadata inference with pandas and dask Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix datatypes for task decontamination Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Use targetted import Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Move tokenizer import Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Reduce inductor threads Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change env int to string Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change location of env var Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add comment linking issue Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Add fast id method Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add type conversion Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix off by one errors in tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Move GPU imports and make them optional Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move gpu dependencies to a seperate install Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove unused import Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Switch to placeholder import that raises on usage Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove deprecated utils usage Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add cuML attribution Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Safe import tests, improve install instruction, update gha workflow Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix pytests due to loc bug Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * update install instructions Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Raise on non module-not-found errors, update logging Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update logging to not change root logger Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* [K8s]: Adds a helper script to create a dask cluster on k8s and includes instructions for how to a Curator workload on k8s Signed-off-by: Terry Kong <terryk@nvidia.com> * black formatting Signed-off-by: Terry Kong <terryk@nvidia.com> * big_english -> my_dataset Signed-off-by: Terry Kong <terryk@nvidia.com> * 24.01 -> 24.03 default container Signed-off-by: Terry Kong <terryk@nvidia.com> * Add help kwarg to all flags Signed-off-by: Terry Kong <terryk@nvidia.com> * Clarify why venv is needed Signed-off-by: Terry Kong <terryk@nvidia.com> * fix precommit failures Signed-off-by: Terry Kong <terryk@nvidia.com> --------- Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Refactor common utils and remove unused code Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * More cleanup Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * More updates/shuffling Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move gpu_dedup scripts into subfolder Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove gpu_deduplication subfolder Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add readme to fuzzy dedup scripts section Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix typo and relative links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove legacy script entrypoints Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove legacy scripts and add init file Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update GpuDeduplication.rst Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Fix lang id example Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add classifier unit tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add test for failure Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove failure test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Add initial dataset blending function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blend unit tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add self parameter Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix return type of blend dataset Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix blending tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change assert statement for very uneven blend Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix key error Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add proper proportion blending test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add four dataset blend and clarify docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add shuffle module Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blend example and tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix random method name Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Wrap return type in DocumentDataset Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Save result of column drop Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change equality check for shuffle tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix expected order after shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add more documents to shuffle test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add assert statement Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add within partition shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Refactor add rand column for shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix filename tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add determinism handling for shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change numpy random function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix tests with new random method Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove length call from blending Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Improve scaling of blending function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix blend tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blending script Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add additional file paths call Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add documentation Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Reformat docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove backticks Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add context manager for shuffle tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add better deterministic shuffle path Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Update documentation and reset index Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Initial pass at fuzzy dedup api Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update deprecated shuffle arg Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * dask_cuda gpu only import Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move fuzzy_dedup imports to optional Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * more tests Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move FuzzyDeDupConfig to it's own class Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add example script and config file, fix typo Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove slurm examples for gpu dedup Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add config module Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Rename FuzzyDeDupConfig and minhash_length to FuzzyDuplicatesConfig, num_hashes Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add comments and update example Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Write to same format as input in fuzzy dedup example Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Fix pii index issue Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add sequential wrapper Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix pii tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
…riting speed (NVIDIA-NeMo#57) This commit fixes issue NVIDIA-NeMo#43 (empty files created when invoking reshard_jsonl method at nemo_curator.utils.file_utils.py) by double-checking the files size after being generated, and deleting them with size zero. In addition to that, I have noticed there is no need to parse to JSON object the content of the different lines, which should be already in json format. By removing that extra-parsing, there is a significant speed up in the execution of this method. Signed-off-by: Miguel Martínez <26169771+miguelusque@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
This PR adds a new tutorial to demonstrate data curation for PEFT use-cases. Signed-off-by: Mehran Maghoumi <Maghoumi@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
* Move PII constants to a seperate file that does not import presidio/spacy and other GPU dependencies Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add comment around import, move constant import to global scope Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: Nicoel Luo <nluo@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com>
…zy deduplication wrapper example Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: Nicole Luo <nluo@nvidia.com>
Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com>
Signed-off-by: Nicole Luo <nluo@nvidia.com>
Contributor
Author
|
@ryantwolf Hi Ryan, I have fixed the commits accordingly. Would you help to review and see if there is any issue? Thank you! |
ryantwolf
reviewed
May 23, 2024
| } | ||
| ], | ||
| "source": [ | ||
| "client = get_client(args, args.device)\n", |
Contributor
There was a problem hiding this comment.
Ah we just merged in a change that changes the function signature of this to no longer require an argparse object. It should be easier to use, but it does mean that you need to update it here. Ping me again when this is changed and I'll merge it in ASAP.
Contributor
Author
There was a problem hiding this comment.
Thanks for noting. I have updated accordingly.
…map_bucket section Signed-off-by: Nicole Luo <nluo@nvidia.com>
ryantwolf
approved these changes
May 24, 2024
ryantwolf
left a comment
ryantwolf
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the tutorial! This is super great
jnke2016
pushed a commit
to jnke2016/Curator
that referenced
this pull request
Nov 12, 2025
…IDIA-NeMo#30) * Init commit for tutorial notebook Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fix metadata inference with pandas and dask (NVIDIA-NeMo#35) * Fix metadata inference with pandas and dask Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix datatypes for task decontamination Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Use targetted import Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Disable PyTorch Compile Multiprocessing (NVIDIA-NeMo#34) * Move tokenizer import Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Reduce inductor threads Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change env int to string Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change location of env var Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add comment linking issue Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Improve speed of AddId module (NVIDIA-NeMo#36) * Add fast id method Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add type conversion Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix off by one errors in tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Make GPU dependencies optional (NVIDIA-NeMo#27) * Move GPU imports and make them optional Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move gpu dependencies to a seperate install Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove unused import Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Switch to placeholder import that raises on usage Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove deprecated utils usage Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add cuML attribution Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Safe import tests, improve install instruction, update gha workflow Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix pytests due to loc bug Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * update install instructions Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Raise on non module-not-found errors, update logging Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update logging to not change root logger Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fix failing GPU tests with latest pandas bump (NVIDIA-NeMo#41) Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Adds Nemo Curator K8s example (NVIDIA-NeMo#40) * [K8s]: Adds a helper script to create a dask cluster on k8s and includes instructions for how to a Curator workload on k8s Signed-off-by: Terry Kong <terryk@nvidia.com> * black formatting Signed-off-by: Terry Kong <terryk@nvidia.com> * big_english -> my_dataset Signed-off-by: Terry Kong <terryk@nvidia.com> * 24.01 -> 24.03 default container Signed-off-by: Terry Kong <terryk@nvidia.com> * Add help kwarg to all flags Signed-off-by: Terry Kong <terryk@nvidia.com> * Clarify why venv is needed Signed-off-by: Terry Kong <terryk@nvidia.com> * fix precommit failures Signed-off-by: Terry Kong <terryk@nvidia.com> --------- Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Move common dedup utils and remove unused code (NVIDIA-NeMo#42) * Refactor common utils and remove unused code Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * More cleanup Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * More updates/shuffling Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move gpu_dedup scripts into subfolder Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove gpu_deduplication subfolder Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add readme to fuzzy dedup scripts section Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix typo and relative links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove legacy script entrypoints Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove legacy scripts and add init file Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update GpuDeduplication.rst Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fix lang id example (NVIDIA-NeMo#37) * Fix lang id example Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add classifier unit tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add test for failure Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove failure test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Add dataset blending tool (NVIDIA-NeMo#32) * Add initial dataset blending function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blend unit tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add self parameter Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix return type of blend dataset Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix blending tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change assert statement for very uneven blend Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix key error Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add proper proportion blending test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add four dataset blend and clarify docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add shuffle module Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blend example and tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix random method name Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Wrap return type in DocumentDataset Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Save result of column drop Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change equality check for shuffle tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix expected order after shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add more documents to shuffle test Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add assert statement Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add within partition shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Refactor add rand column for shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix filename tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add determinism handling for shuffle Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Change numpy random function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix tests with new random method Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove length call from blending Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Improve scaling of blending function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix blend tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add blending script Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add additional file paths call Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add documentation Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Reformat docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Remove backticks Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add context manager for shuffle tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add better deterministic shuffle path Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Update documentation and reset index Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * High level fuzzy duplicates module (NVIDIA-NeMo#46) * Initial pass at fuzzy dedup api Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Update deprecated shuffle arg Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * dask_cuda gpu only import Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move fuzzy_dedup imports to optional Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * more tests Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Move FuzzyDeDupConfig to it's own class Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add example script and config file, fix typo Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Remove slurm examples for gpu dedup Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add config module Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Rename FuzzyDeDupConfig and minhash_length to FuzzyDuplicatesConfig, num_hashes Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add comments and update example Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Write to same format as input in fuzzy dedup example Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fix indexing in PII Modifier (NVIDIA-NeMo#55) * Fix pii index issue Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add sequential wrapper Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix pii tests Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Disable string conversion globally (NVIDIA-NeMo#56) Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fix issue NVIDIA-NeMo#43 (empty files creation) and improve reading/writing speed (NVIDIA-NeMo#57) This commit fixes issue NVIDIA-NeMo#43 (empty files created when invoking reshard_jsonl method at nemo_curator.utils.file_utils.py) by double-checking the files size after being generated, and deleting them with size zero. In addition to that, I have noticed there is no need to parse to JSON object the content of the different lines, which should be already in json format. By removing that extra-parsing, there is a significant speed up in the execution of this method. Signed-off-by: Miguel Martínez <26169771+miguelusque@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * [Tutorials] Add a tutorial for PEFT data curation (NVIDIA-NeMo#45) This PR adds a new tutorial to demonstrate data curation for PEFT use-cases. Signed-off-by: Mehran Maghoumi <Maghoumi@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Only import PII constants during Curator import (NVIDIA-NeMo#61) * Move PII constants to a seperate file that does not import presidio/spacy and other GPU dependencies Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add comment around import, move constant import to global scope Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Deleting links Signed-off-by: Nicoel Luo <nluo@nvidia.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fixed typo. Update content to lastest NeMo Curator version. Added fuzzy deduplication wrapper example Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fixing Style Signed-off-by: Nicole Luo <nluo@nvidia.com> * Updating container version Signed-off-by: Nicole Luo <nluo@nvidia.com> * Fixing style Signed-off-by: Nicole Luo <nluo@nvidia.com> * Update get_client() according to latest version; Update log path for map_bucket section Signed-off-by: Nicole Luo <nluo@nvidia.com> --------- Signed-off-by: Nicole Luo <nluo@nvidia.com> Signed-off-by: Ryan Wolf <rywolf@nvidia.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Miguel Martínez <26169771+miguelusque@users.noreply.github.com> Signed-off-by: Mehran Maghoumi <Maghoumi@users.noreply.github.com> Signed-off-by: Nicoel Luo <nluo@nvidia.com> Signed-off-by: nicoleeeluo <157772168+nicoleeeluo@users.noreply.github.com> Co-authored-by: Ryan Wolf <rywolf@nvidia.com> Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Terry Kong <terrycurtiskong@gmail.com> Co-authored-by: Miguel Martínez <26169771+miguelusque@users.noreply.github.com> Co-authored-by: Mehran Maghoumi <Maghoumi@users.noreply.github.com> Co-authored-by: Ryan Wolf <ryantwolf1@gmail.com>
mohammadaaftabv
added a commit
to mohammadaaftabv/Curator
that referenced
this pull request
Jun 11, 2026
…ng, perf per-actor/per-GPU, reader/writer hardening Adapters + inference co-location (review #3/NVIDIA-NeMo#25): - Move nemo_curator/adapters/asr/* -> stages/audio/inference/asr/adapters/* - Move stages/inference/{batch_policy,bucketed_stage}.py -> stages/audio/inference/ - Update all imports, YAML adapter_target / batch_policy._target_, READMEs, tests (tests mirror the new package paths). vLLM teardown (review NVIDIA-NeMo#15): - _cleanup_gpu() no longer calls torch.distributed.destroy_process_group(); vLLM owns its TP group, so tearing down the global PG could corrupt other components. Tutorial backend (review NVIDIA-NeMo#30): - Default backend -> ray_data (YAML + main.py); trim the "Choosing a Backend" section. - Drop tutorial test tests/tutorials/.../test_main_resources.py (review NVIDIA-NeMo#28). Performance reporting: - Unify metrics into a single per_actor block keyed by actor_id; GPU actors embed physical_address (host_ip:indices, now the canonical per-GPU key), gpu_indices/ gpu_uuids, and per-GPU utilization percentiles (nested gpus map). - Add first-class StagePerfStats.invocation_id for exact dedup; __add__ preserves identity only within the same worker. - Add inference_compute_fraction (inference time / total process time). - New GpuUtilSampler (NVML, bounded deque + lock); harden _collect_gpu_uuids to map physical indices -> visible ordinals under CUDA_VISIBLE_DEVICES. - Remove per-shard <output_dir>/<shard_key>_perf.jsonl output. Reader (nemo_tarred_reader): - Collision-safe _ManifestIndex (exact path, then longest path-suffix; ambiguous basenames resolve to no-match). - Enforce max_duration_s against decoded audio duration; narrow decode except to audio errors; write .done for empty/fully-filtered shards. Writer (sharded_manifest_writer): - Batch manifest writes per shard instead of per-utterance open/close. Misc: - Remove redundant super().setup_on_node() in RayActorPoolStageAdapter.__init__ (executor already runs once-per-node setup). - Fix stale docstring (disfluency_text_key); drop bucketize redundancy; trim adapter duplicate volume math; reduce comment verbosity across PR files. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
mohammadaaftabv
added a commit
to mohammadaaftabv/Curator
that referenced
this pull request
Jul 8, 2026
…ng, perf per-actor/per-GPU, reader/writer hardening Adapters + inference co-location (review #3/NVIDIA-NeMo#25): - Move nemo_curator/adapters/asr/* -> stages/audio/inference/asr/adapters/* - Move stages/inference/{batch_policy,bucketed_stage}.py -> stages/audio/inference/ - Update all imports, YAML adapter_target / batch_policy._target_, READMEs, tests (tests mirror the new package paths). vLLM teardown (review NVIDIA-NeMo#15): - _cleanup_gpu() no longer calls torch.distributed.destroy_process_group(); vLLM owns its TP group, so tearing down the global PG could corrupt other components. Tutorial backend (review NVIDIA-NeMo#30): - Default backend -> ray_data (YAML + main.py); trim the "Choosing a Backend" section. - Drop tutorial test tests/tutorials/.../test_main_resources.py (review NVIDIA-NeMo#28). Performance reporting: - Unify metrics into a single per_actor block keyed by actor_id; GPU actors embed physical_address (host_ip:indices, now the canonical per-GPU key), gpu_indices/ gpu_uuids, and per-GPU utilization percentiles (nested gpus map). - Add first-class StagePerfStats.invocation_id for exact dedup; __add__ preserves identity only within the same worker. - Add inference_compute_fraction (inference time / total process time). - New GpuUtilSampler (NVML, bounded deque + lock); harden _collect_gpu_uuids to map physical indices -> visible ordinals under CUDA_VISIBLE_DEVICES. - Remove per-shard <output_dir>/<shard_key>_perf.jsonl output. Reader (nemo_tarred_reader): - Collision-safe _ManifestIndex (exact path, then longest path-suffix; ambiguous basenames resolve to no-match). - Enforce max_duration_s against decoded audio duration; narrow decode except to audio errors; write .done for empty/fully-filtered shards. Writer (sharded_manifest_writer): - Batch manifest writes per shard instead of per-utterance open/close. Misc: - Remove redundant super().setup_on_node() in RayActorPoolStageAdapter.__init__ (executor already runs once-per-node setup). - Fix stale docstring (disfluency_text_key); drop bucketize redundancy; trim adapter duplicate volume math; reduce comment verbosity across PR files. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a jupyter notebook workflow for a sample curation pipeline for Thai Wikipedia data.
Modules included in this workflows are