diff --git a/tutorials/text/README.md b/tutorials/text/README.md index 6e2bb8ed99..bbe7b3e3f2 100644 --- a/tutorials/text/README.md +++ b/tutorials/text/README.md @@ -11,11 +11,13 @@ Hands-on tutorials for curating text data with NeMo Curator. Complete working ex | Tutorial | Description | Files | |----------|-------------|-------| | **[Download & Extract](download-and-extract/)** | Data acquisition workflows | `download_extract_tutorial.ipynb` | -| **[Deduplication](deduplication/)** | Remove duplicate content | `semantic_e2e.ipynb`, `semantic_step_by_step.ipynb` | -| **[Classification](distributed-data-classification/)** | Quality assessment and categorization | `quality-classification.ipynb`, `domain-classification.ipynb`, `aegis-classification.ipynb`, `fineweb-edu-classification.ipynb` | +| **[Deduplication](deduplication/)** | Remove duplicate content | Fuzzy and semantic deduplication notebooks | +| **[Classification](distributed-data-classification/)** | Quality assessment and categorization | `quality-classification.ipynb`, `domain-classification.ipynb`, `fineweb-edu-classification.ipynb`, and more | | **[PEFT Curation](peft-curation/)** | Instruction-tuning data preparation | `main.py`, `stages.py` | | **[TinyStories](tinystories/)** | End-to-end processing pipeline | `main.py`, `stages.py` | | **[Megatron Tokenizing](megatron-tokenizer/)** | Tokenization pipeline that produces Megatron-LM ready files | `main.py` | +| **[Llama Nemotron Data Curation](llama-nemotron-data-curation/)** | Data curation on the Llama Nemotron Post-Training Dataset | `main.py` and helper files | +| **[GLiNER-based PII Redaction](gliner-pii-redaction/)** | Redacting personally identifiable information with NVIDIA's GLiNER-PII model | `gliner_pii_redaction.ipynb` | ## Documentation Links diff --git a/tutorials/text/gliner-pii-redaction/README.md b/tutorials/text/gliner-pii-redaction/README.md index 7a01603c1a..b329beeef4 100644 --- a/tutorials/text/gliner-pii-redaction/README.md +++ b/tutorials/text/gliner-pii-redaction/README.md @@ -13,3 +13,5 @@ Additionally, the tutorial uses the [GLiNER](https://github.com/urchade/GLiNER) ```bash uv pip install gliner ``` + +For reference, this tutorial was tested with `gliner==0.2.24`. diff --git a/tutorials/text/gliner-pii-redaction/gliner_pii_redaction.ipynb b/tutorials/text/gliner-pii-redaction/gliner_pii_redaction.ipynb index 7c993f55df..2d551b70f5 100644 --- a/tutorials/text/gliner-pii-redaction/gliner_pii_redaction.ipynb +++ b/tutorials/text/gliner-pii-redaction/gliner_pii_redaction.ipynb @@ -26,6 +26,13 @@ "# !uv pip install gliner" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For reference, this tutorial was tested with `gliner==0.2.24`." + ] + }, { "cell_type": "code", "execution_count": 2, diff --git a/tutorials/text/gliner-pii-redaction/gliner_pii_redactor.py b/tutorials/text/gliner-pii-redaction/gliner_pii_redactor.py index 32fe3bf547..2274381897 100644 --- a/tutorials/text/gliner-pii-redaction/gliner_pii_redactor.py +++ b/tutorials/text/gliner-pii-redaction/gliner_pii_redactor.py @@ -164,7 +164,7 @@ def process(self, batch: DocumentBatch) -> DocumentBatch | None: # Run model inference via the GLiNER library # This returns a list of dictionaries per document, each containing the entities and their confidence scores - entities = self.model.run( + entities = self.model.inference( df[self.text_field].tolist(), self.labels, threshold=self.threshold,