From 368349b79c202b63e78f973fb6694e075c36c2b6 Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Mon, 9 Feb 2026 10:28:27 -0800 Subject: [PATCH 1/4] Add feedback to tutorials Signed-off-by: Sarah Yurick --- tutorials/text/llama-nemotron-data-curation/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/text/llama-nemotron-data-curation/README.md b/tutorials/text/llama-nemotron-data-curation/README.md index f70f128767..e25aa56249 100644 --- a/tutorials/text/llama-nemotron-data-curation/README.md +++ b/tutorials/text/llama-nemotron-data-curation/README.md @@ -119,7 +119,7 @@ LOGURU_LEVEL="ERROR" python main.py \ --num-cpus 16 ``` -Setting `LOGURU_LEVEL="ERROR"` helps minimize long logs. Removing it can be helpful for debugging. +Setting `LOGURU_LEVEL="ERROR"` helps minimize long logs. Removing it can be helpful for debugging. See the **Debugging Out of Memory Errors** section for help (most likely, the answer is to reduce `--num-cpus`). The user may set `--hf-token` as needed for the tokenizer. @@ -138,9 +138,9 @@ After filtering, it sorts all samples by completion (output response) length, th ## Debugging Out of Memory Errors -If you are running into out of memory (OOM) errors, there are a couple of approaches you can try. One is to avoid very large partitions of data. By default, the JSONL data is read with a blocksize of 100 MB per partition. To customize the file reading logic, the user may specify `--json-blocksize "100mb"` with any string representation for the partition size in MB (e.g., "100mb", "256mb"). +If you are running into out of memory (OOM) errors, there are a couple of approaches you can try. The first thing the user should try is reducing the `--num-cpus` parameter. The goal is to maximize it for improved performance without oversubscribing your available hardware. -It can be useful to play around with the `--num-cpus` parameter as well. The goal is to maximize it for improved performance without oversubscribing your available hardware. +It is recommended to avoid very large partitions of data. By default, the JSONL data is read with a blocksize of 100 MB per partition. To customize the file reading logic, the user may specify `--json-blocksize "100mb"` with any string representation for the partition size in MB (e.g., "100mb", "256mb"). ## Next Steps From 9af952ee8d9dbecd6fd96c8c0b93b9825ca419fe Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Mon, 9 Feb 2026 11:03:19 -0800 Subject: [PATCH 2/4] clarify install instructions for classifier tutorials Signed-off-by: Sarah Yurick --- .../aegis-classification.ipynb | 44 ++++++++++++++++-- .../content-type-classification.ipynb | 46 +++++++++++++++++-- .../domain-classification.ipynb | 46 +++++++++++++++++-- .../fineweb-edu-classification.ipynb | 46 +++++++++++++++++-- .../fineweb-mixtral-edu-classification.ipynb | 46 +++++++++++++++++-- .../fineweb-nemotron-edu-classification.ipynb | 46 +++++++++++++++++-- ...nstruction-data-guard-classification.ipynb | 44 ++++++++++++++++-- .../multilingual-domain-classification.ipynb | 46 +++++++++++++++++-- ...rompt-task-complexity-classification.ipynb | 46 +++++++++++++++++-- .../quality-classification.ipynb | 46 +++++++++++++++++-- 10 files changed, 418 insertions(+), 38 deletions(-) diff --git a/tutorials/text/distributed-data-classification/aegis-classification.ipynb b/tutorials/text/distributed-data-classification/aegis-classification.ipynb index 314c8b3d1c..917f9bae07 100644 --- a/tutorials/text/distributed-data-classification/aegis-classification.ipynb +++ b/tutorials/text/distributed-data-classification/aegis-classification.ipynb @@ -14,9 +14,9 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { @@ -25,12 +25,50 @@ "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/content-type-classification.ipynb b/tutorials/text/distributed-data-classification/content-type-classification.ipynb index 02e0dbc832..18c8d0cf28 100644 --- a/tutorials/text/distributed-data-classification/content-type-classification.ipynb +++ b/tutorials/text/distributed-data-classification/content-type-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/domain-classification.ipynb b/tutorials/text/distributed-data-classification/domain-classification.ipynb index e8b1a0648f..8a7ecf1386 100644 --- a/tutorials/text/distributed-data-classification/domain-classification.ipynb +++ b/tutorials/text/distributed-data-classification/domain-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb index 21d080ebfe..b3d1daba60 100644 --- a/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb +++ b/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb index ca5d416c29..0ec66b99b6 100644 --- a/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb +++ b/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb index 78611bcbbd..7b9996993c 100644 --- a/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb +++ b/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb b/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb index 411d0fbc40..d6a508b5fc 100644 --- a/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb +++ b/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb @@ -14,9 +14,9 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { @@ -25,12 +25,50 @@ "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb b/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb index e852e96eb2..9e53044e25 100644 --- a/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb +++ b/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb b/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb index 57877a2778..96dbc175a0 100644 --- a/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb +++ b/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tutorials/text/distributed-data-classification/quality-classification.ipynb b/tutorials/text/distributed-data-classification/quality-classification.ipynb index 339851d194..6d646f44e8 100644 --- a/tutorials/text/distributed-data-classification/quality-classification.ipynb +++ b/tutorials/text/distributed-data-classification/quality-classification.ipynb @@ -12,23 +12,61 @@ " - Volta™ or higher (compute capability 7.0+)\n", " - CUDA 12.x\n", "\n", - "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n", + "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page. Refer to the [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/classifier.html) page for more information about quality classification in NeMo Curator.\n", "\n", - "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page. Refer to the [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/classifier.html) page for more information about quality classification in NeMo Curator." + "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Silence Curator logs via Loguru\n", + "# First, silence Curator logs via Loguru\n", "import os\n", "\n", "os.environ[\"LOGURU_LEVEL\"] = \"ERROR\"" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import nemo_curator\n", + "\n", + "nemo_curator.__version__ # should be >= 1.0.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can check that GPUs are available, then check that the `gpustat` dependency was installed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gpustat\n", + "\n", + "gpustat.__version__ # check gpu dependency is installed" + ] + }, { "cell_type": "markdown", "metadata": {}, From ef82cd328c98cf4994f34bbcabc7d491fae1ceda Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Mon, 9 Feb 2026 11:21:52 -0800 Subject: [PATCH 3/4] byo classifiers Signed-off-by: Sarah Yurick --- tutorials/text/distributed-data-classification/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tutorials/text/distributed-data-classification/README.md b/tutorials/text/distributed-data-classification/README.md index a4c72b417e..8076442223 100644 --- a/tutorials/text/distributed-data-classification/README.md +++ b/tutorials/text/distributed-data-classification/README.md @@ -29,3 +29,7 @@ For more information about the classifiers, refer to our [Distributed Data Class | `QualityClassifier` | [quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | + +## Bring Your Own Classifier + +Advanced users may want to integrate their own Hugging Face classifier(s) into NeMo Curator. Broadly, this requires creating a `CompositeStage` consisting of a CPU-based tokenizer stage and a GPU-based model inference stage. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for details about how to do this. From 7506b6b65c9eb1f6477e9bedc1127a777f02a9be Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Mon, 9 Feb 2026 11:43:18 -0800 Subject: [PATCH 4/4] add descriptions Signed-off-by: Sarah Yurick --- .../distributed-data-classification/README.md | 26 ++++++++++--------- 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/tutorials/text/distributed-data-classification/README.md b/tutorials/text/distributed-data-classification/README.md index 8076442223..ae298c8bf2 100644 --- a/tutorials/text/distributed-data-classification/README.md +++ b/tutorials/text/distributed-data-classification/README.md @@ -15,21 +15,23 @@ For more information about the classifiers, refer to our [Distributed Data Class
-| NeMo Curator Classifier | Hugging Face Page | -| --- | --- | -| `AegisClassifier` | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) and [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0) | -| `ContentTypeClassifier` | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | -| `DomainClassifier` | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | -| `FineWebEduClassifier` | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | -| `FineWebMixtralEduClassifier` | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | -| `FineWebNemotronEduClassifier` | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | -| `InstructionDataGuardClassifier` | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | -| `MultilingualDomainClassifier` | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | -| `PromptTaskComplexityClassifier` | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | -| `QualityClassifier` | [quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | +| NeMo Curator Classifier | Description | Hugging Face Page | +| --- | --- | --- | +| `AegisClassifier` | Identify and categorize unsafe content per document | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) and [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0) | +| `ContentTypeClassifier` | Categorize the type-of-speech per document | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | +| `DomainClassifier` | Categorize the domain per document | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | +| `FineWebEduClassifier` | Determine the educational value per document; this model was trained using annotations from Llama 3 70B-Instruct | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | +| `FineWebMixtralEduClassifier` | Determine the educational value per document; this model was trained using annotations from Mixtral 8x22B-Instruct | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | +| `FineWebNemotronEduClassifier` | Determine the educational value per document; this model was trained using annotations from Nemotron-4-340B-Instruct | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | +| `InstructionDataGuardClassifier` | Identify LLM poisoning attacks per document | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | +| `MultilingualDomainClassifier` | Categorize the domain per document; supports classification in 52 languages | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | +| `PromptTaskComplexityClassifier` | Classifies text prompts across task types and complexity dimensions | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | +| `QualityClassifier` | Categorize documents as high, medium, or low quality | [quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) |
+Note that all classifiers support English text classification only, except the `MultilingualDomainClassifier`. + ## Bring Your Own Classifier Advanced users may want to integrate their own Hugging Face classifier(s) into NeMo Curator. Broadly, this requires creating a `CompositeStage` consisting of a CPU-based tokenizer stage and a GPU-based model inference stage. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for details about how to do this.