diff --git a/tutorials/text/distributed-data-classification/README.md b/tutorials/text/distributed-data-classification/README.md
index a4c72b417e..ae298c8bf2 100644
--- a/tutorials/text/distributed-data-classification/README.md
+++ b/tutorials/text/distributed-data-classification/README.md
@@ -15,17 +15,23 @@ For more information about the classifiers, refer to our [Distributed Data Class
-| NeMo Curator Classifier | Hugging Face Page |
-| --- | --- |
-| `AegisClassifier` | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) and [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0) |
-| `ContentTypeClassifier` | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) |
-| `DomainClassifier` | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) |
-| `FineWebEduClassifier` | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) |
-| `FineWebMixtralEduClassifier` | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) |
-| `FineWebNemotronEduClassifier` | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) |
-| `InstructionDataGuardClassifier` | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) |
-| `MultilingualDomainClassifier` | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) |
-| `PromptTaskComplexityClassifier` | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) |
-| `QualityClassifier` | [quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) |
+| NeMo Curator Classifier | Description | Hugging Face Page |
+| --- | --- | --- |
+| `AegisClassifier` | Identify and categorize unsafe content per document | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) and [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Permissive-1.0) |
+| `ContentTypeClassifier` | Categorize the type-of-speech per document | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) |
+| `DomainClassifier` | Categorize the domain per document | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) |
+| `FineWebEduClassifier` | Determine the educational value per document; this model was trained using annotations from Llama 3 70B-Instruct | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) |
+| `FineWebMixtralEduClassifier` | Determine the educational value per document; this model was trained using annotations from Mixtral 8x22B-Instruct | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) |
+| `FineWebNemotronEduClassifier` | Determine the educational value per document; this model was trained using annotations from Nemotron-4-340B-Instruct | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) |
+| `InstructionDataGuardClassifier` | Identify LLM poisoning attacks per document | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) |
+| `MultilingualDomainClassifier` | Categorize the domain per document; supports classification in 52 languages | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) |
+| `PromptTaskComplexityClassifier` | Classifies text prompts across task types and complexity dimensions | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) |
+| `QualityClassifier` | Categorize documents as high, medium, or low quality | [quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) |
+
+Note that all classifiers support English text classification only, except the `MultilingualDomainClassifier`.
+
+## Bring Your Own Classifier
+
+Advanced users may want to integrate their own Hugging Face classifier(s) into NeMo Curator. Broadly, this requires creating a `CompositeStage` consisting of a CPU-based tokenizer stage and a GPU-based model inference stage. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for details about how to do this.
diff --git a/tutorials/text/distributed-data-classification/aegis-classification.ipynb b/tutorials/text/distributed-data-classification/aegis-classification.ipynb
index 314c8b3d1c..917f9bae07 100644
--- a/tutorials/text/distributed-data-classification/aegis-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/aegis-classification.ipynb
@@ -14,9 +14,9 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
@@ -25,12 +25,50 @@
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/content-type-classification.ipynb b/tutorials/text/distributed-data-classification/content-type-classification.ipynb
index 02e0dbc832..18c8d0cf28 100644
--- a/tutorials/text/distributed-data-classification/content-type-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/content-type-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/domain-classification.ipynb b/tutorials/text/distributed-data-classification/domain-classification.ipynb
index e8b1a0648f..8a7ecf1386 100644
--- a/tutorials/text/distributed-data-classification/domain-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/domain-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb
index 21d080ebfe..b3d1daba60 100644
--- a/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/fineweb-edu-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb
index ca5d416c29..0ec66b99b6 100644
--- a/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/fineweb-mixtral-edu-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb b/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb
index 78611bcbbd..7b9996993c 100644
--- a/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/fineweb-nemotron-edu-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb b/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb
index 411d0fbc40..d6a508b5fc 100644
--- a/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/instruction-data-guard-classification.ipynb
@@ -14,9 +14,9 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
@@ -25,12 +25,50 @@
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb b/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb
index e852e96eb2..9e53044e25 100644
--- a/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/multilingual-domain-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb b/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb
index 57877a2778..96dbc175a0 100644
--- a/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/prompt-task-complexity-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/distributed-data-classification/quality-classification.ipynb b/tutorials/text/distributed-data-classification/quality-classification.ipynb
index 339851d194..6d646f44e8 100644
--- a/tutorials/text/distributed-data-classification/quality-classification.ipynb
+++ b/tutorials/text/distributed-data-classification/quality-classification.ipynb
@@ -12,23 +12,61 @@
" - Volta™ or higher (compute capability 7.0+)\n",
" - CUDA 12.x\n",
"\n",
- "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies.\n",
+ "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page. Refer to the [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/classifier.html) page for more information about quality classification in NeMo Curator.\n",
"\n",
- "For more information about the classifiers, refer to our [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/distributed-classifier.html) documentation page. Refer to the [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/classifier.html) page for more information about quality classification in NeMo Curator."
+ "Before running this notebook, see this [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html#admin-installation) page for instructions on how to install NeMo Curator. Be sure to use an installation method which includes GPU dependencies (`text_cuda12` or `all`). Check proper installation with:"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# Silence Curator logs via Loguru\n",
+ "# First, silence Curator logs via Loguru\n",
"import os\n",
"\n",
"os.environ[\"LOGURU_LEVEL\"] = \"ERROR\""
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import nemo_curator\n",
+ "\n",
+ "nemo_curator.__version__ # should be >= 1.0.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can check that GPUs are available, then check that the `gpustat` dependency was installed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!nvidia-smi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import gpustat\n",
+ "\n",
+ "gpustat.__version__ # check gpu dependency is installed"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/tutorials/text/llama-nemotron-data-curation/README.md b/tutorials/text/llama-nemotron-data-curation/README.md
index 28b8d5a2da..f2846e96bf 100644
--- a/tutorials/text/llama-nemotron-data-curation/README.md
+++ b/tutorials/text/llama-nemotron-data-curation/README.md
@@ -119,7 +119,7 @@ LOGURU_LEVEL="ERROR" python main.py \
--num-cpus 16
```
-Setting `LOGURU_LEVEL="ERROR"` helps minimize long logs. Removing it can be helpful for debugging.
+Setting `LOGURU_LEVEL="ERROR"` helps minimize long logs. Removing it can be helpful for debugging. See the **Debugging Out of Memory Errors** section for help (most likely, the answer is to reduce `--num-cpus`).
The user may set `--hf-token` as needed for the tokenizer.
@@ -147,7 +147,7 @@ If you encounter out-of-memory (OOM) errors:
1. **Reduce partition size**: Lower the blocksize to reduce per-partition memory. Set `--json-blocksize "50mb"` (default is "100mb").
2. **Reduce CPU count**: Lower `--num-cpus` to reduce parallel memory pressure rather than using all available cores.
-3. **Subset the data**: Use `--filename-filter` to process only specific subsets (such as `--filename-filter "chat"`).
+3. **Subset the data**: Use `--filename-filter` to process only specific subsets relevant to your use case (such as `--filename-filter "chat"`).
## Next Steps