From af55a75a2b6ff0a971810251507683625531a560 Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Wed, 27 Mar 2024 15:25:47 -0700 Subject: [PATCH 1/7] [K8s]: Adds a helper script to create a dask cluster on k8s and includes instructions for how to a Curator workload on k8s Signed-off-by: Terry Kong --- docs/user-guide/CPUvsGPU.rst | 8 + docs/user-guide/KubernetesCurator.rst | 382 ++++++++++++++++++++++++++ docs/user-guide/index.rst | 4 + examples/k8s/create_dask_cluster.py | 98 +++++++ 4 files changed, 492 insertions(+) create mode 100644 docs/user-guide/KubernetesCurator.rst create mode 100644 examples/k8s/create_dask_cluster.py diff --git a/docs/user-guide/CPUvsGPU.rst b/docs/user-guide/CPUvsGPU.rst index 5fd901d199..fa5ea6aa86 100644 --- a/docs/user-guide/CPUvsGPU.rst +++ b/docs/user-guide/CPUvsGPU.rst @@ -96,3 +96,11 @@ Every SLURM cluster is different, so make sure you understand how your SLURM clu Our Python examples are designed to work such that they can be run locally on their own, or easily substituted into the ``start-slurm.sh`` to run on multiple nodes. You can adapt your scripts easily too by simply following the pattern of adding ``get_client`` with ``add_distributed_args``. + +----------------------------------------- +Dask with K8s +----------------------------------------- + +We also provide an example guide for how to get started with NeMo Curator on a Kubernetes cluster. + +Please visit :ref:`curator_kubernetes` for more information. diff --git a/docs/user-guide/KubernetesCurator.rst b/docs/user-guide/KubernetesCurator.rst new file mode 100644 index 0000000000..b36472718c --- /dev/null +++ b/docs/user-guide/KubernetesCurator.rst @@ -0,0 +1,382 @@ +.. _curator_kubernetes: + +====================================== +Running NeMo Curator on Kubernetes +====================================== +The following example demonstrates how to run the NeMo Curator with NVIDIA GPUs on a Kubernetes cluster, +with PersistentVolumeClaims as the storage option. + +.. note:: + This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. + +Prerequisuites +-------------- +* Kubernetes cluster + * `GPU operator `__ + * `Dask Operator `__ +* `kubectl `__: the Kubernetes Cluster CLI + * Please reach out to your Kubernetes cluster admin for how to setup your ``kubectl`` KUBECONFIG +* `ReadWriteMany `__ `StorageClass `__ (setup by Kubernetes cluster admin) + +Storage +------- +To run NeMo Curator, we need to setup storage to upload and store the input +files, as well as any processed outputs. + +Here is an example of how to create a dynamic PV from a StorageClass setup +by your cluster admin. Replace ``STORAGE_CLASS=<...>`` with the name of +your StorageClass. + +This example requests ``150Gi`` of space. Adjust that number for your +workloads and be aware that not all storage provisioners support volume +resizing. + +.. code-block:: bash + + STORAGE_CLASS=<...> + PVC_NAME=nemo-workspace + + kubectl apply -f - <`` with your NGC secret +key. Note that if you have any special characters in your key you might need to wrap +the key in single quotes (``'``) so it can be parsed correctly by k8s):: + + kubectl create secret docker-registry ngc-registry --docker-server=nvcr.io --docker-username=\$oauthtoken --docker-password= + +Setup Python Environment +------------------------ + +Setup a virtual environment: + +.. code-block:: bash + + python3 -m venv venv + source venv/bin/activate + + pip install 'dask_kubernetes>=2024.4.1' + +Upload Data to PVC +------------------ + +To copy into the ``nemo-workspace`` PVC, we will do so with ``kubectl exec``. You may also +use ``kubectl cp``, but ``exec`` has fewer surprises regarding compressed files: + +.. code-block:: bash + + # Replace <...> with a path on your local machine + LOCAL_WORKSPACE=<...> + + # This copies $LOCAL_WORKSPACE/big_english to /big_english within the PVC. + # Change foobar to the directory or file you wish to upload. + ( cd $LOCAL_WORKSPACE; tar cf - big_english | kubectl exec -i nemo-workspace-busybox -- tar xf - -C /nemo-workspace ) + +.. note:: + See :ref:`data-curator-download` for an example of how to download local data that can be uploaded to the PVC + with the above instruction. + +Create a Dask Cluster +--------------------- + +Use the ``create_dask_cluster.py`` to create a CPU or GPU dask cluster. + +.. note:: + If you are creating another Dask cluster with the same ``--name ``, first delete it via:: + + kubectl delete daskcluster + +.. code-block:: bash + + # Creates a CPU Dask cluster with 1 worker + python create_dask_cluster.py \ + --name rapids-dask \ + --n_workers 1 \ + --image nvcr.io/nvidian/bignlp-train:nemofw-nightly \ + --image_pull_secret ngc-registry \ + --pvcs nemo-workspace:/nemo-workspace + + #╭───────────────────── Creating KubeCluster 'rapids-dask' ─────────────────────╮ + #│ │ + #│ DaskCluster Running │ + #│ Scheduler Pod Running │ + #│ Scheduler Service Created │ + #│ Default Worker Group Created │ + #│ │ + #│ ⠧ Getting dashboard URL │ + #╰──────────────────────────────────────────────────────────────────────────────╯ + #cluster = KubeCluster(rapids-dask, 'tcp://localhost:61757', workers=2, threads=510, memory=3.94 TiB) + + # Creates a GPU Dask cluster with 2 workers with 1 GPU each + python create_dask_cluster.py \ + --name rapids-dask \ + --n_workers 2 \ + --n_gpus_per_worker 1 \ + --image nvcr.io/nvidian/bignlp-train:nemofw-nightly \ + --image_pull_secret ngc-registry \ + --pvcs nemo-workspace:/nemo-workspace + +After creating a cluster, you should be able to proceed after confirming the scheduler and the workers are all ``Running``: + +.. code-block:: bash + + # Set DASK_CLUSTER_NAME to the value of --name + DASK_CLUSTER_NAME=rapids-dask + kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME" + + # NAME READY STATUS RESTARTS AGE + # rapids-dask-default-worker-587238cf2c-7d685f4d75-k6rnq 1/1 Running 0 57m + # rapids-dask-default-worker-f8ff963886-5577fff76b-qmvcd 1/1 Running 3 (52m ago) 57m + # rapids-dask-scheduler-654799869d-9bw4z 1/1 Running 0 57m + +.. tab-set:: + + .. tab-item:: Running Existing Module + + Here is an example of running the existing `gpu_exact_dedup` Curator module. The arguments and script name + will need to be changed according to the module you wish to run: + + .. code-block:: bash + + # Set DASK_CLUSTER_NAME to the value of --name + DASK_CLUSTER_NAME=rapids-dask + SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) + # Starts an interactive shell session in the scheduler pod + kubectl exec -it $SCHEDULER_POD -- bash + + ######################## + # Inside SCHEDULER_POD # + ######################## + # Run the following inside the interactive shell to launch script in the background and + # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. + # The command line flags will need to be replaced with whatever the module script accepts. + # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written + # to somewhere under /nemo-workspace. + + mkdir -p /nemo-workspace/curator/{output,log,profile} + # Write logs to script.log and to a log file with a date suffix + LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" + ( + echo "Writing to: $LOGS" + gpu_exact_dedup \ + --input-data-dirs /nemo-workspace/big_english \ + --output-dir /nemo-workspace/curator/output \ + --hash-method md5 \ + --log-dir /nemo-workspace/curator/log \ + --num-files -1 \ + --files-per-partition 1 \ + --profile-path /nemo-workspace/curator/profile \ + --log-frequency 250 \ + --scheduler-address localhost:8786 \ + 2>&1 + echo "Finished!" + ) | tee $LOGS & + + # At this point, feel free to disconnect the shell via Ctrl+D or simply + exit + + At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: + + .. code-block:: bash + + # Command will follow the logs of the running module (Press ctrl+C to close) + kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log + + # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-15-52-31 + # Computing hashes for /nemo-workspace/big_english + # adlr_id _hashes + # 0 cc-2023-14-0397113620 91b77eae49c10a65d485ac8ca18d6c43 + # 1 cc-2023-14-0397113621 a266f0794cc8ffbd431823e6930e4f80 + # 2 cc-2023-14-0397113622 baee533e2eddae764de2cd6faaa1286c + # 3 cc-2023-14-0397113623 87dd52a468448b99078f97e76f528eab + # 4 cc-2023-14-0397113624 a17664daf4f24be58e0e3a3dcf81124a + # Finished! + + + .. tab-item:: Running Custom Module + + In this example, we'll demonstrate how to run a NeMo Curator module that you have defined locally. + + Since your curator module may depend on version of the Curator that differs from what is in the + container, we will need to build a custom image with your code installed: + + .. code-block:: bash + + # Clone your repo. This example uses the official repo + git clone https://github.com/NVIDIA/NeMo-Curator.git NeMo-Curator-dev + + # Checkout specific ref. This example uses a commit in the main branch + git -C NeMo-Curator-dev checkout fc167a6edffd38a55c333742972a5a25b901cb26 + + # Example NeMo base image. Change it according to your requirements + BASE_IMAGE=nvcr.io/nvidian/bignlp-train:nemofw-nightly + docker build -t nemo-curator-custom ./NeMo-Curator-dev -f - </: accordingly + docker tag nemo-curator-custom /: + docker push /: + + .. note:: + When using a custom image, you will likely need to create a different secret unless you pushed to a public registry: + + .. code-block:: bash + + # Fill in // + kubectl create secret docker-registry my-private-registry --docker-server= --docker-username= --docker-password= + + And with this new secret, you create your new dask cluster: + + .. code-block:: bash + + # Fill in // + python create_dask_cluster.py \ + --name rapids-dask \ + --n_workers 2 \ + --n_gpus_per_worker 1 \ + --image /: \ + --image_pull_secret my-private-registry \ + --pvcs nemo-workspace:/nemo-workspace + + After the Dask cluster is deployed, you can proceed to run your module. In this example we'll use + the ``NeMo-Curator/nemo_curator/scripts/find_exact_duplicates.py`` module, but you can find other templates + in `NeMo-Curator/examples `__: + + .. code-block:: bash + + # Set DASK_CLUSTER_NAME to the value of --name + DASK_CLUSTER_NAME=rapids-dask + SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) + # Starts an interactive shell session in the scheduler pod + kubectl exec -it $SCHEDULER_POD -- bash + + ######################## + # Inside SCHEDULER_POD # + ######################## + # Run the following inside the interactive shell to launch script in the background and + # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. + # The command line flags will need to be replaced with whatever the module script accepts. + # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written + # to somewhere under /nemo-workspace. + + mkdir -p /nemo-workspace/curator/{output,log,profile} + # Append logs to script.log and write to a log file with a date suffix + LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" + ( + echo "Writing to: $LOGS" + # Recall that /NeMo-Curator-dev was copied and installed in the Dockerfile above + python3 -u /NeMo-Curator-dev/nemo_curator/scripts/find_exact_duplicates.py \ + --input-data-dirs /nemo-workspace/big_english \ + --output-dir /nemo-workspace/curator/output \ + --hash-method md5 \ + --log-dir /nemo-workspace/curator/log \ + --files-per-partition 1 \ + --profile-path /nemo-workspace/curator/profile \ + --log-frequency 250 \ + --scheduler-address localhost:8786 \ + 2>&1 + echo "Finished!" + ) | tee $LOGS & + + # At this point, feel free to disconnect the shell via Ctrl+D or simply + exit + + At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: + + .. code-block:: bash + + # Command will follow the logs of the running module (Press ctrl+C to close) + kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log + + # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-20-52-07 + # Reading 2 files + # /NeMo-Curator-dev/nemo_curator/modules/exact_dedup.py:157: UserWarning: Output path f/nemo-workspace/curator/output/_exact_duplicates.parquet already exists and will be overwritten + # warnings.warn( + # Finished! + +Deleting Cluster +---------------- +After you have finished using the created dask cluster, you can delete it to release the resources:: + + # Where is the flag passed to create_dask_cluster.py. Example: `--name ` + kubectl delete daskcluster + +Download Data from PVC +---------------------- + +To download data from your PVC, you can use the ``nemo-workspace-busybox`` Pod created earlier: + +.. code-block:: bash + + # Replace <...> with a path on your local machine + LOCAL_WORKSPACE=<...> + + # Tar will fail if LOCAL_WORKSPACE doesn't exist + mkdir -p $LOCAL_WORKSPACE + + # Copy file in PVC at /nemo-workspace/foobar.txt to local file-system at $LOCAL_WORKSPACE/nemo-workspace/foobar.txt + kubectl exec nemo-workspace-busybox -- tar cf - /nemo-workspace/foobar.txt | tar xf - -C $LOCAL_WORKSPACE + + # Copy directory in PVC /nemo-workspace/fizzbuzz to local file-system at $LOCAL_WORKSPACE/fizzbuzz + kubectl exec nemo-workspace-busybox -- tar cf - /nemo-workspace/fizzbuzz | tar xf - -C $LOCAL_WORKSPACE diff --git a/docs/user-guide/index.rst b/docs/user-guide/index.rst index 278e47ab3e..7ba84c03e5 100644 --- a/docs/user-guide/index.rst +++ b/docs/user-guide/index.rst @@ -27,6 +27,9 @@ :ref:`Personally Identifiable Information Identification and Removal ` The purpose of the personally identifiable information (PII) redaction tool is to help scrub sensitive data out of training datasets +:ref:`curator-kubernetes` + Demonstration of how to run the NeMo Curator on a Dask Cluster deployed on top of Kubernetes + .. toctree:: :maxdepth: 4 :titlesonly: @@ -41,3 +44,4 @@ TaskDecontamination.rst PersonalIdentifiableInformationIdentificationAndRemoval.rst DistributedDataClassification.rst + KubernetesCurator.rst diff --git a/examples/k8s/create_dask_cluster.py b/examples/k8s/create_dask_cluster.py new file mode 100644 index 0000000000..326f24f340 --- /dev/null +++ b/examples/k8s/create_dask_cluster.py @@ -0,0 +1,98 @@ +from dask_kubernetes.operator.kubecluster import make_cluster_spec, KubeCluster +import argparse + + +def create_cluster( + name: str, + n_workers: int, + n_gpus_per_worker: int, + n_cpus_per_worker: int, + image: str, + image_pull_secret: str, + pvcs: dict[str, str], +): + dask_worker_command = "dask-worker" + if n_gpus_per_worker and n_gpus_per_worker > 0: + dask_worker_command = "dask-cuda-worker" + + custom_cluster_spec = make_cluster_spec( + name=name, + worker_command=dask_worker_command, + n_workers=n_workers, + image=image, + ) + scheduler_spec = custom_cluster_spec["spec"]["scheduler"]["spec"] + worker_spec = custom_cluster_spec["spec"]["worker"]["spec"] + if image_pull_secret: + scheduler_spec["imagePullSecrets"] = [{"name": image_pull_secret}] + worker_spec["imagePullSecrets"] = [{"name": image_pull_secret}] + + obj_vols = [] + obj_vol_mounts = [] + for pvc_name, mount_path in pvcs.items(): + obj_vols.append( + { + "name": pvc_name, + "persistentVolumeClaim": { + "claimName": pvc_name, + }, + } + ) + obj_vol_mounts.append( + { + "name": pvc_name, + "mountPath": mount_path, + } + ) + + scheduler_spec["volumes"] = obj_vols + for ctr in scheduler_spec["containers"]: + ctr["volumeMounts"] = obj_vol_mounts + + worker_spec["volumes"] = obj_vols + for ctr in worker_spec["containers"]: + ctr["volumeMounts"] = obj_vol_mounts + # Resources are added to only the worker, since the scheduler doesn't need GPUs + if n_gpus_per_worker or n_cpus_per_worker: + if not ctr["resources"]: + ctr["resources"] = {"limits": {}} + if n_gpus_per_worker: + ctr["resources"]["limits"]["nvidia.com/gpu"] = str(n_gpus_per_worker) + if n_cpus_per_worker: + ctr["resources"]["limits"]["cpu"] = str(n_cpus_per_worker) + + cluster = KubeCluster(custom_cluster_spec=custom_cluster_spec, shutdown_on_close=False) + print(f'{cluster = }') + +if __name__ == "__main__": + + def parse_pvcs(specs: str) -> dict[str, str]: + name_to_path = {} + for pvc in specs.split(","): + # Can be empty + if not pvc: + continue + name, _, path = pvc.partition(":") + name_to_path[name] = path + return name_to_path + + parser = argparse.ArgumentParser() + parser.add_argument("-n", "--name", type=str, default="rapids-dask") + parser.add_argument("-w", "--n_workers", type=int, default=2) + parser.add_argument("-g", "--n_gpus_per_worker", type=int, default=None) + parser.add_argument("-c", "--n_cpus_per_worker", type=int, default=None) + parser.add_argument("-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.01.framework") + parser.add_argument("-s", "--image_pull_secret", type=str, default=None) + parser.add_argument( + "-p", + "--pvcs", + type=parse_pvcs, + default="", + help="Comma sep PVC specificiation of $pvc_name_1:$mount_path_1,$pvc_name_2:$mount_path_2. Example: foo:/foo,bar:/bar mounts pvcs named foo and bar to /foo and /bar respectively.", + ) + + args = parser.parse_args() + + create_cluster( + **vars(args), + ) From df51fd806e38cc38f48d05868dd8dacced3fce9e Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Mon, 22 Apr 2024 11:52:04 -0700 Subject: [PATCH 2/7] black formatting Signed-off-by: Terry Kong --- examples/k8s/create_dask_cluster.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/examples/k8s/create_dask_cluster.py b/examples/k8s/create_dask_cluster.py index 326f24f340..a49cfc8156 100644 --- a/examples/k8s/create_dask_cluster.py +++ b/examples/k8s/create_dask_cluster.py @@ -48,7 +48,7 @@ def create_cluster( scheduler_spec["volumes"] = obj_vols for ctr in scheduler_spec["containers"]: ctr["volumeMounts"] = obj_vol_mounts - + worker_spec["volumes"] = obj_vols for ctr in worker_spec["containers"]: ctr["volumeMounts"] = obj_vol_mounts @@ -61,8 +61,11 @@ def create_cluster( if n_cpus_per_worker: ctr["resources"]["limits"]["cpu"] = str(n_cpus_per_worker) - cluster = KubeCluster(custom_cluster_spec=custom_cluster_spec, shutdown_on_close=False) - print(f'{cluster = }') + cluster = KubeCluster( + custom_cluster_spec=custom_cluster_spec, shutdown_on_close=False + ) + print(f"{cluster = }") + if __name__ == "__main__": @@ -81,7 +84,9 @@ def parse_pvcs(specs: str) -> dict[str, str]: parser.add_argument("-w", "--n_workers", type=int, default=2) parser.add_argument("-g", "--n_gpus_per_worker", type=int, default=None) parser.add_argument("-c", "--n_cpus_per_worker", type=int, default=None) - parser.add_argument("-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.01.framework") + parser.add_argument( + "-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.01.framework" + ) parser.add_argument("-s", "--image_pull_secret", type=str, default=None) parser.add_argument( "-p", From f99890bfd7b6b23995220cf879a7ebe1d09db52d Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Tue, 23 Apr 2024 10:37:49 -0700 Subject: [PATCH 3/7] big_english -> my_dataset Signed-off-by: Terry Kong --- docs/user-guide/KubernetesCurator.rst | 334 +++++++++++++------------- 1 file changed, 168 insertions(+), 166 deletions(-) diff --git a/docs/user-guide/KubernetesCurator.rst b/docs/user-guide/KubernetesCurator.rst index b36472718c..57ad3b89d2 100644 --- a/docs/user-guide/KubernetesCurator.rst +++ b/docs/user-guide/KubernetesCurator.rst @@ -125,9 +125,9 @@ use ``kubectl cp``, but ``exec`` has fewer surprises regarding compressed files: # Replace <...> with a path on your local machine LOCAL_WORKSPACE=<...> - # This copies $LOCAL_WORKSPACE/big_english to /big_english within the PVC. + # This copies $LOCAL_WORKSPACE/my_dataset to /my_dataset within the PVC. # Change foobar to the directory or file you wish to upload. - ( cd $LOCAL_WORKSPACE; tar cf - big_english | kubectl exec -i nemo-workspace-busybox -- tar xf - -C /nemo-workspace ) + ( cd $LOCAL_WORKSPACE; tar cf - my_dataset | kubectl exec -i nemo-workspace-busybox -- tar xf - -C /nemo-workspace ) .. note:: See :ref:`data-curator-download` for an example of how to download local data that can be uploaded to the PVC @@ -186,178 +186,180 @@ After creating a cluster, you should be able to proceed after confirming the sch # rapids-dask-default-worker-f8ff963886-5577fff76b-qmvcd 1/1 Running 3 (52m ago) 57m # rapids-dask-scheduler-654799869d-9bw4z 1/1 Running 0 57m -.. tab-set:: +(Opt #1) Running Existing Module +-------------------------------- - .. tab-item:: Running Existing Module +Here is an example of running the existing `gpu_exact_dedup` Curator module. The arguments and script name +will need to be changed according to the module you wish to run: - Here is an example of running the existing `gpu_exact_dedup` Curator module. The arguments and script name - will need to be changed according to the module you wish to run: +.. code-block:: bash - .. code-block:: bash - - # Set DASK_CLUSTER_NAME to the value of --name - DASK_CLUSTER_NAME=rapids-dask - SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) - # Starts an interactive shell session in the scheduler pod - kubectl exec -it $SCHEDULER_POD -- bash - - ######################## - # Inside SCHEDULER_POD # - ######################## - # Run the following inside the interactive shell to launch script in the background and - # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. - # The command line flags will need to be replaced with whatever the module script accepts. - # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written - # to somewhere under /nemo-workspace. - - mkdir -p /nemo-workspace/curator/{output,log,profile} - # Write logs to script.log and to a log file with a date suffix - LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" - ( - echo "Writing to: $LOGS" - gpu_exact_dedup \ - --input-data-dirs /nemo-workspace/big_english \ - --output-dir /nemo-workspace/curator/output \ - --hash-method md5 \ - --log-dir /nemo-workspace/curator/log \ - --num-files -1 \ - --files-per-partition 1 \ - --profile-path /nemo-workspace/curator/profile \ - --log-frequency 250 \ - --scheduler-address localhost:8786 \ - 2>&1 - echo "Finished!" - ) | tee $LOGS & - - # At this point, feel free to disconnect the shell via Ctrl+D or simply - exit - - At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: - - .. code-block:: bash - - # Command will follow the logs of the running module (Press ctrl+C to close) - kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log - - # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-15-52-31 - # Computing hashes for /nemo-workspace/big_english - # adlr_id _hashes - # 0 cc-2023-14-0397113620 91b77eae49c10a65d485ac8ca18d6c43 - # 1 cc-2023-14-0397113621 a266f0794cc8ffbd431823e6930e4f80 - # 2 cc-2023-14-0397113622 baee533e2eddae764de2cd6faaa1286c - # 3 cc-2023-14-0397113623 87dd52a468448b99078f97e76f528eab - # 4 cc-2023-14-0397113624 a17664daf4f24be58e0e3a3dcf81124a - # Finished! - - - .. tab-item:: Running Custom Module - - In this example, we'll demonstrate how to run a NeMo Curator module that you have defined locally. - - Since your curator module may depend on version of the Curator that differs from what is in the - container, we will need to build a custom image with your code installed: - - .. code-block:: bash - - # Clone your repo. This example uses the official repo - git clone https://github.com/NVIDIA/NeMo-Curator.git NeMo-Curator-dev - - # Checkout specific ref. This example uses a commit in the main branch - git -C NeMo-Curator-dev checkout fc167a6edffd38a55c333742972a5a25b901cb26 - - # Example NeMo base image. Change it according to your requirements - BASE_IMAGE=nvcr.io/nvidian/bignlp-train:nemofw-nightly - docker build -t nemo-curator-custom ./NeMo-Curator-dev -f - </: accordingly - docker tag nemo-curator-custom /: - docker push /: - - .. note:: - When using a custom image, you will likely need to create a different secret unless you pushed to a public registry: - - .. code-block:: bash - - # Fill in // - kubectl create secret docker-registry my-private-registry --docker-server= --docker-username= --docker-password= - - And with this new secret, you create your new dask cluster: - - .. code-block:: bash - - # Fill in // - python create_dask_cluster.py \ - --name rapids-dask \ - --n_workers 2 \ - --n_gpus_per_worker 1 \ - --image /: \ - --image_pull_secret my-private-registry \ - --pvcs nemo-workspace:/nemo-workspace - - After the Dask cluster is deployed, you can proceed to run your module. In this example we'll use - the ``NeMo-Curator/nemo_curator/scripts/find_exact_duplicates.py`` module, but you can find other templates - in `NeMo-Curator/examples `__: - - .. code-block:: bash - - # Set DASK_CLUSTER_NAME to the value of --name - DASK_CLUSTER_NAME=rapids-dask - SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) - # Starts an interactive shell session in the scheduler pod - kubectl exec -it $SCHEDULER_POD -- bash - - ######################## - # Inside SCHEDULER_POD # - ######################## - # Run the following inside the interactive shell to launch script in the background and - # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. - # The command line flags will need to be replaced with whatever the module script accepts. - # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written - # to somewhere under /nemo-workspace. - - mkdir -p /nemo-workspace/curator/{output,log,profile} - # Append logs to script.log and write to a log file with a date suffix - LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" - ( - echo "Writing to: $LOGS" - # Recall that /NeMo-Curator-dev was copied and installed in the Dockerfile above - python3 -u /NeMo-Curator-dev/nemo_curator/scripts/find_exact_duplicates.py \ - --input-data-dirs /nemo-workspace/big_english \ - --output-dir /nemo-workspace/curator/output \ - --hash-method md5 \ - --log-dir /nemo-workspace/curator/log \ - --files-per-partition 1 \ - --profile-path /nemo-workspace/curator/profile \ - --log-frequency 250 \ - --scheduler-address localhost:8786 \ - 2>&1 - echo "Finished!" - ) | tee $LOGS & - - # At this point, feel free to disconnect the shell via Ctrl+D or simply - exit - - At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: - - .. code-block:: bash + # Set DASK_CLUSTER_NAME to the value of --name + DASK_CLUSTER_NAME=rapids-dask + SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) + # Starts an interactive shell session in the scheduler pod + kubectl exec -it $SCHEDULER_POD -- bash + + ######################## + # Inside SCHEDULER_POD # + ######################## + # Run the following inside the interactive shell to launch script in the background and + # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. + # The command line flags will need to be replaced with whatever the module script accepts. + # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written + # to somewhere under /nemo-workspace. + + mkdir -p /nemo-workspace/curator/{output,log,profile} + # Write logs to script.log and to a log file with a date suffix + LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" + ( + echo "Writing to: $LOGS" + gpu_exact_dedup \ + --input-data-dirs /nemo-workspace/my_dataset \ + --output-dir /nemo-workspace/curator/output \ + --hash-method md5 \ + --log-dir /nemo-workspace/curator/log \ + --num-files -1 \ + --files-per-partition 1 \ + --profile-path /nemo-workspace/curator/profile \ + --log-frequency 250 \ + --scheduler-address localhost:8786 \ + 2>&1 + echo "Finished!" + ) | tee $LOGS & + + # At this point, feel free to disconnect the shell via Ctrl+D or simply + exit + +At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: + +.. code-block:: bash + + # Command will follow the logs of the running module (Press ctrl+C to close) + kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log + + # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-15-52-31 + # Computing hashes for /nemo-workspace/my_dataset + # adlr_id _hashes + # 0 cc-2023-14-0397113620 91b77eae49c10a65d485ac8ca18d6c43 + # 1 cc-2023-14-0397113621 a266f0794cc8ffbd431823e6930e4f80 + # 2 cc-2023-14-0397113622 baee533e2eddae764de2cd6faaa1286c + # 3 cc-2023-14-0397113623 87dd52a468448b99078f97e76f528eab + # 4 cc-2023-14-0397113624 a17664daf4f24be58e0e3a3dcf81124a + # Finished! + + +(Opt #2) Running Custom Module +------------------------------ + +In this example, we'll demonstrate how to run a NeMo Curator module that you have defined locally. + +Since your curator module may depend on version of the Curator that differs from what is in the +container, we will need to build a custom image with your code installed: + +.. code-block:: bash + + # Clone your repo. This example uses the official repo + git clone https://github.com/NVIDIA/NeMo-Curator.git NeMo-Curator-dev + + # Checkout specific ref. This example uses a commit in the main branch + git -C NeMo-Curator-dev checkout fc167a6edffd38a55c333742972a5a25b901cb26 + + # Example NeMo base image. Change it according to your requirements + BASE_IMAGE=nvcr.io/nvidian/bignlp-train:nemofw-nightly + docker build -t nemo-curator-custom ./NeMo-Curator-dev -f - </: accordingly + docker tag nemo-curator-custom /: + docker push /: + +.. note:: + When using a custom image, you will likely need to create a different secret unless you pushed to a public registry: + + .. code-block:: bash - # Command will follow the logs of the running module (Press ctrl+C to close) - kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log + # Fill in // + kubectl create secret docker-registry my-private-registry --docker-server= --docker-username= --docker-password= - # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-20-52-07 - # Reading 2 files - # /NeMo-Curator-dev/nemo_curator/modules/exact_dedup.py:157: UserWarning: Output path f/nemo-workspace/curator/output/_exact_duplicates.parquet already exists and will be overwritten - # warnings.warn( - # Finished! + And with this new secret, you create your new dask cluster: + + .. code-block:: bash + + # Fill in // + python create_dask_cluster.py \ + --name rapids-dask \ + --n_workers 2 \ + --n_gpus_per_worker 1 \ + --image /: \ + --image_pull_secret my-private-registry \ + --pvcs nemo-workspace:/nemo-workspace + +After the Dask cluster is deployed, you can proceed to run your module. In this example we'll use +the ``NeMo-Curator/nemo_curator/scripts/find_exact_duplicates.py`` module, but you can find other templates +in `NeMo-Curator/examples `__: + +.. code-block:: bash + + # Set DASK_CLUSTER_NAME to the value of --name + DASK_CLUSTER_NAME=rapids-dask + SCHEDULER_POD=$(kubectl get pods -l "dask.org/cluster-name=$DASK_CLUSTER_NAME,dask.org/component=scheduler" -o name) + # Starts an interactive shell session in the scheduler pod + kubectl exec -it $SCHEDULER_POD -- bash + + ######################## + # Inside SCHEDULER_POD # + ######################## + # Run the following inside the interactive shell to launch script in the background and + # tee the logs to the /nemo-workspace PVC that was mounted in for persistence. + # The command line flags will need to be replaced with whatever the module script accepts. + # Recall that the PVC is mounted at /nemo-workspace, so any outputs should be written + # to somewhere under /nemo-workspace. + + mkdir -p /nemo-workspace/curator/{output,log,profile} + # Append logs to script.log and write to a log file with a date suffix + LOGS="/nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.$(date +%y_%m_%d-%H-%M-%S)" + ( + echo "Writing to: $LOGS" + # Recall that /NeMo-Curator-dev was copied and installed in the Dockerfile above + python3 -u /NeMo-Curator-dev/nemo_curator/scripts/find_exact_duplicates.py \ + --input-data-dirs /nemo-workspace/my_dataset \ + --output-dir /nemo-workspace/curator/output \ + --hash-method md5 \ + --log-dir /nemo-workspace/curator/log \ + --files-per-partition 1 \ + --profile-path /nemo-workspace/curator/profile \ + --log-frequency 250 \ + --scheduler-address localhost:8786 \ + 2>&1 + echo "Finished!" + ) | tee $LOGS & + + # At this point, feel free to disconnect the shell via Ctrl+D or simply + exit + +At this point you can tail the logs and look for ``Finished!`` in ``/nemo-workspace/curator/script.log``: + +.. code-block:: bash + + # Command will follow the logs of the running module (Press ctrl+C to close) + kubectl exec -it $SCHEDULER_POD -- tail -f /nemo-workspace/curator/script.log + + # Writing to: /nemo-workspace/curator/script.log /nemo-workspace/curator/script.log.24_03_27-20-52-07 + # Reading 2 files + # /NeMo-Curator-dev/nemo_curator/modules/exact_dedup.py:157: UserWarning: Output path f/nemo-workspace/curator/output/_exact_duplicates.parquet already exists and will be overwritten + # warnings.warn( + # Finished! Deleting Cluster ---------------- -After you have finished using the created dask cluster, you can delete it to release the resources:: +After you have finished using the created dask cluster, you can delete it to release the resources: + +.. code-block:: bash # Where is the flag passed to create_dask_cluster.py. Example: `--name ` kubectl delete daskcluster From ea4f34ba88ba52ac5bc9a58a07bad1bff45d6993 Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Tue, 23 Apr 2024 10:38:24 -0700 Subject: [PATCH 4/7] 24.01 -> 24.03 default container Signed-off-by: Terry Kong --- examples/k8s/create_dask_cluster.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/k8s/create_dask_cluster.py b/examples/k8s/create_dask_cluster.py index a49cfc8156..b77227efc7 100644 --- a/examples/k8s/create_dask_cluster.py +++ b/examples/k8s/create_dask_cluster.py @@ -85,7 +85,7 @@ def parse_pvcs(specs: str) -> dict[str, str]: parser.add_argument("-g", "--n_gpus_per_worker", type=int, default=None) parser.add_argument("-c", "--n_cpus_per_worker", type=int, default=None) parser.add_argument( - "-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.01.framework" + "-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.03.framework" ) parser.add_argument("-s", "--image_pull_secret", type=str, default=None) parser.add_argument( From 866668cfdd7902269f31ff4524b1d7cf5abf352c Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Tue, 23 Apr 2024 10:45:33 -0700 Subject: [PATCH 5/7] Add help kwarg to all flags Signed-off-by: Terry Kong --- examples/k8s/create_dask_cluster.py | 42 ++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/examples/k8s/create_dask_cluster.py b/examples/k8s/create_dask_cluster.py index b77227efc7..a520985251 100644 --- a/examples/k8s/create_dask_cluster.py +++ b/examples/k8s/create_dask_cluster.py @@ -80,14 +80,44 @@ def parse_pvcs(specs: str) -> dict[str, str]: return name_to_path parser = argparse.ArgumentParser() - parser.add_argument("-n", "--name", type=str, default="rapids-dask") - parser.add_argument("-w", "--n_workers", type=int, default=2) - parser.add_argument("-g", "--n_gpus_per_worker", type=int, default=None) - parser.add_argument("-c", "--n_cpus_per_worker", type=int, default=None) parser.add_argument( - "-i", "--image", type=str, default="nvcr.io/nvidia/nemo:24.03.framework" + "-n", + "--name", + type=str, + default="rapids-dask", + help="The name of the DaskCluster which you would be able to inspect via `kubectl describe daskcluster `.", + ) + parser.add_argument( + "-w", "--n_workers", type=int, default=2, help="Number of workers" + ) + parser.add_argument( + "-g", + "--n_gpus_per_worker", + type=int, + default=None, + help="Number of GPUs per worker. If not specified, the Dask Cluster defaults to a CPU cluster.", + ) + parser.add_argument( + "-c", + "--n_cpus_per_worker", + type=int, + default=None, + help="Number of CPUs per worker. Provide this flag if you want to limit your CPU resources and K8s will throttle the workers to make sure this limit is satisfied.", + ) + parser.add_argument( + "-i", + "--image", + type=str, + default="nvcr.io/nvidia/nemo:24.03.framework", + help="The image used for the Dask Cluster scheduler and workers.", + ) + parser.add_argument( + "-s", + "--image_pull_secret", + type=str, + default=None, + help="If --image is from a private registry, specify the appropriate pull secret you created to allow these to be pulled.", ) - parser.add_argument("-s", "--image_pull_secret", type=str, default=None) parser.add_argument( "-p", "--pvcs", From f78cdf10441eac454912e65c7dfbd9ee80d2758f Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Tue, 23 Apr 2024 11:01:18 -0700 Subject: [PATCH 6/7] Clarify why venv is needed Signed-off-by: Terry Kong --- docs/user-guide/KubernetesCurator.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/KubernetesCurator.rst b/docs/user-guide/KubernetesCurator.rst index 57ad3b89d2..3f3614d752 100644 --- a/docs/user-guide/KubernetesCurator.rst +++ b/docs/user-guide/KubernetesCurator.rst @@ -105,7 +105,9 @@ the key in single quotes (``'``) so it can be parsed correctly by k8s):: Setup Python Environment ------------------------ -Setup a virtual environment: +The environment to run the provided scripts in this example does not need the full +``nemo_curator`` package, so you can create a virtual environment with just the +required packages as follows: .. code-block:: bash From c53b9fcba035b1fa400aa301a9a3d2ab773da252 Mon Sep 17 00:00:00 2001 From: Terry Kong Date: Tue, 23 Apr 2024 15:02:50 -0700 Subject: [PATCH 7/7] fix precommit failures Signed-off-by: Terry Kong --- docs/user-guide/KubernetesCurator.rst | 10 +++++----- examples/k8s/create_dask_cluster.py | 3 ++- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/user-guide/KubernetesCurator.rst b/docs/user-guide/KubernetesCurator.rst index 3f3614d752..c7f727df0a 100644 --- a/docs/user-guide/KubernetesCurator.rst +++ b/docs/user-guide/KubernetesCurator.rst @@ -59,7 +59,7 @@ Setup PVC Busybox Helper Pod ############################ Inspecting the PVC and copying to and from it is facilitated with a busybox container. -Some examples below assume you have this Pod running to copy to and from the PVC. +Some examples below assume you have this Pod running to copy to and from the PVC. .. code-block:: bash @@ -123,8 +123,8 @@ To copy into the ``nemo-workspace`` PVC, we will do so with ``kubectl exec``. Yo use ``kubectl cp``, but ``exec`` has fewer surprises regarding compressed files: .. code-block:: bash - - # Replace <...> with a path on your local machine + + # Replace <...> with a path on your local machine LOCAL_WORKSPACE=<...> # This copies $LOCAL_WORKSPACE/my_dataset to /my_dataset within the PVC. @@ -284,7 +284,7 @@ container, we will need to build a custom image with your code installed: When using a custom image, you will likely need to create a different secret unless you pushed to a public registry: .. code-block:: bash - + # Fill in // kubectl create secret docker-registry my-private-registry --docker-server= --docker-username= --docker-password= @@ -373,7 +373,7 @@ To download data from your PVC, you can use the ``nemo-workspace-busybox`` Pod c .. code-block:: bash - # Replace <...> with a path on your local machine + # Replace <...> with a path on your local machine LOCAL_WORKSPACE=<...> # Tar will fail if LOCAL_WORKSPACE doesn't exist diff --git a/examples/k8s/create_dask_cluster.py b/examples/k8s/create_dask_cluster.py index a520985251..28be575ebe 100644 --- a/examples/k8s/create_dask_cluster.py +++ b/examples/k8s/create_dask_cluster.py @@ -1,6 +1,7 @@ -from dask_kubernetes.operator.kubecluster import make_cluster_spec, KubeCluster import argparse +from dask_kubernetes.operator.kubecluster import KubeCluster, make_cluster_spec + def create_cluster( name: str,