From eecb2025fc733abf9904f7502481c9c1179983a1 Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Thu, 15 Jan 2026 14:10:04 -0800 Subject: [PATCH 1/2] Update instructions for AWS credentials in ArXiv download and extract tutorial Signed-off-by: Sarah Yurick --- .../download_extract_tutorial.ipynb | 66 +++++++++++-------- 1 file changed, 37 insertions(+), 29 deletions(-) diff --git a/tutorials/text/download-and-extract/download_extract_tutorial.ipynb b/tutorials/text/download-and-extract/download_extract_tutorial.ipynb index 46b1278f88..81fc233acf 100644 --- a/tutorials/text/download-and-extract/download_extract_tutorial.ipynb +++ b/tutorials/text/download-and-extract/download_extract_tutorial.ipynb @@ -39,11 +39,33 @@ "!apt install -y wget" ] }, + { + "cell_type": "markdown", + "id": "abffed09", + "metadata": {}, + "source": [ + "If running the ArXiv Download and Extract section of this notebook, you must set your AWS credentials here:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "08e15d88", + "metadata": {}, + "outputs": [], + "source": [ + "%env AWS_ACCESS_KEY_ID=\n", + "%env AWS_SECRET_ACCESS_KEY=\n", + "%env AWS_SESSION_TOKEN=" + ] + }, { "cell_type": "markdown", "id": "c6c131c1", "metadata": {}, "source": [ + "To avoid credentials-related errors, you must specify the above environment variables **before** starting the Ray client (see below). Do not include quotes when setting your credentials (e.g., it should be `%env AWS_ACCESS_KEY_ID=XYZ` and not `%env AWS_ACCESS_KEY_ID=\"XYZ\"`).\n", + "\n", "To run a pipeline in NeMo Curator, we must start a Ray cluster. This can be done manually (see the [Ray documentation](https://docs.ray.io/en/latest/ray-core/starting-ray.html)) or with Curator's `RayClient`:" ] }, @@ -398,34 +420,8 @@ "More information about downloading ArXiv data with NeMo Curator can be found in the [documentation](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/arxiv.html).\n", "\n", "### Prerequisites\n", - "Before proceeding, we need to set **AWS credentials** in the environment as the data are stored in AWS S3 [requester pays buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) (pricing details could be found [here](https://aws.amazon.com/s3/pricing/))." - ] - }, - { - "cell_type": "markdown", - "id": "ef101752-aa99-470a-b587-4548b83c1625", - "metadata": {}, - "source": [ - "#### Set your AWS Credentials" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "193812d1-69b0-46d7-a1f4-525e52573671", - "metadata": {}, - "outputs": [], - "source": [ - "%env AWS_ACCESS_KEY_ID=\n", - "%env AWS_SECRET_ACCESS_KEY=\n", - "%env AWS_SESSION_TOKEN=" - ] - }, - { - "cell_type": "markdown", - "id": "988105cf-5b4e-4288-be1d-4c4219c98bf9", - "metadata": {}, - "source": [ + "Before proceeding, we need to set **AWS credentials** in the environment as the data are stored in AWS S3 [requester pays buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) (pricing details could be found [here](https://aws.amazon.com/s3/pricing/)). Credentials should be set at the top of this notebook, before starting a Ray client object.\n", + "\n", "#### Ensure the ArXiv S3 bucket accessibility" ] }, @@ -438,7 +434,19 @@ }, "outputs": [], "source": [ - "!s5cmd --request-payer=requester ls s3://arxiv/src/ | head | grep '.tar'" + "import subprocess\n", + "\n", + "cmd = \"s5cmd --request-payer=requester ls s3://arxiv/src/ | head | grep '.tar'\"\n", + "\n", + "# shell=True needed because we use pipes\n", + "result = subprocess.run(cmd, shell=True, capture_output=True, text=True)\n", + "\n", + "# stdout has the command output\n", + "print(result.stdout)\n", + "\n", + "# stderr has errors\n", + "if result.stderr:\n", + " print(\"Errors:\", result.stderr)" ] }, { From 26858aaffbee3403ade1c7b74b0c00883063bd35 Mon Sep 17 00:00:00 2001 From: Sarah Yurick Date: Thu, 15 Jan 2026 14:15:33 -0800 Subject: [PATCH 2/2] ruff Signed-off-by: Sarah Yurick --- .../text/download-and-extract/download_extract_tutorial.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/text/download-and-extract/download_extract_tutorial.ipynb b/tutorials/text/download-and-extract/download_extract_tutorial.ipynb index 81fc233acf..8619e01742 100644 --- a/tutorials/text/download-and-extract/download_extract_tutorial.ipynb +++ b/tutorials/text/download-and-extract/download_extract_tutorial.ipynb @@ -439,7 +439,7 @@ "cmd = \"s5cmd --request-payer=requester ls s3://arxiv/src/ | head | grep '.tar'\"\n", "\n", "# shell=True needed because we use pipes\n", - "result = subprocess.run(cmd, shell=True, capture_output=True, text=True)\n", + "result = subprocess.run(cmd, shell=True, capture_output=True, text=True, check=True) # noqa: S602\n", "\n", "# stdout has the command output\n", "print(result.stdout)\n",