Skip to content

Migrate gsutil usage to gcloud storage #13722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bhandarivijay-png wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Migrate gsutil usage to gcloud storage #13722

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dataflow/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ def bucket_name(test_name: str, location: str, unique_id: str) -> Iterator[str]:
# Try to remove all files before deleting the bucket.
# Deleting a bucket with too many files results in an error.
try:
run_cmd("gsutil", "-m", "rm", "-rf", f"gs://{bucket_name}/*")
run_cmd("gcloud", "storage", "rm", "--recursive", "--continue-on-error", f"gs://{bucket_name}/*")
except RuntimeError:
# If no files were found and it fails, ignore the error.
pass
Expand Down
6 changes: 3 additions & 3 deletions dataflow/encryption-keys/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Additionally, for this sample you need the following:

```sh
export BUCKET=your-gcs-bucket
gsutil mb gs://$BUCKET
gcloud storage buckets create gs://$BUCKET
```

1. [Create a symmetric key ring](https://cloud.google.com/kms/docs/creating-keys).
Expand Down Expand Up @@ -174,10 +174,10 @@ To avoid incurring charges to your GCP account for the resources used:

```sh
# Remove only the files created by this sample.
gsutil -m rm -rf "gs://$BUCKET/samples/dataflow/kms"
gcloud storage rm --recursive --continue-on-error "gs://$BUCKET/samples/dataflow/kms"

# [optional] Remove the Cloud Storage bucket.
gsutil rb gs://$BUCKET
gcloud storage buckets delete gs://$BUCKET

# Remove the BigQuery table.
bq rm -f -t $PROJECT:$DATASET.$TABLE
Expand Down
3 changes: 1 addition & 2 deletions dataflow/flex-templates/getting_started/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Make sure you have followed the

```sh
export BUCKET="your--bucket"
gsutil mb gs://$BUCKET
gcloud storage buckets create gs://$BUCKET
```

## create an Artifact Registry repository
Expand Down Expand Up @@ -51,4 +51,3 @@ gcloud dataflow flex-template run "flex-`date +%Y%m%d-%H%M%S`" \

For more information about building and running flex templates, see
📝 [Use Flex Templates](https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates).

4 changes: 2 additions & 2 deletions dataflow/flex-templates/pipeline_with_dependencies/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ rules. It is optional.
export PROJECT="project-id"
export BUCKET="your-bucket"
export REGION="us-central1"
gsutil mb -p $PROJECT gs://$BUCKET
gcloud storage buckets create gs://$BUCKET --project=$PROJECT
```

## Create an Artifact Registry repository
Expand Down Expand Up @@ -165,7 +165,7 @@ gcloud dataflow flex-template run "flex-`date +%Y%m%d-%H%M%S`" \
After the pipeline finishes, use the following command to inspect the output:

```bash
gsutil cat gs://$BUCKET/output*
gcloud storage cat gs://$BUCKET/output*
```

## Optional: Update the dependencies in the requirements file and rebuild the Docker images
Expand Down
6 changes: 3 additions & 3 deletions dataflow/flex-templates/streaming_beam/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Additionally, for this sample you need the following:

```sh
export BUCKET="your-gcs-bucket"
gsutil mb gs://$BUCKET
gcloud storage buckets create gs://$BUCKET
```

1. Create a
Expand Down Expand Up @@ -231,7 +231,7 @@ The following sections describe how to delete or turn off these resources.
1. Delete the template spec file from Cloud Storage.

```sh
gsutil rm $TEMPLATE_PATH
gcloud storage rm $TEMPLATE_PATH
```

1. Delete the Flex Template container image from Container Registry.
Expand Down Expand Up @@ -277,7 +277,7 @@ The following sections describe how to delete or turn off these resources.
> These objects cannot be recovered.
>
> ```sh
> gsutil rm -r gs://$BUCKET
> gcloud storage rm --recursive gs://$BUCKET
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The command gcloud storage rm --recursive gs://$BUCKET is not the correct equivalent for gsutil rm -r gs://$BUCKET when the goal is to empty a bucket. The gsutil command removes all objects within the bucket, but not the bucket itself. The gcloud storage rm command used here with a bucket URL will fail.

To correctly empty the bucket using gcloud storage, you should use a wildcard /** to target all objects within the bucket.

Suggested change
> gcloud storage rm --recursive gs://$BUCKET
> gcloud storage rm --recursive "gs://$BUCKET/**"

> ```

## Limitations
Expand Down
2 changes: 1 addition & 1 deletion dataflow/gemma-flex-template/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Click [here to create a GCS bucket](https://console.cloud.google.com/storage/cre

```sh
export GCS_BUCKET="your--bucket"
gsutil mb gs://$GCS_BUCKET
gcloud storage buckets create gs://$GCS_BUCKET
```

Make sure your GCS bucket name does __not__ include the `gs://` prefix
Expand Down
2 changes: 1 addition & 1 deletion dataflow/gemma-flex-template/e2e_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def responses_subscription(

@pytest.fixture(scope="session")
def flex_template_image(utils: Utils) -> str:
conftest.run_cmd("gsutil", "cp", "-r", GEMMA_GCS, ".")
conftest.run_cmd("gcloud", "storage", "cp", "--recursive", GEMMA_GCS, ".")
yield from utils.cloud_build_submit(NAME)


Expand Down
2 changes: 1 addition & 1 deletion dataflow/gemma/e2e_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def test_name() -> str:
@pytest.fixture(scope="session")
def container_image(utils: Utils) -> str:
# Copy Gemma onto the local environment
conftest.run_cmd("gsutil", "cp", "-r", GEMMA_GCS, ".")
conftest.run_cmd("gcloud", "storage", "cp", "--recursive", GEMMA_GCS, ".")
yield from utils.cloud_build_submit(NAME)


Expand Down
2 changes: 1 addition & 1 deletion dataflow/run-inference/tests/e2e_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ def dataflow_job(
) -> Iterator[str]:
# Upload the state dict to Cloud Storage.
state_dict_gcs = f"gs://{bucket_name}/temp/state_dict.pt"
conftest.run_cmd("gsutil", "cp", "-n", state_dict_path, state_dict_gcs)
conftest.run_cmd("gcloud", "storage", "cp", "--no-clobber", state_dict_path, state_dict_gcs)

# Launch the streaming Dataflow pipeline.
conftest.run_cmd(
Expand Down
2 changes: 1 addition & 1 deletion dataflow/run_template/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Additionally, for this sample you need the following:

```sh
export BUCKET=your-gcs-bucket
gsutil mb gs://$BUCKET
gcloud storage buckets create gs://$BUCKET
```

1. Clone the `python-docs-samples` repository.
Expand Down
2 changes: 1 addition & 1 deletion dataproc/snippets/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ To run list_clusters.py:

To run submit_job_to_cluster.py, first create a GCS bucket (used by Cloud Dataproc to stage files) from the Cloud Console or with gsutil:

gsutil mb gs://<your-staging-bucket-name>
gcloud storage buckets create gs://<your-staging-bucket-name>

Next, set the following environment variables:

Expand Down
6 changes: 3 additions & 3 deletions dataproc/snippets/python-api-walkthrough.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ an explanation of how the code works.

* To create a new bucket, run the following command. Your bucket name must be unique.

gsutil mb -p {{project-id}} gs://your-bucket-name
gcloud storage buckets create --project={{project-id}} gs://your-bucket-name


2. Set environment variables.
Expand Down Expand Up @@ -145,12 +145,12 @@ Cluster cluster-name successfully deleted.
If you created a Cloud Storage bucket to use for this walkthrough,
you can run the following command to delete the bucket (the bucket must be empty).

gsutil rb gs://$BUCKET
gcloud storage buckets delete gs://$BUCKET

* You can run the following command to **delete the bucket and all
objects within it. Note: the deleted objects cannot be recovered.**

gsutil rm -r gs://$BUCKET
gcloud storage rm --recursive gs://$BUCKET
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The command gcloud storage rm --recursive gs://$BUCKET is not the correct equivalent for gsutil rm -r gs://$BUCKET when emptying a bucket. The gsutil command removes all objects within the bucket, not the bucket itself, while the gcloud storage rm command with a bucket URL will fail.

To correctly empty the bucket with gcloud storage, you should use a wildcard /**. This change makes the gcloud command's behavior match the original gsutil command.

Suggested change
gcloud storage rm --recursive gs://$BUCKET
gcloud storage rm --recursive "gs://$BUCKET/**"



* **For more information.** See the [Dataproc documentation](https://cloud.google.com/dataproc/docs/)
Expand Down