Skip to content

Commit

Permalink
Image uploader script (#2164)
Browse files Browse the repository at this point in the history
<!--
Open the PR up as a draft until you feel it is ready for a proper
review.

Do not make PR:s from your own `main` branch, as that makes it difficult
for reviewers to add their own fixes.

Add any improvements to the branch as new commits to make it easier for
reviewers to follow the progress. All commits will be squashed to a
single commit once the PR is merged into `main`.

Make sure you mention any issues that this PR closes in the description,
as well as any other related issues.

To get an auto-generated PR description you can put "copilot:summary" or
"copilot:walkthrough" anywhere.
-->

Closes #2132

### What

Adds `just upload <image>`, which uploads the image to google cloud
using `gsutil`. The resulting URL contains the content hash plus the
original file name, which ensures that it is unique.

```
$ just upload image.png
https://static.rerun.io/dd31922030a6bf7223d5e3728d8da5407f4d6b1a_image.png
```

The script itself contains instructions for how to setup `gsutil`. It's
really just a link to Google's tutorial. Eventually it will be
documented separately together with the rest of the new workflow.

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [ ] I've included a screenshot or gif (if applicable)

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2164

---------

Co-authored-by: Emil Ernerfeldt <[email protected]>
  • Loading branch information
jprochazk and emilk committed Jun 15, 2023
1 parent e2ee66c commit 1cc39c1
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 2 deletions.
6 changes: 4 additions & 2 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@ py-dev-env:
set -euxo pipefail
python3 -m venv venv
venv/bin/pip install --upgrade pip
venv/bin/pip install -r rerun_py/requirements-build.txt
venv/bin/pip install -r rerun_py/requirements-lint.txt
venv/bin/pip install -r scripts/requirements-dev.txt
echo "Do 'source venv/bin/activate' to use the virtual environment!"
# Run all examples with the specified args
Expand Down Expand Up @@ -139,3 +138,6 @@ download-design-tokens:
# Update the results of `insta` snapshot regression tests
update-insta-tests:
cargo test; cargo insta review

upload *ARGS:
python3 "scripts/upload_image.py" {{ARGS}}
7 changes: 7 additions & 0 deletions scripts/requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Pip packages all developers need

-r ../rerun_py/requirements-build.txt
-r ../rerun_py/requirements-doc.txt
-r ../rerun_py/requirements-lint.txt

google-cloud-storage==2.9.0 # for scripts/upload_image.py
51 changes: 51 additions & 0 deletions scripts/upload_image.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/usr/bin/env python3

"""
Upload an image to Google Cloud.
Requires the following packages:
pip install google-cloud-storage # NOLINT
Before running, you have to authenticate via the Google Cloud CLI:
- Install it (https://cloud.google.com/storage/docs/gsutil_install)
- Set up credentials (https://cloud.google.com/storage/docs/gsutil_install#authenticate)
"""

import argparse
import hashlib
import mimetypes
import os

from google.cloud import storage


def content_hash(path: str) -> str:
h = hashlib.sha1()
b = bytearray(128 * 1024)
mv = memoryview(b)
with open(path, "rb", buffering=0) as f:
while n := f.readinto(mv):
h.update(mv[:n])
return h.hexdigest()


def main() -> None:
parser = argparse.ArgumentParser(description="Upload an image.")
parser.add_argument("path", type=str, help="Path to the image.")
args = parser.parse_args()

hash = content_hash(args.path)
object_name = f"{hash}_{os.path.basename(args.path)}"

gcs = storage.Client()
bucket = gcs.bucket("rerun-static-img")
destination = bucket.blob(object_name)
destination.content_type, destination.content_encoding = mimetypes.guess_type(args.path)
with open(args.path, "rb") as f:
destination.upload_from_file(f)

print(f"https://static.rerun.io/{object_name}")


if __name__ == "__main__":
main()

0 comments on commit 1cc39c1

Please sign in to comment.