Skip to content

Commit

Permalink
Image uploader script (#2164)
Browse files Browse the repository at this point in the history
<!--
Open the PR up as a draft until you feel it is ready for a proper
review.

Do not make PR:s from your own `main` branch, as that makes it difficult
for reviewers to add their own fixes.

Add any improvements to the branch as new commits to make it easier for
reviewers to follow the progress. All commits will be squashed to a
single commit once the PR is merged into `main`.

Make sure you mention any issues that this PR closes in the description,
as well as any other related issues.

To get an auto-generated PR description you can put "copilot:summary" or
"copilot:walkthrough" anywhere.
-->

Closes #2132

### What

Adds `just upload <image>`, which uploads the image to google cloud
using `gsutil`. The resulting URL contains the content hash plus the
original file name, which ensures that it is unique.

```
$ just upload image.png
https://static.rerun.io/dd31922030a6bf7223d5e3728d8da5407f4d6b1a_image.png
```

The script itself contains instructions for how to setup `gsutil`. It's
really just a link to Google's tutorial. Eventually it will be
documented separately together with the rest of the new workflow.

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [ ] I've included a screenshot or gif (if applicable)

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2164

---------

Co-authored-by: Emil Ernerfeldt <[email protected]>
  • Loading branch information
jprochazk and emilk authored May 29, 2023
1 parent 9a4597a commit e1f0e39
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 2 deletions.
6 changes: 4 additions & 2 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@ py-dev-env:
set -euxo pipefail
python3 -m venv venv
venv/bin/pip install --upgrade pip
venv/bin/pip install -r rerun_py/requirements-build.txt
venv/bin/pip install -r rerun_py/requirements-lint.txt
venv/bin/pip install -r scripts/requirements-dev.txt
echo "Do 'source venv/bin/activate' to use the virtual environment!"
# Run all examples with the specified args
Expand Down Expand Up @@ -139,3 +138,6 @@ download-design-tokens:
# Update the results of `insta` snapshot regression tests
update-insta-tests:
cargo test; cargo insta review

upload *ARGS:
python3 "scripts/upload_image.py" {{ARGS}}
7 changes: 7 additions & 0 deletions scripts/requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Pip packages all developers need

-r ../rerun_py/requirements-build.txt
-r ../rerun_py/requirements-doc.txt
-r ../rerun_py/requirements-lint.txt

google-cloud-storage==2.9.0 # for scripts/upload_image.py
51 changes: 51 additions & 0 deletions scripts/upload_image.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/usr/bin/env python3

"""
Upload an image to Google Cloud.
Requires the following packages:
pip install google-cloud-storage # NOLINT
Before running, you have to authenticate via the Google Cloud CLI:
- Install it (https://cloud.google.com/storage/docs/gsutil_install)
- Set up credentials (https://cloud.google.com/storage/docs/gsutil_install#authenticate)
"""

import argparse
import hashlib
import mimetypes
import os

from google.cloud import storage


def content_hash(path: str) -> str:
h = hashlib.sha1()
b = bytearray(128 * 1024)
mv = memoryview(b)
with open(path, "rb", buffering=0) as f:
while n := f.readinto(mv):
h.update(mv[:n])
return h.hexdigest()


def main() -> None:
parser = argparse.ArgumentParser(description="Upload an image.")
parser.add_argument("path", type=str, help="Path to the image.")
args = parser.parse_args()

hash = content_hash(args.path)
object_name = f"{hash}_{os.path.basename(args.path)}"

gcs = storage.Client()
bucket = gcs.bucket("rerun-static-img")
destination = bucket.blob(object_name)
destination.content_type, destination.content_encoding = mimetypes.guess_type(args.path)
with open(args.path, "rb") as f:
destination.upload_from_file(f)

print(f"https://static.rerun.io/{object_name}")


if __name__ == "__main__":
main()

1 comment on commit e1f0e39

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.25.

Benchmark suite Current: e1f0e39 Previous: 9a4597a Ratio
datastore/num_rows=1000/num_instances=1000/packed=false/insert/default 3986366 ns/iter (± 51193) 2861613 ns/iter (± 3167) 1.39
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/default 400 ns/iter (± 1) 306 ns/iter (± 1) 1.31
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/default 292 ns/iter (± 0) 224 ns/iter (± 0) 1.30
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/default 449 ns/iter (± 0) 344 ns/iter (± 0) 1.31
datastore/num_rows=1000/num_instances=1000/packed=false/range/default 4058851 ns/iter (± 86370) 2905120 ns/iter (± 6005) 1.40
datastore/num_rows=1000/num_instances=1000/gc/default 2551256 ns/iter (± 5120) 1702497 ns/iter (± 3280) 1.50
mono_points_arrow_batched/generate_message_bundles 25210570 ns/iter (± 1154057) 18559607 ns/iter (± 88976) 1.36
mono_points_arrow_batched/generate_messages 5749356 ns/iter (± 232511) 3720628 ns/iter (± 17390) 1.55
mono_points_arrow_batched/encode_log_msg 621028 ns/iter (± 1983) 410100 ns/iter (± 2353) 1.51
mono_points_arrow_batched/encode_total 33064980 ns/iter (± 1193657) 23669533 ns/iter (± 172879) 1.40
mono_points_arrow_batched/decode_log_msg 514444 ns/iter (± 1694) 305703 ns/iter (± 1318) 1.68
mono_points_arrow_batched/decode_message_bundles 9457301 ns/iter (± 133343) 7383920 ns/iter (± 13050) 1.28
mono_points_arrow_batched/decode_total 9984228 ns/iter (± 205940) 7715481 ns/iter (± 9251) 1.29
batch_points_arrow/encode_log_msg 95731 ns/iter (± 681) 56019 ns/iter (± 199) 1.71
batch_points_arrow/encode_total 363114 ns/iter (± 1574) 273793 ns/iter (± 406) 1.33
batch_points_arrow/decode_log_msg 70018 ns/iter (± 1493) 47439 ns/iter (± 113) 1.48
batch_points_arrow/decode_total 79141 ns/iter (± 305) 49945 ns/iter (± 120) 1.58
arrow_mono_points/insert 2553751553 ns/iter (± 5508725) 1751781104 ns/iter (± 7484297) 1.46
arrow_mono_points/query 1347956 ns/iter (± 9729) 964693 ns/iter (± 10469) 1.40
arrow_batch_points/query 15017 ns/iter (± 36) 11158 ns/iter (± 6) 1.35
arrow_batch_vecs/query 387910 ns/iter (± 310) 297006 ns/iter (± 1091) 1.31

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.