Stream files to storage backend chunk by chunk #304

ddelange · 2022-06-21T13:56:08Z

ref #302 (comment)

builds upon #302, #303

fixes #210

depends on piskvorky/smart_open#702, getmoto/moto#5255

API:

added self.get_uri to s3,gcs,azure storages. for the rest no changes

Functionality:

does not need to load the whole package into memory to perform request.db.upload (will now do multi-part uploads)
does not need to generate signed urls to perform a streamed response (rather re-uses the storage client) when stream_files is true
does not need to load a whole package into memory when generate_hashes is true

CI:

tests use snakepacker/python:all as base image
moto PR was done to support ACL
azure mock (vcrpy) replaced with azurite server
gcs mock expanded for multipart upload

pypicloud/storage/azure_blob.py

pypicloud/storage/gcs.py

stevearc

Seems like it's currently in a bad rebase state. Sending back to you

ddelange · 2022-06-22T09:29:58Z

mocking multipart gcs still needs some work https://github.com/stevearc/pypicloud/runs/6999063103?check_suite_focus=true#step:4:2385

as well as mocking s3 ref getmoto/moto#5252

coveralls · 2022-07-10T09:54:30Z

Coverage increased (+0.04%) to 82.916% when pulling 6edcc42 on ddelange:multipart_upload into 523e0c5 on stevearc:master.

ddelange · 2022-07-10T10:10:25Z

docker/Dockerfile

-RUN curl https://raw.githubusercontent.com/fkrull/docker-multi-python/master/setup.sh -o /setup.sh \
-  && bash setup.sh \
-  && rm /setup.sh
+RUN apt-install libldap2-dev libsasl2-dev default-jre


ref https://github.com/snakepacker/python#apt-install

this image contains git, tox, all the python versions etc. shaves off 1+ minute from CI for each env

ddelange · 2022-07-10T10:13:47Z

setup.py

@@ -13,7 +13,7 @@

 REQUIREMENTS_TEST = open(os.path.join(HERE, "requirements_test.txt")).readlines()
 REQUIREMENTS = [
-    "smart_open[http]",
+    "smart_open[s3,http]@git+https://github.com/ddelange/smart_open@patch-1",


waiting for piskvorky/smart_open#702

@stevearc I'll update the PR once this merges, until then any PR comments are appreciated :)

when I start a fresh pod running stevearc/pypicloud:1.3.6, k8s still registers 141Mi. potentially a memprofiling candidate 🤔

memprofiling using filprofiler

a default server.ini generated using local filesystem option, killed once the server booted succesfully:

fil-profile run -m pyramid.scripts.pserve server.ini

https://user-images.githubusercontent.com/14880945/178993376-ef2c9111-33fa-4dda-b27e-956fcbe1ccb0.svg

stevearc · 2022-07-14T01:56:30Z

Sorry for the delay, will do a review soon.

stevearc

Two small comments, otherwise looks good. We're just waiting on the smart_open PR to merge?

pypicloud/storage/azure_blob.py

tests/test_storage.py

ref piskvorky/smart_open#702 (comment)

ddelange · 2022-08-21T19:09:13Z

@stevearc I think this is ready to ship :)

ddelange · 2022-08-21T21:14:25Z

if this looks good to you, could you puah the docker image also for arm64 ref stevearc/pypicloud-docker#37?

stevearc · 2022-08-23T03:43:02Z

Sick! I'll release shortly

stevearc · 2022-08-23T04:55:42Z

Release is cut, but the docker build and publish is broken. Will debug when I get time

ddelange changed the title ~~Multipart upload~~ Stream files to storage backend chunk by chunk Jun 21, 2022

This was referenced Jun 21, 2022

Stream files from fallback index chunk by chunk #302

Merged

Caching large packages craches server #210

Closed

ddelange force-pushed the multipart_upload branch 12 times, most recently from 4a62ff5 to fea86fa Compare June 21, 2022 19:03

ddelange commented Jun 21, 2022

View reviewed changes

pypicloud/storage/azure_blob.py Outdated Show resolved Hide resolved

ddelange commented Jun 21, 2022

View reviewed changes

pypicloud/storage/gcs.py Outdated Show resolved Hide resolved

ddelange force-pushed the multipart_upload branch 2 times, most recently from e436d98 to b2ab1f2 Compare June 21, 2022 19:27

stevearc suggested changes Jun 22, 2022

View reviewed changes

ddelange force-pushed the multipart_upload branch from b2ab1f2 to e7c51c4 Compare June 22, 2022 05:46

ddelange mentioned this pull request Jun 22, 2022

Stream files from storage backend chunk by chunk #303

Merged

ddelange force-pushed the multipart_upload branch 3 times, most recently from 2d290cd to cb97fcb Compare June 22, 2022 07:12

ddelange marked this pull request as draft June 22, 2022 09:32

ddelange force-pushed the multipart_upload branch 3 times, most recently from 4969dae to dd047ce Compare June 23, 2022 09:57

ddelange commented Jul 10, 2022

View reviewed changes

ddelange marked this pull request as ready for review July 10, 2022 10:14

ddelange requested a review from stevearc July 10, 2022 10:14

ddelange force-pushed the multipart_upload branch 7 times, most recently from d6a3b3f to 7a511df Compare July 10, 2022 12:35

Improve stream download efficiency

ecb3f1e

ddelange force-pushed the multipart_upload branch from 7a511df to ecb3f1e Compare July 11, 2022 08:00

stevearc suggested changes Jul 15, 2022

View reviewed changes

pypicloud/storage/azure_blob.py Outdated Show resolved Hide resolved

tests/test_storage.py Show resolved Hide resolved

Skip TestAzureStorage when azurite container is not running

511b791

ddelange force-pushed the multipart_upload branch from 396f7ac to 511b791 Compare July 15, 2022 07:15

ddelange mentioned this pull request Jul 19, 2022

How to use private s3 bucket in pypicloud ? stevearc/pypicloud-docker#34

Open

ddelange added 2 commits August 21, 2022 20:05

Replace smart_open VCS requirement

d1fb601

ref piskvorky/smart_open#702 (comment)

Merge branch 'master' into multipart_upload

6edcc42

stevearc merged commit 37b7666 into stevearc:master Aug 23, 2022

yannayh mentioned this pull request Aug 26, 2022

Support python 3.10 #313

Open

vanguard737 mentioned this pull request Sep 25, 2022

Package upload fails with "AttributeError: can't set attribute" when using pypicloud[gcs] 1.3.9 or later at GAE #317

Closed

ddelange mentioned this pull request Oct 17, 2022

Replace GCS testing fakes with fake-gcs-server ddelange/pypicloud#4

Merged

ddelange mentioned this pull request Nov 19, 2022

Fix passing object_acl in gcs.py #320

Merged

4 tasks

ddelange mentioned this pull request Aug 28, 2023

DRAFT: Support Pluggable Storage Backends pypiserver/pypiserver#519

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream files to storage backend chunk by chunk #304

Stream files to storage backend chunk by chunk #304

ddelange commented Jun 21, 2022 •

edited

Loading

stevearc left a comment

ddelange commented Jun 22, 2022

coveralls commented Jul 10, 2022 •

edited

Loading

ddelange Jul 10, 2022

ddelange Jul 10, 2022 •

edited

Loading

ddelange Jul 11, 2022

ddelange Jul 11, 2022

ddelange Jul 14, 2022

stevearc commented Jul 14, 2022

stevearc left a comment

ddelange commented Aug 21, 2022

ddelange commented Aug 21, 2022

stevearc commented Aug 23, 2022

stevearc commented Aug 23, 2022

Stream files to storage backend chunk by chunk #304

Stream files to storage backend chunk by chunk #304

Conversation

ddelange commented Jun 21, 2022 • edited Loading

stevearc left a comment

Choose a reason for hiding this comment

ddelange commented Jun 22, 2022

coveralls commented Jul 10, 2022 • edited Loading

ddelange Jul 10, 2022

Choose a reason for hiding this comment

ddelange Jul 10, 2022 • edited Loading

Choose a reason for hiding this comment

ddelange Jul 11, 2022

Choose a reason for hiding this comment

ddelange Jul 11, 2022

Choose a reason for hiding this comment

ddelange Jul 14, 2022

Choose a reason for hiding this comment

stevearc commented Jul 14, 2022

stevearc left a comment

Choose a reason for hiding this comment

ddelange commented Aug 21, 2022

ddelange commented Aug 21, 2022

stevearc commented Aug 23, 2022

stevearc commented Aug 23, 2022

ddelange commented Jun 21, 2022 •

edited

Loading

coveralls commented Jul 10, 2022 •

edited

Loading

ddelange Jul 10, 2022 •

edited

Loading