Skip to content

Storage batch API #7387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lmazuel merged 22 commits into from
Oct 3, 2019
Merged

Storage batch API #7387

lmazuel merged 22 commits into from
Oct 3, 2019

Conversation

@lmazuel
Copy link
Member

@lmazuel lmazuel commented Sep 20, 2019

Support for Storage batch API (delete blobs, etc.)

Needs: #7083

@lmazuel lmazuel added Storage Storage Service (Queues, Blobs, Files) Client This issue points to a problem in the data-plane of the library. labels Sep 20, 2019
@lmazuel lmazuel force-pushed the storage_batch branch 3 times, most recently from ffc5d67 to 3f3769d Compare September 30, 2019 22:46
@lmazuel lmazuel marked this pull request as ready for review October 2, 2019 15:41
@lmazuel lmazuel force-pushed the storage_batch branch 3 times, most recently from d9e7c6a to e094f1a Compare October 2, 2019 18:04
@lmazuel lmazuel changed the base branch from features/http_serialization to feature/storage-preview4 October 2, 2019 20:27
@Azure Azure deleted a comment from AutorestCI Oct 2, 2019
@AutorestCI
Copy link
Contributor

AutorestCI commented Oct 2, 2019 •
edited
Loading

(message created by the CI based on PR content)

Installation instruction

Package azure-storage-blob

You can install the package azure-storage-blob of this PR using the following command:
pip install "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-blob&subdirectory=azure-storage-blob"

You can build a wheel to distribute for test using the following command:
pip wheel --no-deps "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-blob&subdirectory=azure-storage-blob"

If you have a local clone of this repository, you can also do:

  • git checkout storage_batch
  • pip install -e ./azure-storage-blob

Or build a wheel file to distribute for testing:

  • git checkout storage_batch
  • pip wheel --no-deps ./azure-storage-blob

Package azure-storage-queue

You can install the package azure-storage-queue of this PR using the following command:
pip install "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-queue&subdirectory=azure-storage-queue"

You can build a wheel to distribute for test using the following command:
pip wheel --no-deps "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-queue&subdirectory=azure-storage-queue"

If you have a local clone of this repository, you can also do:

  • git checkout storage_batch
  • pip install -e ./azure-storage-queue

Or build a wheel file to distribute for testing:

  • git checkout storage_batch
  • pip wheel --no-deps ./azure-storage-queue

Package azure-storage-file

You can install the package azure-storage-file of this PR using the following command:
pip install "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-file&subdirectory=azure-storage-file"

You can build a wheel to distribute for test using the following command:
pip wheel --no-deps "git+https://github.com/Azure/azure-sdk-for-python@storage_batch#egg=azure-storage-file&subdirectory=azure-storage-file"

If you have a local clone of this repository, you can also do:

  • git checkout storage_batch
  • pip install -e ./azure-storage-file

Or build a wheel file to distribute for testing:

  • git checkout storage_batch
  • pip wheel --no-deps ./azure-storage-file

Direct download

Your files can be directly downloaded here:

@lmazuel lmazuel added the P0 label Oct 3, 2019
annatisch
annatisch approved these changes Oct 3, 2019
View reviewed changes
Copy link
Member

@annatisch annatisch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)

@lmazuel lmazuel merged commit a80734c into feature/storage-preview4 Oct 3, 2019
@lmazuel lmazuel deleted the storage_batch branch October 3, 2019 19:47
@lmazuel lmazuel mentioned this pull request Oct 3, 2019
Closed
zezha-msft pushed a commit that referenced this pull request Oct 8, 2019
@zezha-msft
Storage Preview 4 release (#7527)
e2266eb 
* Update msrest and regenerate swagger. (#7308)

* Update msrest and regenerate swagger.

* override to get green CI.

* Fix to have no content type when request body is empty (#7343)

* Correct return types for list_secrets, _versions (#7268)

* More useful exceptions for Key Vault errors (#7086)

* Fix to have no content type when request body is empty

* [Blob] Set tier and rehydrate (#7269)

* [BlockBlob][Tier]Add BlobTier Support For CopyBlob/PutBlob/PutBlockList

* [BlockBlob][Tier]Add Recordings for Standard BlobTier Support

* [BlockBlob][Rehydrate]Enable Rehydrate Priority for Copy/SetTier

* [BlockBlob][Rehydrate]Recordings for Rehydrate Priority for Copy/SetTier

* [Blob] Echo client (#7350)

* [EchoClientId]Verify Client Request ID in Response Same as in Request

* [BlockBlob][Rehydrate]Fix Recordings

* Refactor max_connections to max_concurrency (#7531)

* Rename max_connections to max_concurrency

* Other non breaking changes

* comment changes

* Use generated version as constant API version (#7547)

* Re-recording from #7555 (#7585)

* Storage batch API (#7387)

* delete_blobs POC

* Storage POC WIP

* Storage POC async

* Storage WIP

* Storage clean-up

* Storage options

* Autorest remark

* Use config option

* SetBlobTier WIP

* Working and tested delete_blobs + options

* Set standard tier with tests

* Adapt tests for playback mode

* Move batch_send to common mixin

* Python 2.7 compat

* Fix snapshot bug

* Re-record

* Pass kwargs to batch

* Clean x-ms-client-request-id from body if multipart response

* remove duplicated comp=tier

* Disable batch tests on Python 2.7

* pylint

* Batch ChangeLog (#7593)

* [Storage Blob] Regenerate Swagger (#7572)

* Regenerate Swagger

* small fix

* adjusts literalincludes for msft docs (#7599)

* Download sparse blob (#7555)

* [PageBlob]Optimize Download Sparse File

* Breaking Changes - Permission models (#7517)

* Rename Permission related models to SasPermissions

* Rewrite Permission Models

* refactor tests

* some tests

* comment changes

* remove length checks

* QueuePermissions

* File Permissions

* blobs minor tweaks

* minor fix

* pylint fix

* pylint

* Fix batch docstrings (#7600)

* Fix batch docstrings

* pylint

* [Rename]rename max_connections to max_concurrency (#7606)

* small edits to lease client docs (#7550)

* small edits to lease client docs

* whitespace

* Setup identity for unified pipelines. (#7578)

* Setup keyvault for unified pipelines. (#7579)

* Setup eventhubs for publishing via unified pipelines. (#7580)

* Setup eventhubs for publishing via unified pipelines.

* Fixed missing punctuation.

* Setup cosmos for release via unified pipelines. (#7581)

* bump CI

* Manual network september (#7576)

* generated files

* adding 2019-07-01

* history and version

* rerun tests

* updated version

* removed subscription id from tests

* rerun tests again

* replaced subscription id

* [AutoPR] reservations/resource-manager (#7569)

* [AutoPR reservations/resource-manager] [Hub Generated] Review request for Microsoft.Capacity to add version preview/2019-04-01 (#7251)

* Generated from f19a2e5b7f384018b74b21b7b8b8782d95b456f9

fixed x-ms-enum value

* Generated from 2f56008117d578bec6cc8b8c832926f45a0fe52e

fixed catalog definition

* Generated from cac978330e8c7b9583812a735cfeac97fb267056

reverted breaking operation id change

* added version and history

* [AutoPR] sql/resource-manager (#7535)

* Generated from 9402dbf3fb3d5fffcb80f501b1fc857ff97ab723 (#7329)

Fixing PR validation errors

* regenerated

* history and version

* Setup app configuration with unified pipelines. (#7582)

* [AutoPR] resources/resource-manager (#7549)

* regenerated

* updated history and version

* updated date

* Correct import_key parameter type (#7590)

* add doc for tracing (#7616)

* Add a from_blob_url method (#7567)

* Add a from_blob_url method

* Update tests

* Recordings - common blob

* Revert "Recordings - common blob"

This reverts commit ad07f34c25f7662cbaed8b33ac05a1a6f6270dae.

* comment changes

* undo get_client changes

* redo recordings

* from_container_url

* fix tests

* tests fix for container url

* pylint fix

* Fix some docstring

* docstrings

* Doc imprvment for Storage (#7601)

* Docstring improvement

* File docstring

* Queue docstrings

* More doc fixes

* More doc fixes

* Revert "small edits to lease client docs (#7550)" (#7631)

This reverts commit ed20b58.

* kwarg-ify methods (#7611)

* changeset1

* changeset-2

* changeset-3

* changeset-4

* minor fix

* some final tweaks

* pylint

* minor fixes

* max_concurrency

* lease_id

* pylint

* fix

* [Storage] Consolidate offset and range parameters (#7598)

* Updated blob sync tests

* Offset refactor APIS (#19)

* rename to offset and length

* fix

* more changes

* Updated blob sync tests

* Temp block async tests

* Fix upload page behaviour

* Fix clear page behaviour

* Update from_url offset behaviour

* Update page ranges

* Fix download blob behaviour

* Some cleanup

* Fixed page size

* More test fixes

* Some more fixes

* Fixed page tests

* Fixed encryption tests

* Fix common test

* append anf page blob async

* some more test fixes

* Fix live tests

* more changes async

* Fix sparse blob test

* Last tests

* update docstrings

* pylint

* Some Final tweaks (#7653)

* Some final tweaks

* comments

* Fix live tests (#7665)

* fix live tests

* oops

* few changes
YijunXieMS pushed a commit to YijunXieMS/azure-sdk-for-python that referenced this pull request Oct 9, 2019
Storage Preview 4 release (Azure#7527)
2a501c9 
* Update msrest and regenerate swagger. (Azure#7308)

* Update msrest and regenerate swagger.

* override to get green CI.

* Fix to have no content type when request body is empty (Azure#7343)

* Correct return types for list_secrets, _versions (Azure#7268)

* More useful exceptions for Key Vault errors (Azure#7086)

* Fix to have no content type when request body is empty

* [Blob] Set tier and rehydrate (Azure#7269)

* [BlockBlob][Tier]Add BlobTier Support For CopyBlob/PutBlob/PutBlockList

* [BlockBlob][Tier]Add Recordings for Standard BlobTier Support

* [BlockBlob][Rehydrate]Enable Rehydrate Priority for Copy/SetTier

* [BlockBlob][Rehydrate]Recordings for Rehydrate Priority for Copy/SetTier

* [Blob] Echo client (Azure#7350)

* [EchoClientId]Verify Client Request ID in Response Same as in Request

* [BlockBlob][Rehydrate]Fix Recordings

* Refactor max_connections to max_concurrency (Azure#7531)

* Rename max_connections to max_concurrency

* Other non breaking changes

* comment changes

* Use generated version as constant API version (Azure#7547)

* Re-recording from Azure#7555 (Azure#7585)

* Storage batch API (Azure#7387)

* delete_blobs POC

* Storage POC WIP

* Storage POC async

* Storage WIP

* Storage clean-up

* Storage options

* Autorest remark

* Use config option

* SetBlobTier WIP

* Working and tested delete_blobs + options

* Set standard tier with tests

* Adapt tests for playback mode

* Move batch_send to common mixin

* Python 2.7 compat

* Fix snapshot bug

* Re-record

* Pass kwargs to batch

* Clean x-ms-client-request-id from body if multipart response

* remove duplicated comp=tier

* Disable batch tests on Python 2.7

* pylint

* Batch ChangeLog (Azure#7593)

* [Storage Blob] Regenerate Swagger (Azure#7572)

* Regenerate Swagger

* small fix

* adjusts literalincludes for msft docs (Azure#7599)

* Download sparse blob (Azure#7555)

* [PageBlob]Optimize Download Sparse File

* Breaking Changes - Permission models (Azure#7517)

* Rename Permission related models to SasPermissions

* Rewrite Permission Models

* refactor tests

* some tests

* comment changes

* remove length checks

* QueuePermissions

* File Permissions

* blobs minor tweaks

* minor fix

* pylint fix

* pylint

* Fix batch docstrings (Azure#7600)

* Fix batch docstrings

* pylint

* [Rename]rename max_connections to max_concurrency (Azure#7606)

* small edits to lease client docs (Azure#7550)

* small edits to lease client docs

* whitespace

* Setup identity for unified pipelines. (Azure#7578)

* Setup keyvault for unified pipelines. (Azure#7579)

* Setup eventhubs for publishing via unified pipelines. (Azure#7580)

* Setup eventhubs for publishing via unified pipelines.

* Fixed missing punctuation.

* Setup cosmos for release via unified pipelines. (Azure#7581)

* bump CI

* Manual network september (Azure#7576)

* generated files

* adding 2019-07-01

* history and version

* rerun tests

* updated version

* removed subscription id from tests

* rerun tests again

* replaced subscription id

* [AutoPR] reservations/resource-manager (Azure#7569)

* [AutoPR reservations/resource-manager] [Hub Generated] Review request for Microsoft.Capacity to add version preview/2019-04-01 (Azure#7251)

* Generated from f19a2e5b7f384018b74b21b7b8b8782d95b456f9

fixed x-ms-enum value

* Generated from 2f56008117d578bec6cc8b8c832926f45a0fe52e

fixed catalog definition

* Generated from cac978330e8c7b9583812a735cfeac97fb267056

reverted breaking operation id change

* added version and history

* [AutoPR] sql/resource-manager (Azure#7535)

* Generated from 9402dbf3fb3d5fffcb80f501b1fc857ff97ab723 (Azure#7329)

Fixing PR validation errors

* regenerated

* history and version

* Setup app configuration with unified pipelines. (Azure#7582)

* [AutoPR] resources/resource-manager (Azure#7549)

* regenerated

* updated history and version

* updated date

* Correct import_key parameter type (Azure#7590)

* add doc for tracing (Azure#7616)

* Add a from_blob_url method (Azure#7567)

* Add a from_blob_url method

* Update tests

* Recordings - common blob

* Revert "Recordings - common blob"

This reverts commit ad07f34c25f7662cbaed8b33ac05a1a6f6270dae.

* comment changes

* undo get_client changes

* redo recordings

* from_container_url

* fix tests

* tests fix for container url

* pylint fix

* Fix some docstring

* docstrings

* Doc imprvment for Storage (Azure#7601)

* Docstring improvement

* File docstring

* Queue docstrings

* More doc fixes

* More doc fixes

* Revert "small edits to lease client docs (Azure#7550)" (Azure#7631)

This reverts commit ed20b58.

* kwarg-ify methods (Azure#7611)

* changeset1

* changeset-2

* changeset-3

* changeset-4

* minor fix

* some final tweaks

* pylint

* minor fixes

* max_concurrency

* lease_id

* pylint

* fix

* [Storage] Consolidate offset and range parameters (Azure#7598)

* Updated blob sync tests

* Offset refactor APIS (Azure#19)

* rename to offset and length

* fix

* more changes

* Updated blob sync tests

* Temp block async tests

* Fix upload page behaviour

* Fix clear page behaviour

* Update from_url offset behaviour

* Update page ranges

* Fix download blob behaviour

* Some cleanup

* Fixed page size

* More test fixes

* Some more fixes

* Fixed page tests

* Fixed encryption tests

* Fix common test

* append anf page blob async

* some more test fixes

* Fix live tests

* more changes async

* Fix sparse blob test

* Last tests

* update docstrings

* pylint

* Some Final tweaks (Azure#7653)

* Some final tweaks

* comments

* Fix live tests (Azure#7665)

* fix live tests

* oops

* few changes
fengzhou-msft pushed a commit that referenced this pull request Nov 5, 2019
@fengzhou-msft
Storage Preview 4 release (#7527)
1eca87d 
* Update msrest and regenerate swagger. (#7308)

* Update msrest and regenerate swagger.

* override to get green CI.

* Fix to have no content type when request body is empty (#7343)

* Correct return types for list_secrets, _versions (#7268)

* More useful exceptions for Key Vault errors (#7086)

* Fix to have no content type when request body is empty

* [Blob] Set tier and rehydrate (#7269)

* [BlockBlob][Tier]Add BlobTier Support For CopyBlob/PutBlob/PutBlockList

* [BlockBlob][Tier]Add Recordings for Standard BlobTier Support

* [BlockBlob][Rehydrate]Enable Rehydrate Priority for Copy/SetTier

* [BlockBlob][Rehydrate]Recordings for Rehydrate Priority for Copy/SetTier

* [Blob] Echo client (#7350)

* [EchoClientId]Verify Client Request ID in Response Same as in Request

* [BlockBlob][Rehydrate]Fix Recordings

* Refactor max_connections to max_concurrency (#7531)

* Rename max_connections to max_concurrency

* Other non breaking changes

* comment changes

* Use generated version as constant API version (#7547)

* Re-recording from #7555 (#7585)

* Storage batch API (#7387)

* delete_blobs POC

* Storage POC WIP

* Storage POC async

* Storage WIP

* Storage clean-up

* Storage options

* Autorest remark

* Use config option

* SetBlobTier WIP

* Working and tested delete_blobs + options

* Set standard tier with tests

* Adapt tests for playback mode

* Move batch_send to common mixin

* Python 2.7 compat

* Fix snapshot bug

* Re-record

* Pass kwargs to batch

* Clean x-ms-client-request-id from body if multipart response

* remove duplicated comp=tier

* Disable batch tests on Python 2.7

* pylint

* Batch ChangeLog (#7593)

* [Storage Blob] Regenerate Swagger (#7572)

* Regenerate Swagger

* small fix

* adjusts literalincludes for msft docs (#7599)

* Download sparse blob (#7555)

* [PageBlob]Optimize Download Sparse File

* Breaking Changes - Permission models (#7517)

* Rename Permission related models to SasPermissions

* Rewrite Permission Models

* refactor tests

* some tests

* comment changes

* remove length checks

* QueuePermissions

* File Permissions

* blobs minor tweaks

* minor fix

* pylint fix

* pylint

* Fix batch docstrings (#7600)

* Fix batch docstrings

* pylint

* [Rename]rename max_connections to max_concurrency (#7606)

* small edits to lease client docs (#7550)

* small edits to lease client docs

* whitespace

* Setup identity for unified pipelines. (#7578)

* Setup keyvault for unified pipelines. (#7579)

* Setup eventhubs for publishing via unified pipelines. (#7580)

* Setup eventhubs for publishing via unified pipelines.

* Fixed missing punctuation.

* Setup cosmos for release via unified pipelines. (#7581)

* bump CI

* Manual network september (#7576)

* generated files

* adding 2019-07-01

* history and version

* rerun tests

* updated version

* removed subscription id from tests

* rerun tests again

* replaced subscription id

* [AutoPR] reservations/resource-manager (#7569)

* [AutoPR reservations/resource-manager] [Hub Generated] Review request for Microsoft.Capacity to add version preview/2019-04-01 (#7251)

* Generated from f19a2e5b7f384018b74b21b7b8b8782d95b456f9

fixed x-ms-enum value

* Generated from 2f56008117d578bec6cc8b8c832926f45a0fe52e

fixed catalog definition

* Generated from cac978330e8c7b9583812a735cfeac97fb267056

reverted breaking operation id change

* added version and history

* [AutoPR] sql/resource-manager (#7535)

* Generated from 9402dbf3fb3d5fffcb80f501b1fc857ff97ab723 (#7329)

Fixing PR validation errors

* regenerated

* history and version

* Setup app configuration with unified pipelines. (#7582)

* [AutoPR] resources/resource-manager (#7549)

* regenerated

* updated history and version

* updated date

* Correct import_key parameter type (#7590)

* add doc for tracing (#7616)

* Add a from_blob_url method (#7567)

* Add a from_blob_url method

* Update tests

* Recordings - common blob

* Revert "Recordings - common blob"

This reverts commit ad07f34c25f7662cbaed8b33ac05a1a6f6270dae.

* comment changes

* undo get_client changes

* redo recordings

* from_container_url

* fix tests

* tests fix for container url

* pylint fix

* Fix some docstring

* docstrings

* Doc imprvment for Storage (#7601)

* Docstring improvement

* File docstring

* Queue docstrings

* More doc fixes

* More doc fixes

* Revert "small edits to lease client docs (#7550)" (#7631)

This reverts commit ed20b58.

* kwarg-ify methods (#7611)

* changeset1

* changeset-2

* changeset-3

* changeset-4

* minor fix

* some final tweaks

* pylint

* minor fixes

* max_concurrency

* lease_id

* pylint

* fix

* [Storage] Consolidate offset and range parameters (#7598)

* Updated blob sync tests

* Offset refactor APIS (#19)

* rename to offset and length

* fix

* more changes

* Updated blob sync tests

* Temp block async tests

* Fix upload page behaviour

* Fix clear page behaviour

* Update from_url offset behaviour

* Update page ranges

* Fix download blob behaviour

* Some cleanup

* Fixed page size

* More test fixes

* Some more fixes

* Fixed page tests

* Fixed encryption tests

* Fix common test

* append anf page blob async

* some more test fixes

* Fix live tests

* more changes async

* Fix sparse blob test

* Last tests

* update docstrings

* pylint

* Some Final tweaks (#7653)

* Some final tweaks

* comments

* Fix live tests (#7665)

* fix live tests

* oops

* few changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@annatisch annatisch annatisch approved these changes

@rakshith91 rakshith91 Awaiting requested review from rakshith91

@zezha-msft zezha-msft Awaiting requested review from zezha-msft

+1 more reviewer

@xiafu-msft xiafu-msft xiafu-msft left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Client This issue points to a problem in the data-plane of the library. P0 Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@lmazuel @AutorestCI @annatisch @xiafu-msft