[Cosmos] Resiliency and Documentation Improvements #36514

simorenoh · 2024-07-18T01:48:24Z

This PR improves and fixes some functionality that was missing in the SDK. Namely, this PR includes:

New logic to retry ServiceRequestErrors up to three times before failing the request, ensuring that requests that were not reaching the service can attempt to run more than once.
Bugfix on a client hang that would happen when switching write regions in an account (running into 403/3 write forbidden error). SDK no longer hangs attempting to resolve the locational endpoint and now properly routes requests. This also adds write failover logic to the SDK.
Adding retry mechanisms for 403/1008 database account not found errors.
Small enhancements to our session retry logic and global endpoint manager refreshes, namely updating the locking logic to the same as async (slightly faster in deciding whether lock is needed) and adding locational routing to 404/1002 retries.
Documentation updates on our client configurations, becoming more specific on what each setting does.
Fixes bug introduced in PR 31096 which removed 503 retries from the SDK.
Adds logic to properly refresh the location cache information in the SDK every 5 minutes.

…errors functional code, missing tests now

simorenoh · 2024-07-18T01:56:49Z

/azp run python - cosmos - tests

azure-pipelines · 2024-07-18T01:57:05Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-sdk · 2024-07-18T02:55:36Z

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-cosmos

simorenoh · 2024-07-18T18:13:03Z

/azp run python - cosmos - tests

azure-pipelines · 2024-07-18T18:13:20Z

Azure Pipelines successfully started running 1 pipeline(s).

sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py

sdk/cosmos/azure-cosmos/CHANGELOG.md

sdk/cosmos/azure-cosmos/azure/cosmos/_global_endpoint_manager.py

sdk/cosmos/azure-cosmos/azure/cosmos/_synchronized_request.py

sdk/cosmos/azure-cosmos/azure/cosmos/http_constants.py

simorenoh · 2024-07-24T20:11:54Z

/azp run python - cosmos - tests

azure-pipelines · 2024-07-24T20:12:09Z

Azure Pipelines successfully started running 1 pipeline(s).

xinlian12

LGTM, thanks

simorenoh · 2024-09-29T17:14:38Z

/azp run python - cosmos - tests

azure-pipelines · 2024-09-29T17:14:57Z

Azure Pipelines successfully started running 1 pipeline(s).

simorenoh · 2024-09-29T22:38:36Z

/azp run python - cosmos - tests

azure-pipelines · 2024-09-29T22:38:52Z

Azure Pipelines successfully started running 1 pipeline(s).

simorenoh · 2024-09-30T19:01:05Z

/azp run python - cosmos - tests

azure-pipelines · 2024-09-30T19:01:25Z

Azure Pipelines successfully started running 1 pipeline(s).

simorenoh · 2024-10-01T22:57:31Z

/azp run python - cosmos - tests

azure-pipelines · 2024-10-01T22:57:49Z

Azure Pipelines successfully started running 1 pipeline(s).

simorenoh · 2024-10-02T14:13:43Z

/azp run python - cosmos - tests

azure-pipelines · 2024-10-02T14:14:05Z

Azure Pipelines successfully started running 1 pipeline(s).

simorenoh · 2024-10-03T21:24:24Z

/azp run python - cosmos - tests

azure-pipelines · 2024-10-03T21:24:41Z

Azure Pipelines successfully started running 1 pipeline(s).

* 403.3 loop fix, regional routing fix, improvement on service request errors functional code, missing tests now * Update ErrorCodesAndRetries.md * Update TimeoutAndRetriesConfig.md * Update http_constants.py * Update CHANGELOG.md * test improvements for 403 retry * fix emulator tests * Update test_globaldb.py * Update test_globaldb.py * add ServiceRequestError test and doc update * addressing comments * Update test_globaldb.py * Update test_globaldb.py * Update test_globaldb.py * Update test_globaldb.py * move policy * revert * fixes * Update test_globaldb.py * Update test_globaldb.py * Update test_globaldb.py * Update test_globaldb.py * Update test_globaldb.py * Update CHANGELOG.md * 503 retries * align readme with changelog * forceful db account refresh * remove premature locational endpoint * make GEM refresh every 5 mins as it should have * Delete drz3-drill.txt * Update CHANGELOG.md * Update test_location_cache.py * Update _global_endpoint_manager.py * ensure only one initial database account call * Delete dr-zdrill-005.txt * Update test_location_cache.py * Update test_location_cache.py * Update test_location_cache.py * overhaul location_cache tests

403.3 loop fix, regional routing fix, improvement on service request …

fc5c059

…errors functional code, missing tests now

github-actions bot added the Cosmos label Jul 18, 2024

Update ErrorCodesAndRetries.md

be8d893

Update TimeoutAndRetriesConfig.md

7aabc1e

simorenoh added 2 commits July 18, 2024 08:29

Update http_constants.py

f7ae57b

Update CHANGELOG.md

94882e4

simorenoh marked this pull request as ready for review July 18, 2024 15:54

simorenoh requested review from Pilchie, annatisch, bambriz, kushagraThapar and xinlian12 as code owners July 18, 2024 15:54

bambriz reviewed Jul 19, 2024

View reviewed changes

sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py Show resolved Hide resolved

simorenoh added 5 commits July 19, 2024 16:33

test improvements for 403 retry

3f8006b

fix emulator tests

fca7aff

Update test_globaldb.py

60188bc

Update test_globaldb.py

45a0862

add ServiceRequestError test and doc update

cff2e25

xinlian12 reviewed Jul 23, 2024

View reviewed changes

simorenoh and others added 2 commits July 24, 2024 12:34

addressing comments

ef4aa85

Merge branch 'main' into retry_validations

b1f33f5

Update test_globaldb.py

2fba121

xinlian12 approved these changes Jul 24, 2024

View reviewed changes

simorenoh added 5 commits September 27, 2024 19:30

remove premature locational endpoint

f587f9c

make GEM refresh every 5 mins as it should have

3f5e6b0

Delete drz3-drill.txt

0bf94a3

Update CHANGELOG.md

85e96e4

Merge branch 'main' into retry_validations

1c9d828

simorenoh added 6 commits September 29, 2024 13:39

Update test_location_cache.py

95b8384

Update _global_endpoint_manager.py

3fae165

ensure only one initial database account call

2e34d59

Delete dr-zdrill-005.txt

0879b0c

Update test_location_cache.py

acfcb3c

Update test_location_cache.py

16bda74

Update test_location_cache.py

98df1b4

Merge branch 'main' into retry_validations

22faf8e

overhaul location_cache tests

629e919

simorenoh mentioned this pull request Oct 4, 2024

Cosmos DB Python SDK is not able to retry request in the new write region during failover #37721

Closed

simorenoh merged commit f8ab118 into Azure:main Oct 7, 2024

simorenoh deleted the retry_validations branch October 7, 2024 16:56

[Cosmos] Resiliency and Documentation Improvements #36514

[Cosmos] Resiliency and Documentation Improvements #36514

Uh oh!

Conversation

simorenoh commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simorenoh commented Jul 18, 2024

Uh oh!

azure-pipelines bot commented Jul 18, 2024

Uh oh!

azure-sdk commented Jul 18, 2024

Uh oh!

simorenoh commented Jul 18, 2024

Uh oh!

azure-pipelines bot commented Jul 18, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simorenoh commented Jul 24, 2024

Uh oh!

azure-pipelines bot commented Jul 24, 2024

Uh oh!

xinlian12 left a comment

Choose a reason for hiding this comment

Uh oh!

simorenoh commented Sep 29, 2024

Uh oh!

azure-pipelines bot commented Sep 29, 2024

Uh oh!

simorenoh commented Sep 29, 2024

Uh oh!

azure-pipelines bot commented Sep 29, 2024

Uh oh!

simorenoh commented Sep 30, 2024

Uh oh!

azure-pipelines bot commented Sep 30, 2024

Uh oh!

simorenoh commented Oct 1, 2024

Uh oh!

azure-pipelines bot commented Oct 1, 2024

Uh oh!

simorenoh commented Oct 2, 2024

Uh oh!

azure-pipelines bot commented Oct 2, 2024

Uh oh!

simorenoh commented Oct 3, 2024

Uh oh!

azure-pipelines bot commented Oct 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

simorenoh commented Jul 18, 2024 •

edited

Loading