Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement retry mechanism #470

Merged
merged 2 commits into from
Jul 16, 2024
Merged

Conversation

jooola
Copy link
Member

@jooola jooola commented Jun 24, 2024

Closes #311

Implement a retry handler to retry requests based on an internal retry policy.

The retry policy will evolve over time to cover more use cases, but for now we want to start simple, and maybe someday allow users to tweak the retry policy.

BEGIN_COMMIT_OVERRIDE
feat: retry requests when the api gateway errors (#470)
feat: retry requests when the rate limit was reached (#470)
feat: retry requests when the network timed out (#470)
feat: respect cancelled contexts during retry sleep (#470)
END_COMMIT_OVERRIDE

Copy link

codecov bot commented Jun 24, 2024

Codecov Report

Attention: Patch coverage is 83.72093% with 7 lines in your changes missing coverage. Please review.

Project coverage is 71.56%. Comparing base (b3e91d9) to head (f895218).

Files Patch % Lines
hcloud/client_handler_retry.go 79.41% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #470      +/-   ##
==========================================
+ Coverage   71.54%   71.56%   +0.02%     
==========================================
  Files          45       45              
  Lines        3876     3904      +28     
==========================================
+ Hits         2773     2794      +21     
- Misses        690      696       +6     
- Partials      413      414       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hcloud/client_handler_retry.go Outdated Show resolved Hide resolved
hcloud/client_handler_retry.go Show resolved Hide resolved
@jooola
Copy link
Member Author

jooola commented Jul 9, 2024

I added a max retry attempts configuration, because we do not want to wait forever if we are stuck in a loop.

@jooola jooola marked this pull request as ready for review July 9, 2024 11:58
@jooola jooola requested a review from a team as a code owner July 9, 2024 11:58
hcloud/client_handler_retry.go Outdated Show resolved Hide resolved
hcloud/client.go Outdated Show resolved Hide resolved
hcloud/client_handler_retry.go Outdated Show resolved Hide resolved
hcloud/client_handler_retry.go Show resolved Hide resolved
hcloud/hcloud.go Outdated Show resolved Hide resolved
Co-authored-by: Julian Tölle <[email protected]>
@jooola jooola merged commit 756f605 into hetznercloud:main Jul 16, 2024
4 checks passed
@jooola jooola deleted the retry-handler branch July 16, 2024 09:50
jooola added a commit that referenced this pull request Jul 18, 2024
Related to #470

We prefer not to retry requests:
- If the ingress return a "bare" 429, as the backend was not able to respond anyway.
- If the ingress return a "bare" 503, to prevent possible self made ddos, and preventing a disaster recovery.
jooola added a commit that referenced this pull request Jul 18, 2024
Related to #470

We prefer not to retry requests:
- If the ingress return a "bare" 429, as the backend was not able to respond anyway.
- If the ingress return a "bare" 503, to prevent possible self made ddos, and preventing a disaster recovery.
jooola added a commit that referenced this pull request Jul 18, 2024
Related to #470

We prefer not to retry requests:
- If the ingress return a "bare" 429, as the backend was not able to respond anyway.
- If the ingress return a "bare" 503, to prevent possible self made ddos, and preventing a disaster recovery.
jooola added a commit that referenced this pull request Jul 19, 2024
Related to #470

This is only a `chore` to prevent confusion with the now updated
release-please commits in #470

We prefer not to retry requests:
- If the ingress returns a "bare" 429, as the backend was not able to
respond anyway.
- If the ingress returns a "bare" 503, to prevent possible self-made
DDoS, and preventing a disaster recovery.
jooola pushed a commit that referenced this pull request Jul 23, 2024
🤖 I have created a release *beep* *boop*
---


##
[2.11.0](v2.10.2...v2.11.0)
(2024-07-23)


### Features

* add truncated exponential backoff with full jitter
([#459](#459))
([fd1f46c](fd1f46c))
* allow configuring retry options
([#488](#488))
([2db9575](2db9575))
* **exp:** add sliceutil package
([#489](#489))
([f4ad6bc](f4ad6bc))
* **exp:** rename `*utils` package to `*util`
([#487](#487))
([19da475](19da475))
* respect cancelled contexts during retry sleep
([#470](#470))
([756f605](756f605))
* retry requests when the api gateway errors
([#470](#470))
([756f605](756f605))
* retry requests when the network timed out
([#470](#470))
([756f605](756f605))
* retry requests when the rate limit was reached
([#470](#470))
([756f605](756f605))


### Bug Fixes

* **exp:** set capacity for each batch
([#490](#490))
([57f53c1](57f53c1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
NachE added a commit to Develatio/nebulant-cli that referenced this pull request Aug 2, 2024
jooola added a commit to hetznercloud/hcloud-python that referenced this pull request Aug 6, 2024
Implement a retry handler to retry requests based on an internal retry
policy.

The retry policy will evolve over time to cover more use cases, but for
now we want to start simple, and maybe someday allow users to tweak the
retry policy.

Related to hetznercloud/hcloud-go#470

BEGIN_COMMIT_OVERRIDE
feat: retry requests when the api gateway errors (#430)
feat: retry requests when the network timed outs (#430)
feat: retry requests when the rate limit was reached (#430)
feat: retry requests when the api returns a conflict error (#430)
END_COMMIT_OVERRIDE

---------

Co-authored-by: Julian Tölle <[email protected]>
apricote pushed a commit to hetznercloud/fleeting-plugin-hetzner that referenced this pull request Sep 12, 2024
…1.0 (hetznercloud/fleeting-plugin-hetzner!98)

This MR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) | require | minor | `v2.10.2` -> `v2.11.0` |

---

### Release Notes

<details>
<summary>hetznercloud/hcloud-go (github.com/hetznercloud/hcloud-go/v2)</summary>

### [`v2.11.0`](https://github.com/hetznercloud/hcloud-go/releases/tag/v2.11.0)

[Compare Source](hetznercloud/hcloud-go@v2.10.2...v2.11.0)

##### Features

-   add truncated exponential backoff with full jitter ([#&#8203;459](hetznercloud/hcloud-go#459)) ([fd1f46c](hetznercloud/hcloud-go@fd1f46c))
-   allow configuring retry options ([#&#8203;488](hetznercloud/hcloud-go#488)) ([2db9575](hetznercloud/hcloud-go@2db9575))
-   **exp:** add sliceutil package ([#&#8203;489](hetznercloud/hcloud-go#489)) ([f4ad6bc](hetznercloud/hcloud-go@f4ad6bc))
-   **exp:** rename `*utils` package to `*util` ([#&#8203;487](hetznercloud/hcloud-go#487)) ([19da475](hetznercloud/hcloud-go@19da475))
-   respect cancelled contexts during retry sleep ([#&#8203;470](hetznercloud/hcloud-go#470)) ([756f605](hetznercloud/hcloud-go@756f605))
-   retry requests when the api gateway errors ([#&#8203;470](hetznercloud/hcloud-go#470)) ([756f605](hetznercloud/hcloud-go@756f605))
-   retry requests when the network timed out ([#&#8203;470](hetznercloud/hcloud-go#470)) ([756f605](hetznercloud/hcloud-go@756f605))
-   retry requests when the rate limit was reached ([#&#8203;470](hetznercloud/hcloud-go#470)) ([756f605](hetznercloud/hcloud-go@756f605))

##### Bug Fixes

-   **exp:** set capacity for each batch ([#&#8203;490](hetznercloud/hcloud-go#490)) ([57f53c1](hetznercloud/hcloud-go@57f53c1))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
@s4ke s4ke mentioned this pull request Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Retry transient HTTP Errors
2 participants