Skip to content

[8.17](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s#45095

Merged
pierrehilbert merged 3 commits into8.17from
mergify/bp/8.17/pr-45073
Jul 1, 2025
Merged

[8.17](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s#45095
pierrehilbert merged 3 commits into8.17from
mergify/bp/8.17/pr-45073

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify Bot commented Jun 27, 2025

See #36926. This fix has two components:

  • Return an error from Publish when the Elasticsearch output gets a 429 (too many requests) from Elasticsearch. This triggers a retry delay and reconnection attempt in the pipeline.
  • Break the backoff counters for Publish and Connect into separate values, so a successful Connect call (which for Elasticsearch just means that an empty http GET gave an ok response) doesn't reset the exponential backoff for bulk ingest requests when they are being throttled.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This comment on the issue has local testing instructions.

Related issues

See #36926. This fix has two components:
- Return an error from `Publish` when the Elasticsearch output gets a 429 (too many requests) from Elasticsearch. This triggers a retry delay and reconnection attempt in the pipeline.
- Break the backoff counters for `Publish` and `Connect` into separate values, so a successful `Connect` call (which for Elasticsearch just means that an empty http GET gave an ok response) doesn't reset the exponential backoff for bulk ingest requests when they are being throttled.

(cherry picked from commit 8b25d5b)

# Conflicts:
#	libbeat/outputs/elasticsearch/client.go
@mergify mergify Bot added backport conflicts There is a conflict in the backported pull request labels Jun 27, 2025
@mergify mergify Bot requested a review from a team as a code owner June 27, 2025 21:30
@mergify
Copy link
Copy Markdown
Contributor Author

mergify Bot commented Jun 27, 2025

Cherry-pick of 8b25d5b has failed:

On branch mergify/bp/8.17/pr-45073
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit 8b25d5b14.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   libbeat/outputs/backoff.go
	modified:   libbeat/outputs/elasticsearch/client_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/outputs/elasticsearch/client.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify Bot assigned faec Jun 27, 2025
@mergify mergify Bot removed the request for review from a team June 27, 2025 21:30
@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 27, 2025
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jun 29, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2025
@mergify
Copy link
Copy Markdown
Contributor Author

mergify Bot commented Jun 30, 2025

This pull request has not been merged yet. Could you please review and merge it @faec? 🙏

@pierrehilbert pierrehilbert merged commit 539693b into 8.17 Jul 1, 2025
170 of 173 checks passed
@pierrehilbert pierrehilbert deleted the mergify/bp/8.17/pr-45073 branch July 1, 2025 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants