Fix Elasticsearch output retry backoff when receiving 429s

[faec]

See #36926. This fix has two components:

• Return an error from Publish when the Elasticsearch output gets a 429 (too many requests) from Elasticsearch. This triggers a retry delay and reconnection attempt in the pipeline.
• Break the backoff counters for Publish and Connect into separate values, so a successful Connect call (which for Elasticsearch just means that an empty http GET gave an ok response) doesn't reset the exponential backoff for bulk ingest requests when they are being throttled.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This comment on the issue has local testing instructions.

Related issues

• Fixes Publish() should backoff if Elasticsearch returns 429 HTTP rate limiting responses #36926.

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @faec? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

Code changes LGTM. I'm wondering about cost/benefit of a test to make sure we keep the behavior in the future. What do you think?

[faec]

I'm wondering about cost/benefit of a test to make sure we keep the behavior in the future. What do you think?

Yeah, I waffled about this, since it's one simple check in the middle of a function that expects a live connection. But ok, I split off the return value into a helper function based on the computed stats, and unit tested that, which is pretty simple but will keep someone from accidentally skipping the check if they reorganize that section of the code.

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

[mergify]

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

Details

• #45095 [8.17](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s has been created for branch 8.17 but encountered conflicts
• #45096 [8.18](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s has been created for branch 8.18 but encountered conflicts
• #45097 [8.19](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s has been created for branch 8.19
• #45099 [9.0](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s has been created for branch 9.0 but encountered conflicts
• #45098 [9.1](backport #45073) Fix Elasticsearch output retry backoff when receiving 429s has been created for branch 9.1