[9.1] (backport #9341) [beatreceiver] Enable telemetry for indexing errors on ES exporter - and test to ensure sensitive logs are not shipped

[mergify]

What does this PR do?

This PR was already reviewed/approved here #8988. The original had to be closed due to security reason.

This PR enables telemetry logs for failed input docs on ES exporter.

This PR also adds a test to ensure sensitive logs from ES exporter or elastic-owned components are not shipped by filestream-monitoring.

---

And for sensitive logs from ES exporter are also blocked. The reasoning follows

If a document is failed to index for any reason, ES exporter logs the document at debug level.

The format looks something like this

 {"log.level":"debug",
 "@timestamp":"2025-07-14T06:35:24.235Z",
 "message":"failed to index document; input may contain sensitive data",
 "input":"{\"create\":{\"_index\":\"logs-test-default\"}}\n{\"host\":{\"hostname\":\"Khushis-MacBook-Pro.local\",\...}\n", 
 "index":"logs-test-default",
 "error.type":"strict_dynamic_mapping_exception",
 "error.reason":"",
 "ecs.version":"1.6.0",
}

Since the input field above contains a json document and filestream-monitoring's ndjson parser does not recurseively parse JSON - we can be sure this field in never indexed.

This test explicity ensure this behavior is always true

This test is a follow up as part of the discussion here #8738 (comment)

Why is it important?

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

source ./env.sh; AGENT_VERSION=9.2.0 go test -tags integration -run TestSensitiveLogsESExporter ./testing/integration/ess -count=1 -v

Related issues

• Relates [beat_receivers] Redact or drop logs that could contain event data in the monitoring filestream instance #8846

---

This is an automatic backport of pull request #9341 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of eb098ec has failed:

On branch mergify/bp/9.1/pr-9341
Your branch is up to date with 'origin/9.1'.

You are currently cherry-picking commit eb098ec53.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/otel/translate/otelconfig.go
	modified:   pkg/testing/fixture.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/otel/translate/otelconfig_test.go
	both modified:   testing/integration/ess/beat_receivers_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @khushijain21? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @khushijain21? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @khushijain21? 🙏

[ebeahan]

Is this backport waiting on other dependencies to go into 9.1? Else what's the status?

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: dd05322

Failed CI Steps

• Unit tests - Windows 10
• Win2022:non-sudo:default
• fips:arm64:sudo-true:fleet

History

• 💔 Build #29363 failed 462dd7d
• 💔 Build #29309 failed 8802f6e
• 💔 Build #29085 failed 63f8b4e
• 💔 Build #28209 failed 7a7bedb
• 💔 Build #28206 failed 67c7a0e

cc @khushijain21