[beatreceiver] Enable telemetry for indexing errors on ES exporter - and test to ensure sensitive logs are not shipped

[khushijain21]

What does this PR do?

This PR was already reviewed/approved here #8988. The original had to be closed due to security reason.

This PR enables telemetry logs for failed input docs on ES exporter.

This PR also adds a test to ensure sensitive logs from ES exporter or elastic-owned components are not shipped by filestream-monitoring.

---

And for sensitive logs from ES exporter are also blocked. The reasoning follows

If a document is failed to index for any reason, ES exporter logs the document at debug level.

The format looks something like this

 {"log.level":"debug",
 "@timestamp":"2025-07-14T06:35:24.235Z",
 "message":"failed to index document; input may contain sensitive data",
 "input":"{\"create\":{\"_index\":\"logs-test-default\"}}\n{\"host\":{\"hostname\":\"Khushis-MacBook-Pro.local\",\...}\n", 
 "index":"logs-test-default",
 "error.type":"strict_dynamic_mapping_exception",
 "error.reason":"",
 "ecs.version":"1.6.0",
}

Since the input field above contains a json document and filestream-monitoring's ndjson parser does not recurseively parse JSON - we can be sure this field in never indexed.

This test explicity ensure this behavior is always true

This test is a follow up as part of the discussion here #8738 (comment)

Why is it important?

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

source ./env.sh; AGENT_VERSION=9.2.0 go test -tags integration -run TestSensitiveLogsESExporter ./testing/integration/ess -count=1 -v

Related issues

• Closes [beat_receivers] Redact or drop logs that could contain event data in the monitoring filestream instance #8846

[khushijain21]

This PR is going to be flaky because of global paths in filebeat. Keeping this in draft mode until elastic/beats#44903 is resolved

[mergify]

This pull request does not have a backport label. Could you fix it @khushijain21? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: bf79e58

Failed CI Steps

• Unit tests - Windows 2016
• v1.28.15:amd64:basic,slim,complete,service,elastic-otel-collector

cc @khushijain21

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)