Skip to content

[9.2] (backport #9341) [beatreceiver] Enable telemetry for indexing errors on ES exporter - and test to ensure sensitive logs are not shipped#10377

Merged
khushijain21 merged 1 commit into9.2from
mergify/bp/9.2/pr-9341
Oct 7, 2025
Merged

[9.2] (backport #9341) [beatreceiver] Enable telemetry for indexing errors on ES exporter - and test to ensure sensitive logs are not shipped#10377
khushijain21 merged 1 commit into9.2from
mergify/bp/9.2/pr-9341

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Oct 7, 2025

What does this PR do?

This PR was already reviewed/approved here #8988. The original had to be closed due to security reason.

This PR enables telemetry logs for failed input docs on ES exporter.

This PR also adds a test to ensure sensitive logs from ES exporter or elastic-owned components are not shipped by filestream-monitoring.


And for sensitive logs from ES exporter are also blocked. The reasoning follows

If a document is failed to index for any reason, ES exporter logs the document at debug level.

The format looks something like this

 {"log.level":"debug",
 "@timestamp":"2025-07-14T06:35:24.235Z",
 "message":"failed to index document; input may contain sensitive data",
 "input":"{\"create\":{\"_index\":\"logs-test-default\"}}\n{\"host\":{\"hostname\":\"Khushis-MacBook-Pro.local\",\...}\n", 
 "index":"logs-test-default",
 "error.type":"strict_dynamic_mapping_exception",
 "error.reason":"",
 "ecs.version":"1.6.0",
}

Since the input field above contains a json document and filestream-monitoring's ndjson parser does not recurseively parse JSON - we can be sure this field in never indexed.

This test explicity ensure this behavior is always true

This test is a follow up as part of the discussion here #8738 (comment)

Why is it important?

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

How to test this PR locally

source ./env.sh; AGENT_VERSION=9.2.0 go test -tags integration -run TestSensitiveLogsESExporter ./testing/integration/ess -count=1 -v

Related issues


This is an automatic backport of pull request #9341 done by [Mergify](https://mergify.com).

…and test to ensure sensitive logs are not shipped (#9341)

* [beatreceiver] Enable telemetry for indexing errors on ES exporter - and test to ensure sensitive logs are not shipped

* remove extra line

(cherry picked from commit eb098ec)
@mergify mergify bot added the backport label Oct 7, 2025
@mergify mergify bot requested a review from a team as a code owner October 7, 2025 10:37
@mergify mergify bot added the backport label Oct 7, 2025
@mergify mergify bot requested review from pkoutsovasilis and rubenruizdegauna and removed request for a team October 7, 2025 10:37
@github-actions github-actions bot added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Oct 7, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@khushijain21 khushijain21 enabled auto-merge (squash) October 7, 2025 11:00
@khushijain21 khushijain21 merged commit d59dd9a into 9.2 Oct 7, 2025
24 checks passed
@khushijain21 khushijain21 deleted the mergify/bp/9.2/pr-9341 branch October 7, 2025 12:38
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

cc @khushijain21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants