Skip to content

[receiver/awss3] add tag_object_after_ingestion flag#46769

Merged
atoulme merged 5 commits into
open-telemetry:mainfrom
tommyzli:s3-receiver-delete-obj
Apr 2, 2026
Merged

[receiver/awss3] add tag_object_after_ingestion flag#46769
atoulme merged 5 commits into
open-telemetry:mainfrom
tommyzli:s3-receiver-delete-obj

Conversation

@tommyzli

@tommyzli tommyzli commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Description

introduce a flag for the receiver to make a best-effort attempt to tag the objects it ingests. This is useful for cost management, as lifecycle policies can be used to delete tagged objects

Link to tracking issue

Fixes #46078

Testing

new unit tests

@tommyzli tommyzli requested review from a team and atoulme as code owners March 9, 2026 16:47
@github-actions github-actions Bot requested a review from adcharre March 9, 2026 16:47
@tommyzli tommyzli force-pushed the s3-receiver-delete-obj branch from 4c2c9f0 to e1ee372 Compare March 9, 2026 18:40
@tommyzli tommyzli force-pushed the s3-receiver-delete-obj branch from e1ee372 to c31a255 Compare March 9, 2026 19:01
@atoulme

atoulme commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Please cover in tests and docs the behavior if the receiver next consumer returns an error on Consume* calls.

@tommyzli tommyzli changed the title [receiver/awss3] add delete_object_after_ingestion flag [receiver/awss3] add tag_object_after_ingestion flag Mar 13, 2026
@tommyzli

Copy link
Copy Markdown
Contributor Author

@atoulme as we discussed I've updated the implementation to tagging objects instead of deleting them. I'll follow up with another PR after this to add a flag to skip tagged objects (cost was one reason for deleting, but the other is to prevent re-ingesting the same object multiple times, most applicable to non-sqs mode)

@tommyzli tommyzli force-pushed the s3-receiver-delete-obj branch from 828080e to 25c90a0 Compare March 13, 2026 17:46
Comment thread receiver/awss3receiver/s3sqsreader.go
Comment thread receiver/awss3receiver/s3sqsreader_test.go Outdated
@atoulme atoulme merged commit 1b25cfc into open-telemetry:main Apr 2, 2026
191 checks passed
@otelbot

otelbot Bot commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution @tommyzli! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey. If you are getting started contributing, you can also join the CNCF Slack channel #opentelemetry-new-contributors to ask for guidance and get help.

araiu pushed a commit to araiu/opentelemetry-collector-contrib that referenced this pull request Apr 6, 2026
…46769)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
introduce a flag for the receiver to make a best-effort attempt to tag
the objects it ingests. This is useful for cost management, as lifecycle
policies can be used to delete tagged objects
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
open-telemetry#46078

<!--Describe what testing was performed and which tests were added.-->
#### Testing
new unit tests

<!--Please delete paragraphs that you did not use before submitting.-->
AndrewCharlesHay pushed a commit to AndrewCharlesHay/opentelemetry-collector-contrib that referenced this pull request Apr 23, 2026
…46769)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
introduce a flag for the receiver to make a best-effort attempt to tag
the objects it ingests. This is useful for cost management, as lifecycle
policies can be used to delete tagged objects
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
open-telemetry#46078

<!--Describe what testing was performed and which tests were added.-->
#### Testing
new unit tests

<!--Please delete paragraphs that you did not use before submitting.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[receiver/awss3] Clean up objects

4 participants