Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Region provider dominates log output #1184

Open
sergiimk opened this issue Sep 5, 2024 · 1 comment
Open

Region provider dominates log output #1184

sergiimk opened this issue Sep 5, 2024 · 1 comment
Labels
bug This issue is a bug. p3 This is a minor priority issue

Comments

@sergiimk
Copy link

sergiimk commented Sep 5, 2024

Describe the bug

This line adds a Debug representation of a dyn ProvideRegion as an argument of a tracing span.

Providers such as DefaultRegionChain are highly nested objects, Debug representation of which can be multiple pages of text long.

Tracing arguments in many configurations of tracing are be printed per every every log event as context.

As a result, every log message produced within this instrumented span creates a wall of output.

Expected Behavior

Tracing spans only include short arguments.

Current Behavior

When log level is set to info we get a 2-page log message per new / close span events.

Under debug logging our server logs are 90% dominated by init_s3_context{..}:load_region{...}. One S3 read produces hundreds of pages of this:
image

Reproduction Steps

n/a

Possible Solution

If you believe that full Debug representation of a provider is useful - instead of adding it to the span it could be logged as an event, so that it only appears in logs once.

Additional Information/Context

No response

Version

`main` branch

Environment details (OS name and version, etc.)

irrelevant

Logs

No response

@sergiimk sergiimk added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 5, 2024
@ysaito1001
Copy link
Collaborator

@sergiimk Thank you for bringing this to our attention! I see that the attached wall of text primarily comes from runtime plugins and runtime components associated with an IMDS client used internally. We have attempted to reduce log verbosity in the past (example PR), but it's clear that further reductions are needed.

@ysaito1001 ysaito1001 removed the needs-triage This issue or PR still needs to be triaged. label Sep 5, 2024
@jmklix jmklix added the p3 This is a minor priority issue label Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p3 This is a minor priority issue
Projects
None yet
Development

No branches or pull requests

3 participants