HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by transfer manager #7048
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
force-pushed
the
s3/HADOOP-19272-xfer-manager-logging
branch
from
48e3762 to
1306ef4
Compare
|
🎊 +1 overall
This message was automatically generated. |
Disables all logging below error in the AWS SDK Transfer Manager. This is done in ClientManagerImpl construction so is automatically done during S3A FS initialization. ITests verify that * It is possible to restore the warning log. This verifies the validity of the test suite, and will identify when an SDK update fixes this regression. * Constructing an S3A FS instance will disable the logging. The log manipulation code is lifted from Cloudstore, where it was used to dynamically enable logging. It uses reflection to load the Log4J binding; all uses of the API catch and swallow exceptions. This is needed to avoid failures when running against different log backends This is an emergency fix -we could come up with a better design for the reflection based code using the new DynMethods classes. But this is based on working code, which is always good. Change-Id: I51b9fc40bd3c7d7df55258889811df6a9cb0ba9a
steveloughran
force-pushed
the
s3/HADOOP-19272-xfer-manager-logging
branch
from
ae12e8c to
da44d53
Compare
|
🎊 +1 overall
This message was automatically generated. |
ahmarsuhail
approved these changes
Sep 18, 2024
Contributor
There was a problem hiding this comment.
+1, LGTM.
What is the plan for this long term though? At some point we will upgrade the SDK, where this message has been downgraded to a debug. And then testNoisyLogging will start failing. So then do we revert or fix the test?
EDIT: I think we want to disable transfer manager logs permanently, so it'll just be fixing the test.
mukund-thakur
approved these changes
Sep 18, 2024
Contributor
mukund-thakur
left a comment
mukund-thakur
left a comment
There was a problem hiding this comment.
+1 overall looks good tome as well.
And yes we need a plan after AWS upgrade.
Contributor
Author
the test failure will be how we verify that all is good. at which point we can modify the test to fail if any warnings come from the manager.
I'm going to propose moving off it entirely
also I no longer trust the code. |
steveloughran
added a commit
that referenced
this pull request
Sep 19, 2024
#7048) Disables all logging below error in the AWS SDK Transfer Manager. This is done in ClientManagerImpl construction so is automatically done during S3A FS initialization. ITests verify that * It is possible to restore the warning log. This verifies the validity of the test suite, and will identify when an SDK update fixes this regression. * Constructing an S3A FS instance will disable the logging. The log manipulation code is lifted from Cloudstore, where it was used to dynamically enable logging. It uses reflection to load the Log4J binding; all uses of the API catch and swallow exceptions. This is needed to avoid failures when running against different log backends This is an emergency fix -we could come up with a better design for the reflection based code using the new DynMethods classes. But this is based on working code, which is always good. Contributed by Steve Loughran
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Disables all logging by the AWS SDK Transfer Manager.
This is done in ClientManagerImpl construction so is automatically done during S3AFS initialization.
ITests verify that
The log manipulation code is lifted from Cloudstore, where it was used to dynamically enable logging. It uses reflection to load the Log4J binding; all uses of the API catch and swallow exceptions.
This is needed to avoid failures when running against different log backends
This is an emergency fix -we could come up with a better design for the reflection based code using the new DynMethods classes. But this is based on working code, which is always good.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?