HADOOP-19348. Integrate analytics accelerator into S3A. #7433
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
force-pushed
the
aal-integration
branch
from
944fbbb to
a8da9b2
Compare
4 tasks
Contributor
|
ok, squash didn't take either |
Contributor
|
build was terminated after 12 hours starts ends raise on common-dev, assume it's a yetus/config/timeout issue. |
steveloughran
approved these changes
Feb 26, 2025
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1 pending that yetus. I don't want to ignore it as it may not be a transient failure, just something that tips yetus over the edge.
Contributor
Author
|
weird that this one is also checking out: |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Feb 27, 2025
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
- spelling in variable name to fix
- test tuning with 1 extra assert
+1 pending these
| isCSEEnabled = CSEUtils.isCSEEnabled(getS3EncryptionAlgorithm().getMethod()); | ||
|
|
||
| isAnalyticsAccelaratorEnabled = StreamIntegration.determineInputStreamType(conf).equals(InputStreamType.Analytics); | ||
| isAnalyticsAccelaratorEnabled = StreamIntegration.determineInputStreamType(conf) |
|
|
||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); | ||
| } | ||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); |
Contributor
There was a problem hiding this comment.
add a check for the filesystem iostats too, to make sure it trickles up
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
commented
Feb 27, 2025
|
|
||
| verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1); | ||
|
|
||
| verifyStatisticCounterValue(getFileSystem().getIOStatistics(), |
Contributor
Author
There was a problem hiding this comment.
@steveloughran can't get this one to work..
Contributor
Author
There was a problem hiding this comment.
Ok think it's to do with the test base class maybe, and these one's don't aggregate to the FS. I added an assertion in ITestS3AOpenCost which extends AbstractS3ACostTest and can see it get's propagated.
s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:dumpFileSystemIOStatistics(130)) - Aggregate FileSystem Statistics counters=((action_file_opened=1)
(action_http_head_request=3)
(audit_request_execution=13)
....
(stream_read_analytics_opened=1)
....
Think this is enough for now, and I'll remove this assertion and merge as is. Once we support statistics properly, we can add in tests in ITestS3AOpenCost
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
YanivKunda
pushed a commit
to YanivKunda/hadoop
that referenced
this pull request
Mar 23, 2025
Initial support for Analytics Accelerator Library, which provides a new parquet aware input stream. Contributed by: Ahmar Suhail Reviewed by: Steve Loughran
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Copy of #7334, checking if a new PR from a new branch keeps yetus happy.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?