-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-45429][CORE] Add helper classes for SSL RPC communication #43244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
...twork-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java
Outdated
Show resolved
Hide resolved
...twork-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java
Outdated
Show resolved
Hide resolved
...twork-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java
Outdated
Show resolved
Hide resolved
...-common/src/test/java/org/apache/spark/network/protocol/EncryptedMessageWithHeaderSuite.java
Outdated
Show resolved
Hide resolved
c70acd2 to
eaa7ec2
Compare
mridulm
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While waiting for tests, please fix the nit as well.
It wont impact the test result anyway
...twork-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java
Show resolved
Hide resolved
|
|
|
Can you retrigger the tests please ? There are a bunch of failures. |
|
done, will re-request review once failures are minimized |
|
@mridulm I believe the only failure now is |
|
There was a failure in Thanks for working on this @hasnain-db ! |
What changes were proposed in this pull request?
This PR adds helper classes for SSL RPC communication that are needed to work around the fact that
nettydoes not support zero-copy transfers.These mirror the existing
MessageWithHeaderandMessageEncoderclasses with very minor differences. But the differences were just enough that it didn't seem easy to refactor/consolidate, and since we don't expect these classes to change much I hope it's ok.Why are the changes needed?
These are needed to support transferring
ManagedBuffers into a form that can be transferred bynettyover the network, since netty's encryption support does not support zero-copy transfers.Does this PR introduce any user-facing change?
No
How was this patch tested?
Added unit tests
The rest of the changes and integration were tested as part of #42685
Was this patch authored or co-authored using generative AI tooling?
No