-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-18188][DOC][FOLLOW-UP]Add spark.broadcast.checksum to configuration
#21825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #93329 has finished for PR 21825 at commit
|
|
Are you sure this is supposed to be documented for end users? |
|
I think it's necessary to let the end users know: |
docs/configuration.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is going to be documented, it should be documented in a way that's useful for end users. This typically means describing what turning it on and off does from an end-user perspective. Does it make things faster, slower? more reliable, compatible, etc? This description here pretty much reiterates what's in the property key itself.
6a85aad to
cf27272
Compare
|
Test build #93823 has finished for PR 21825 at commit
|
docs/configuration.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nits like: space after a period, and the default is already documented above. I think this could still be more useful. What about: "If enabled, broadcasts will include a checksum, which can help detect corrupted blocks, at the cost of computing and sending a little more data. It's possible to disable it if the network has other mechanisms to guarantee data won't be corrupted during broadcast."
CC @davies . I guess even I'm not sure when I would disable this ... what would a network have to guarantee to avoid whatever corruption is possible here? Here it isn't clear yet when it's safe, when it won't lead to correctness issues.
cf27272 to
d18d1b2
Compare
|
Test build #93876 has finished for PR 21825 at commit
|
docs/configuration.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Final nit: add the space after the period.
d18d1b2 to
09cbe26
Compare
|
Test build #93919 has finished for PR 21825 at commit
|
|
Merged to master |
|
Thanks @srowen |
What changes were proposed in this pull request?
This pr add
spark.broadcast.checksumto configuration.How was this patch tested?
manually tested