Core: Add list/map block sizes #10973
Conversation
This PR switches the encoder to the `BlockingDirectBinaryEncoder` that actually encodes the size of list and map objects in the Avro file. When doing projected reads, it can skip of the fields that are not relevant, without having to skip over each of the element or entry. The `ManifestWriteBenchmark` shows a slight decrease in write speed: ``` Before: 22.51099815 s/op with 0.1981214357719109 error After: 22.2674114166 s/op with 0.47593907631201676 error ```
|
Cool! I'm glad to see this coming along. |
| writer.setMeta(entry.getKey(), entry.getValue()); | ||
| } | ||
|
|
||
| writer.setEncoder(b -> EncoderFactory.get().blockingDirectBinaryEncoder(b, null)); |
There was a problem hiding this comment.
Should we also do it here:
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This still seems to be pending. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
Seems like its still pending. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
3 participants
This PR switches the encoder to the
BlockingDirectBinaryEncoderthat actually encodes the size of list and map objects in the Avro file.When doing projected reads, it can skip of the fields that are not relevant, without having to skip over each of the element or entry.
The
ManifestWriteBenchmarkshows a slight decrease in write speed:cc @rustyconover @aokolnychyi