Skip to content

Conversation

@Chen-Yuan-Lai
Copy link
Contributor

Which issue does this PR close?

Closes #7099 .

Rationale for this change

As #7099 says, allocated_size() in GenericByteViewBuilder needs to be adjusted since allocated_size() in NullBufferBuilder returns bits, not bytes.

What changes are included in this PR?

Are there any user-facing changes?

  1. Divide the allocated_size by 8.
  2. Add related test

@github-actions github-actions bot added the arrow Changes to the arrow crate label Feb 10, 2025
@Chen-Yuan-Lai Chen-Yuan-Lai changed the title bugfix: adjust allocated_size() in GenericByteViewBuilde bugfix: adjust allocated_size() in GenericByteViewBuilder Feb 10, 2025
Copy link
Contributor

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Member

@mbrobbel mbrobbel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should re-consider https://github.com/apache/arrow-rs/pull/7089/files#r1945293119, because having allocated_size return both bytes and bits may cause confusion; this PR is an example.

@tustvold
Copy link
Contributor

IMO allocated_size being in bits is a bug and should be fixed - #7089 (comment).

Allocations can't be made at bit granularity and so the notion this method should return in bits seems odd to me

@tustvold
Copy link
Contributor

I've filed #7121 to change the allocated_size to be in bytes, going to mark this as a draft in the meantime

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory accounting in GenericBytesView overcounts allocation size

4 participants