Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: roundtrip FixedSizeList Scalar to protobuf #8239

Merged
merged 1 commit into from
Nov 17, 2023

Conversation

wjones127
Copy link
Member

Which issue does this PR close?

Closes #8224.

Rationale for this change

This scalar type was missing from the protobuf serialization. Adding it for completeness.

What changes are included in this PR?

Updates protobuf to add a FixedSizeList scalar variant. I've re-used the same serialization mechanism as list, to minimize the changes needed.

Are these changes tested?

I've added a roundtrip test, which I think should be sufficient. I'm not deeply familiar with the proto module codebase, so if you have additional suggestions for tests, I am all ears.

Are there any user-facing changes?

This should be a backwards compatible change to the protobuf messages.

@github-actions github-actions bot added documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Physical Expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait labels Nov 16, 2023
@wjones127 wjones127 changed the base branch from master to main November 16, 2023 17:10
@wjones127 wjones127 removed documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Physical Expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait labels Nov 16, 2023
@wjones127 wjones127 marked this pull request as ready for review November 16, 2023 17:52
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @wjones127

match value {
Value::ListValue(_) => Self::List(arr.to_owned()),
Value::FixedSizeListValue(_) => Self::FixedSizeList(arr.to_owned()),
_ => unreachable!(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in theory this code could be reached with malformed input -- it might be nice to return an error rather than panic

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which panic are you referring to?

If you mean the unreachable!(), this statement is inside a match guard that guarantees value is one of these two, so I think it is truly unreachable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI. For the sake of code robustness, I agree with @alamb 's suggestion to return an error instead of unreachable. If someone accidentally modifies the match branch, this panic might get exposed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see -- I missed the fact that this is in a match outer branch. I agree this code is really unreachable. Perhaps we can make it more robust per @yukkit 's and my suggestion in some later PR

@alamb
Copy link
Contributor

alamb commented Nov 17, 2023

Thanks @wjones127 and @yukkit

@alamb alamb merged commit 2c5e237 into apache:main Nov 17, 2023
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Protobuf serialization for FSL scalar
3 participants