-
Notifications
You must be signed in to change notification settings - Fork 95
Description
Describe the bug, including details regarding any error messages, version, and platform.
In the main branch, DecimalVector and Decimal256Vector are losing nullability information during a zero-copy transfer. It currently forces a nullable FieldType onto the "to" ValueVector.
The deficiency in the code is in DecimalVector.TransferImpl class when trying to get a transfer getTransferPair(String, BufferAllocator) creating a DecimalVector with a nullable FieldType (although the "from" ValueVector's FieldType is non-nullable):
arrow-java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java
Lines 563 to 569 in 48a20ba
| private class TransferImpl implements TransferPair { | |
| DecimalVector to; | |
| public TransferImpl(String ref, BufferAllocator allocator) { | |
| to = | |
| new DecimalVector(ref, allocator, DecimalVector.this.precision, DecimalVector.this.scale); | |
| } |
Note that all the other (at least the primitive) ValueVectors are transferred without losing nullability information by using the "from" ValueVector's field. E.g.:
arrow-java/vector/src/main/java/org/apache/arrow/vector/BigIntVector.java
Lines 319 to 324 in 48a20ba
| private class TransferImpl implements TransferPair { | |
| BigIntVector to; | |
| public TransferImpl(String ref, BufferAllocator allocator) { | |
| to = new BigIntVector(ref, field.getFieldType(), allocator); | |
| } |
To reproduce the issue: add the following test in src/test/java/org/apache/arrow/vector/TestDecimalVector.java to see it fail.
@Test
public void testGetTransferPairWithoutFieldNonNullable() {
final FieldType decimalNonNullableType =
new FieldType(false, new ArrowType.Decimal(10, scale), null);
final DecimalVector fromVector =
new DecimalVector("decimal", decimalNonNullableType, allocator);
final TransferPair transferPair =
fromVector.getTransferPair(fromVector.getField().getName(), allocator);
final DecimalVector toVector = (DecimalVector) transferPair.getTo();
// These asserts fail
assertSame(fromVector.getField().isNullable(), toVector.getField().isNullable());
assertSame(fromVector.getField().getFieldType(), toVector.getField().getFieldType());
}