Skip to content

[arrow-vector] Zero-copy transfer of DecimalVector and Decimal256Vector is losing nullability information #692

@bodduv

Description

@bodduv

Describe the bug, including details regarding any error messages, version, and platform.

In the main branch, DecimalVector and Decimal256Vector are losing nullability information during a zero-copy transfer. It currently forces a nullable FieldType onto the "to" ValueVector.

The deficiency in the code is in DecimalVector.TransferImpl class when trying to get a transfer getTransferPair(String, BufferAllocator) creating a DecimalVector with a nullable FieldType (although the "from" ValueVector's FieldType is non-nullable):

private class TransferImpl implements TransferPair {
DecimalVector to;
public TransferImpl(String ref, BufferAllocator allocator) {
to =
new DecimalVector(ref, allocator, DecimalVector.this.precision, DecimalVector.this.scale);
}

Note that all the other (at least the primitive) ValueVectors are transferred without losing nullability information by using the "from" ValueVector's field. E.g.:

private class TransferImpl implements TransferPair {
BigIntVector to;
public TransferImpl(String ref, BufferAllocator allocator) {
to = new BigIntVector(ref, field.getFieldType(), allocator);
}


To reproduce the issue: add the following test in src/test/java/org/apache/arrow/vector/TestDecimalVector.java to see it fail.

  @Test
  public void testGetTransferPairWithoutFieldNonNullable() {
    final FieldType decimalNonNullableType =
        new FieldType(false, new ArrowType.Decimal(10, scale), null);
    final DecimalVector fromVector =
        new DecimalVector("decimal", decimalNonNullableType, allocator);
    final TransferPair transferPair =
        fromVector.getTransferPair(fromVector.getField().getName(), allocator);
    final DecimalVector toVector = (DecimalVector) transferPair.getTo();

    // These asserts fail
    assertSame(fromVector.getField().isNullable(), toVector.getField().isNullable());
    assertSame(fromVector.getField().getFieldType(), toVector.getField().getFieldType());
  }

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions