Skip to content

Avro: Fix pruning columns when a logical-map array's value type is nested#1321

Merged
rdblue merged 1 commit intoapache:masterfrom
lxynov:arraymap
Aug 14, 2020
Merged

Avro: Fix pruning columns when a logical-map array's value type is nested#1321
rdblue merged 1 commit intoapache:masterfrom
lxynov:arraymap

Conversation

@lxynov
Copy link
Contributor

@lxynov lxynov commented Aug 11, 2020

Added a unit test which fails without the change


Consider type map(long, list<long>) (a logical-map array in Iceberg-Avro) and a full projection.

In GetProjectedIds::map, the key id and value id won't be added if the value type is nested.

  public Set<Integer> map(Types.MapType map, Set<Integer> keyResult, Set<Integer> valueResult) {
    if (valueResult == null) {
      for (Types.NestedField field : map.fields()) {
        fieldIds.add(field.fieldId());
      }
    }
    return fieldIds;
  }

Missing the key id can pose a problem in PruneColumns::array, since the array's child record's key field will be omitted.

Copy link
Contributor

@rdsr rdsr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@rdblue rdblue merged commit 14ad4db into apache:master Aug 14, 2020
aokolnychyi pushed a commit to aokolnychyi/iceberg that referenced this pull request Aug 18, 2020
cmathiesen pushed a commit to ExpediaGroup/iceberg that referenced this pull request Aug 19, 2020
@lxynov lxynov deleted the arraymap branch September 16, 2020 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants