Skip to content

Upgrade Parquet to version 1.12.2#54

Merged
aweisberg merged 2 commits intoprestodb:masterfrom
shangxinli:parquet.1.12
Mar 10, 2022
Merged

Upgrade Parquet to version 1.12.2#54
aweisberg merged 2 commits intoprestodb:masterfrom
shangxinli:parquet.1.12

Conversation

@shangxinli
Copy link
Copy Markdown
Contributor

No description provided.

@shangxinli
Copy link
Copy Markdown
Contributor Author

@aweisberg, can we build a staging jar to have a try in presto repo, like we did when upgrading to 1.11.x? Thanks in advance!

@aweisberg
Copy link
Copy Markdown
Contributor

I deployed hive-apache-3.0.0-8-20220307.153144-32

Thanks!

@shangxinli
Copy link
Copy Markdown
Contributor Author

Thanks @aweisberg! I created PR with this version but found building errors because of the zstd-jni duplicate but version mismatch. So I just add a fix here to exclude zstd-jni in parquet. Can you help to release another new version to try out?

@aweisberg
Copy link
Copy Markdown
Contributor

Deployed 3.0.0-8-20220309.171836-34

@shangxinli
Copy link
Copy Markdown
Contributor Author

Thanks. This new version succeeded (PR). Can you merge and release an official version?

Copy link
Copy Markdown
Contributor

@aweisberg aweisberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, LGTM. Will release.

@aweisberg aweisberg merged commit 38afa73 into prestodb:master Mar 10, 2022
@aweisberg
Copy link
Copy Markdown
Contributor

I just released 3.0.0-8, it should appear in mirrors soon.

@7c00
Copy link
Copy Markdown
Member

7c00 commented Mar 17, 2022

@shangxinli Do we really need to upgrade Avro? After upgrading avro to 1.10.1, Presto fails to query Hudi MOR tables. The reason is that there is a breaking change in GenericData class

Old version:

    @Override
    public Object get(String key) {
      Field field = schema.getField(key);
      if (field == null)
        return null;
      return values[field.pos()];
    }

New version:

    @Override
    public Object get(String key) {
      Field field = schema.getField(key);
      if (field == null) {
        throw new AvroRuntimeException("Not a valid schema field: " + key);
      }
      return values[field.pos()];
    }

ref:

prestodb/presto#17463
https://github.com/prestodb/presto/runs/5581838178?check_suite_focus=true

CC: @codope @pratyakshsharma

@shangxinli
Copy link
Copy Markdown
Contributor Author

In that case, we can upgrade avro in Hudi to support that? I guess we need to upgrade sooner or later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants