Skip to content

Use by default ZSTD Parquet compression codec for Delta Lake #17426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Dec 24, 2024
Merged

Use by default ZSTD Parquet compression codec for Delta Lake #17426

merged 2 commits into from
Dec 24, 2024

Conversation

findinpath
Copy link
Contributor

@findinpath findinpath commented May 10, 2023 •
edited by raunaqmorarka
Loading

Description

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Delta Lake
* Improve performance of reading from delta lake tables by compressing files with `ZSTD` by default ({issue}`17426`)

@cla-bot cla-bot bot added the cla-signed label May 10, 2023
@findinpath findinpath self-assigned this May 10, 2023
@findepi
Copy link
Member

findepi commented May 10, 2023

/test-with-secrets sha=3a66eaefac8743b41b53098bad3d31a28a9a00df

findepi
findepi reviewed May 10, 2023
View reviewed changes
@github-actions github-actions bot added delta-lake Delta Lake connector docs tests:hive labels May 10, 2023
@findinpath findinpath force-pushed the findinpath/delta-parquet-zstd branch from 3a66eae to db6136c Compare May 10, 2023 15:13
@findinpath findinpath requested review from ebyhr and findepi May 10, 2023 15:15
@findinpath
Copy link
Contributor Author

@findepi pls run the PR again with secrets

@github-actions
Copy link

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4936844641

@ebyhr
Copy link
Member

ebyhr commented May 10, 2023

/test-with-secrets sha=db6136c7157a841809d0ae9ed4ff9702754bc141

@findepi
Copy link
Member

findepi commented May 11, 2023

/test-with-secrets sha=db6136c7157a841809d0ae9ed4ff9702754bc141

did it run?

@ebyhr
Copy link
Member

ebyhr commented May 11, 2023

Yes, it was green in https://github.com/trinodb/trino/actions/runs/4944033785

findepi
findepi reviewed May 12, 2023
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@findinpath findinpath force-pushed the findinpath/delta-parquet-zstd branch from db6136c to 94f4831 Compare May 14, 2023 15:12
@findinpath findinpath requested a review from findepi May 14, 2023 15:14
@ebyhr
Copy link
Member

ebyhr commented May 14, 2023 •
edited
Loading

/test-with-secrets sha=94f48315b8c97489c48e0962ec2922d7b47f75da

https://github.com/trinodb/trino/actions/runs/4975087974

@jordandakota
Copy link

What compression level (1-22) is used with Trino zstd?

@github-actions GitHub Actions
Copy link

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

@github-actions github-actions bot added the stale label Jan 15, 2024
@mosabua
Copy link
Member

mosabua commented Jan 15, 2024

@findinpath could you rebase and then work with @raunaqmorarka @ebyhr and others as needed?

@github-actions github-actions bot removed the stale label Jan 16, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 7, 2024

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

@github-actions github-actions bot added the stale label Feb 7, 2024
@github-actions GitHub Actions
Copy link

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

@github-actions github-actions bot closed this Feb 29, 2024
@ebyhr ebyhr mentioned this pull request Dec 20, 2024
Merged
@raunaqmorarka
Copy link
Member

Now that we have native compression and decompression for ZSTD and #24553 to deprecate DB 9.1, I think this PR can go ahead.

@raunaqmorarka raunaqmorarka reopened this Dec 23, 2024
@github-actions github-actions bot removed the stale label Dec 23, 2024
@findinpath findinpath force-pushed the findinpath/delta-parquet-zstd branch from 94f4831 to 45552cb Compare December 24, 2024 11:22
@raunaqmorarka
Copy link
Member

/test-with-secrets sha=45552cb195d3d4dfa9e3b10998cbef11f319e8a9

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 24, 2024 •
edited
Loading

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/12481074009

@findinpath findinpath force-pushed the findinpath/delta-parquet-zstd branch from 45552cb to d7e9722 Compare December 24, 2024 12:42
@raunaqmorarka raunaqmorarka force-pushed the findinpath/delta-parquet-zstd branch from d7e9722 to f2d09ac Compare December 24, 2024 17:17
@raunaqmorarka
Copy link
Member

/test-with-secrets sha=f2d09ac4fd93a0a5edc638606953352c877d9280

@github-actions GitHub Actions
Copy link

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/12484498657

@raunaqmorarka raunaqmorarka merged commit 3257342 into trinodb:master Dec 24, 2024
50 checks passed
@github-actions github-actions bot added this to the 469 milestone Dec 24, 2024
@mosabua mosabua mentioned this pull request Dec 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raunaqmorarka raunaqmorarka raunaqmorarka approved these changes

@ebyhr ebyhr Awaiting requested review from ebyhr

@findepi findepi Awaiting requested review from findepi

Assignees

@findinpath findinpath

Labels

cla-signed delta-lake Delta Lake connector docs performance

Milestone

469

Development

Successfully merging this pull request may close these issues.

6 participants

@findinpath @findepi @ebyhr @jordandakota @mosabua @raunaqmorarka