Fix reading specifying format for date partition projection in Hive connector #25817
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chenjian2664
force-pushed
the
ref_date_projection
branch
from
af8efa8 to
6ef1fa6
Compare
pettyjamesm
reviewed
May 19, 2025
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/DateProjection.java
Outdated
Show resolved
Hide resolved
chenjian2664
force-pushed
the
ref_date_projection
branch
from
6ef1fa6 to
bc09276
Compare
pettyjamesm
reviewed
May 27, 2025
plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
chenjian2664
force-pushed
the
ref_date_projection
branch
5 times, most recently
from
e60ed0f to
6c9b3ea
Compare
Member
|
Hi folks - any progress on this one? I'm writing an article explaining how to use Trino to query access logs, and I've had to include a caveat explaining that partition projection currently doesn't work, so time-based queries scan all rows, rather than just those included by the query conditions. |
Replace legacy formatting with `DateTimeFormatter` for better compatibility with the `java.time` package, providing more precise and predictable behavior This refactor follows AWS Athena's date-type partition projection format (https://docs.aws.amazon.com/athena/latest/ug/partition-projection-supported-types.html) in the description. Additionally, this commit replaces the error-prone Supplier<Instant> with a direct Instant type for `leftBound` and `rightBound`.
chenjian2664
force-pushed
the
ref_date_projection
branch
from
6c9b3ea to
ab6238b
Compare
chenjian2664
force-pushed
the
ref_date_projection
branch
2 times, most recently
from
3d352bd to
e8232e7
Compare
pettyjamesm
reviewed
Jun 23, 2025
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/Projection.java
Outdated
Show resolved
Hide resolved
pettyjamesm
reviewed
Jun 26, 2025
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/DateProjection.java
Outdated
Show resolved
Hide resolved
chenjian2664
force-pushed
the
ref_date_projection
branch
from
e8232e7 to
19099b7
Compare
Contributor
Author
|
@pettyjamesm comments addressed, PTAL |
pettyjamesm
reviewed
Jul 10, 2025
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/Projection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
chenjian2664
force-pushed
the
ref_date_projection
branch
2 times, most recently
from
145fd3e to
2e4699f
Compare
pettyjamesm
reviewed
Jul 14, 2025
Member
pettyjamesm
left a comment
pettyjamesm
left a comment
There was a problem hiding this comment.
Mostly looks good to me, test coverage looks good. Some code style comments / nits.
plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/projection/PartitionProjection.java
Outdated
Show resolved
Hide resolved
chenjian2664
force-pushed
the
ref_date_projection
branch
from
2e4699f to
3be16bc
Compare
chenjian2664
force-pushed
the
ref_date_projection
branch
from
3be16bc to
bf5c8a3
Compare
pettyjamesm
approved these changes
Jul 16, 2025
Member
pettyjamesm
left a comment
pettyjamesm
left a comment
There was a problem hiding this comment.
The changes here look good to me, thanks for taking this over the finish line @chenjian2664!
Contributor
Author
|
@pettyjamesm I really appreciate your guidance and the time you took to assist me |
Member
|
Thanks for your work on this, @chenjian2664. I'll test it against the Backblaze B2 Bucket Access Log and let you know how it works out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Closes #25642
This PR assumes that the underlying partition paths already follow the user-defined format, specifically for date projection.
If the projection
storage.location.templateisn't set, orstorage.location.templateis set and compatible with the Hive path -- the partition paths follow the default Hive format:partName=${partValue}. In this case,partValueis escaped for special characters (e.g., / becomes %2F), same as the existing writing logic in Hive today.If the
storage.location.templateis set, NOT compatible with the hive format, the partition value is inserted into the path that follow the template(or hive format) WITHOUT escaped special characters:• With a custom date projection format and a partition column
dtformatted asyyyy/MM/dd, this PR allows reading the partition path:dt=2015%2F10%2F10• If the
storage.location.templateis set to something like/aaa/bbb/${dt}-xxx, this PR enables reading the partition path:/aaa/bbb/2015/10/10-xxxThis PR not support writing
DateProjectionwith user-defined format that not compatible with Hive date/timestamp format.Additional context and related issues
Alternative for #25657
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: