findinpath: Support OR-ed condition in Delta checkpoint iterator#17
Closed
findinpath wants to merge 106 commits intomasterfrom
Closed
findinpath: Support OR-ed condition in Delta checkpoint iterator#17findinpath wants to merge 106 commits intomasterfrom
findinpath wants to merge 106 commits intomasterfrom
Conversation
findinpath
changed the title
findinpath
force-pushed
the
findinpath/delta-checkpoint-entry-or
branch
from
dcc9c6d to
6d8e652
Compare
Typically metadata files should be accessed once, so prefer `.add(x)` over `.addCopies(x, 1)`, so that `.addCopies` stand out as potentially something to address.
Once the metadata & protocol entries are found, the scanning of multi-part checkpoint files can be stopped.
The `metadata` & `protocol` entries are already read (and saved) once when retrieving the table handle. Reuse this information while retrieving the active files for the table.
findinpath
force-pushed
the
findinpath/delta-checkpoint-entry-or
branch
from
6d8e652 to
b00803c
Compare
It was found to be confusing.
Adds SESSION_AUTHORIZATION to ClientCapabilities, so that the client side support for this feature can be identified using the headers sent to Trino.
This upgrades Delta Lake version to 3.0.0.
Refactor TestRetryingConnectionFactory and TestLazyConnectionFactory to use Dependency Injection Motivation behind this change is to be able to later add dependent objects to tested class without a need for major changes in the test class itself.
Extend the logic to specially treat metadata queries and not allocate any memory for those, so they are scheduled immediatelly, to cover all schematas from system catalog (previously only jdbc schama was covered).
findinpath
force-pushed
the
findinpath/delta-checkpoint-entry-or
branch
2 times, most recently
from
e134ec5 to
000da7e
Compare
This bit of code is making the assumption that it's safe to remove the Sort because there are no exchanges yet. However, it's forgetting the fact that the data needs to be sorted, so later when the exchange is added this expectation is broken.
Implements column-wise hash calculations in FlatHashCompiler and changes FlatGroupByHash to use it to implement a batched approach to first computing a range of position hashes and then attempting to insert those positions using the hashes that were precomputed. By calculating position hashes in a columnar tarversal path, we can avoid repeated expensive bounds checking and allow the JIT to unroll loops into much more efficient forms. Additionally, when attempting to insert the positions into the FlatGroupByHash after precomputing the hash code we can start loading the relevant portition of the hash table into memory sooner since we aren't intermixing computing the hash with the memory accesses.
These implementations delegate to Block, but the block implementation for these types do not support getSlice
The connector runs connector smoke tests with HMS.
The synthetic dictionary, rle, and lazy blocks require special handling in the codebase, and new synthetic blocks cannot reasonable be added without updating the existing code. This commit blocks direct extensions of Block, only allowing extensions to ValueBlock. Seal Block interface Mark ValueBlock interface non-sealed Mark all current ValueBlock implementations as final
Co-authored-by: Yuya Ebihara <ebyhry@gmail.com>
findinpath
force-pushed
the
findinpath/delta-checkpoint-entry-or
branch
from
d03c3fe to
8c9986f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Alternative for trinodb#19240
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: