Fix GlueHiveMetastore#batchGetPartition to handle unprocessedKeys#10696
Fix GlueHiveMetastore#batchGetPartition to handle unprocessedKeys#10696findepi merged 1 commit intotrinodb:masterfrom
Conversation
|
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Rentao Wu.
|
cacts
force-pushed
the
handle-glue-unprocessed-keys
branch
from
5279525 to
aab1128
Compare
|
Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to cla@trino.io. For more information, see https://github.com/trinodb/cla. |
|
@cactauz thank you for your PR. Please ping me once you have the CLA signed. |
There was a problem hiding this comment.
would it make sense here to log the fact that Glue API returned UnprocessedKeys ?
BatchGetPartitionsRequest could potentially not return all requested partition values. If Glue returns incomplete partitions, the query will error out. This change is to handle unprocessedKeys from BatchGetParititonsResults.
There was a problem hiding this comment.
partitionValueLists -> pendingPartitions
There was a problem hiding this comment.
batchGetPartitionResult.getPartitions().isEmpty() implies that !batchGetPartitionResult.getUnprocessedKeys().isEmpty(), or Glue service is cheating (or we have wrong assumptions how it works)
findepi
force-pushed
the
handle-glue-unprocessed-keys
branch
from
aab1128 to
cf160e9
Compare
|
Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to cla@trino.io. For more information, see https://github.com/trinodb/cla. |
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
| // In the unlikely scenario where batchGetPartition call cannot make progress on retrieving partitions, avoid infinite loop | ||
| if (partitions.isEmpty()) { | ||
| verify(!unprocessedKeys.isEmpty(), "Empty unprocessedKeys for non-empty BatchGetPartitionRequest and empty partitions result"); | ||
| throw new TrinoException(HIVE_METASTORE_ERROR, "Cannot make progress retrieving partitions. Unable to retrieve partitions: " + unprocessedKeys); |
There was a problem hiding this comment.
Seems like Glue doesn't guarantee there will be at least one partition processed.
See #12516
4 participants
Cherry-picked and updated for presto -> trino changes from original PR #6155. @rentaow please let me know if you'd like to update yours instead and I'll close this :)
BatchGetPartitionsRequest could potentially not return all requested partition values. These unprocessed keys are returned in a separate field and can be retrieved successfully on subsequent requests. If these keys are ignored, this results in a HIVE_PARTITION_DROPPED_DURING_QUERY exception.