[Feature] Support glue hive metastore #12250
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imay
reviewed
Oct 17, 2022
Because we use the AWS glue data catalog as the metastore for Hive here, it's just have special "hive.metastore.type" which is "glue", but it still need use hive connector/catalog to get meta info |
| * a) Hides the non-metastore related operations present in AWSGlue | ||
| * b) Hides away the batching and pagination related limitations of AWSGlue | ||
| */ | ||
| public interface AWSGlueMetastore { |
There was a problem hiding this comment.
Why not reuse IHiveMetastore
There was a problem hiding this comment.
This interface is AWSCatalogMetastoreClient's internal, and AWSCatalogMetastoreClient is the client of IHiveMetastore（such like HiveMetaStore). This interface should retrun the meta info of glue instead of the meta info of SR.
|
run starrocks_fe_unittest |
|
starrocks_be_unittest succeeded. |
|
starrocks_be_unittest succeeded. |
|
starrocks_be_unittest succeeded. |
|
starrocks_be_unittest succeeded. |
|
starrocks_be_unittest succeeded. |
|
run starrocks_admit_test |
| @@ -35,7 +30,7 @@ public DeltaLakeInternalMgr(String catalogName, Map<String, String> properties) | |||
|
|
|||
| public IHiveMetastore createHiveMetastore() { | |||
| // TODO(stephen): Abstract the creator class to construct hive meta client | |||
| HiveMetaClient metaClient = createHiveMetaClient(); | |||
| HiveMetaClient metaClient = HiveMetaClient.createHiveMetaClient(properties); | |||
There was a problem hiding this comment.
My origin idea was to construct the hive conf in an external component.
fe/fe-core/src/main/java/com/starrocks/external/hive/HiveMetastoreApiConverter.java
Show resolved
Hide resolved
| @@ -79,11 +87,17 @@ public static HiveTable toHiveTable(Table table, String catalogName) { | |||
| .setFullSchema(toFullSchemasForHiveTable(table)) | |||
| .setTableLocation(table.getSd().getLocation()) | |||
| .setCreateTime(table.getCreateTime()); | |||
|
|
|||
| Map<String, String> tableParams = table.getParameters() == null ? ImmutableMap.of() : table.getParameters(); | |||
| if (isDeltaLakeTable(tableParams)) { | |||
There was a problem hiding this comment.
abstract a function named 'toTableLocation()'. we may try our best to keep clear here.
| properties.forEach(conf::set); | ||
| if (!properties.containsKey(HIVE_METASTORE_URIS)) { | ||
| // set value for compatible with the rollback | ||
| properties.put(HIVE_METASTORE_URIS, DUMMY_THRIFT_URI); |
There was a problem hiding this comment.
why use this ？ The properties must have "hive.metastore.uris".
There was a problem hiding this comment.
glue doesn't have this property
|
starrocks_be_unittest succeeded. |
|
starrocks_be_unittest succeeded. |
|
run starrocks_admit_test |
stephen-shelby
previously approved these changes
Oct 26, 2022
|
starrocks_be_unittest succeeded. |
|
SonarCloud Quality Gate failed. |
|
run starrocks_fe_unittest |
stephen-shelby
approved these changes
Oct 26, 2022
dirtysalt
approved these changes
Oct 26, 2022
|
run starrocks_admit_test |
gengjun-git
approved these changes
Oct 26, 2022
|
run starrocks_fe_unittest |
1 similar comment
|
run starrocks_fe_unittest |
[FE PR Coverage Check]😞 fail : 26 / 2204 (01.18%) file detail
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this：
Which issues of this PR fixes ：
#12249
In This PR, we support create hive type catalog whcih metasotre type is glue, you can create the catalog like this
If the data file is stored in S3, also need S3 configuration, refer to https://docs.starrocks.io/zh-cn/latest/using_starrocks/External_table#aws-s3tencent-cloud-cos%E6%94%AF%E6%8C%81
Problem Summary(Required) ：
Checklist: