Fix table listing in Iceberg to skip non-Iceberg tables #1354
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| /* | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package io.prestosql.plugin.hive.metastore; | ||
|
|
||
| import com.fasterxml.jackson.annotation.JsonCreator; | ||
| import com.fasterxml.jackson.annotation.JsonProperty; | ||
|
|
||
| import javax.annotation.concurrent.Immutable; | ||
|
|
||
| import java.util.Objects; | ||
|
|
||
| @Immutable | ||
| public class TablesWithParameterCacheKey | ||
| { | ||
| private final String databaseName; | ||
| private final String parameterKey; | ||
| private final String parameterValue; | ||
|
|
||
| @JsonCreator | ||
| public TablesWithParameterCacheKey( | ||
| @JsonProperty("databaseName") String databaseName, | ||
| @JsonProperty("parameterKey") String parameterKey, | ||
| @JsonProperty("parameterValue") String parameterValue) | ||
| { | ||
| this.databaseName = databaseName; | ||
| this.parameterKey = parameterKey; | ||
| this.parameterValue = parameterValue; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public String getDatabaseName() | ||
| { | ||
| return databaseName; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public String getParameterKey() | ||
| { | ||
| return parameterKey; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public String getParameterValue() | ||
| { | ||
| return parameterValue; | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object o) | ||
| { | ||
| if (this == o) { | ||
| return true; | ||
| } | ||
| if (o == null || getClass() != o.getClass()) { | ||
| return false; | ||
| } | ||
|
|
||
| TablesWithParameterCacheKey other = (TablesWithParameterCacheKey) o; | ||
| return Objects.equals(databaseName, other.databaseName) && | ||
| Objects.equals(parameterKey, other.parameterKey) && | ||
| Objects.equals(parameterValue, other.parameterValue); | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() | ||
| { | ||
| return Objects.hash(databaseName, parameterKey, parameterValue); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -73,6 +73,7 @@ | |
| import java.util.Set; | ||
| import java.util.concurrent.atomic.AtomicInteger; | ||
| import java.util.function.Function; | ||
| import java.util.regex.Pattern; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| import static com.google.common.base.Preconditions.checkArgument; | ||
|
|
@@ -122,6 +123,9 @@ public class ThriftHiveMetastore | |
| private final AtomicInteger chosenGetTableAlternative = new AtomicInteger(Integer.MAX_VALUE); | ||
| private final AtomicInteger chosenTableParamAlternative = new AtomicInteger(Integer.MAX_VALUE); | ||
|
|
||
| private static final Pattern TABLE_PARAMETER_SAFE_KEY_PATTERN = Pattern.compile("^[a-zA-Z_]+$"); | ||
|
Member
There was a problem hiding this comment. Is these a reference to what are the safe values for HMS?
Member
Author
There was a problem hiding this comment. I don't think there is one. See #1354 (comment).
Member
There was a problem hiding this comment. @phd3 this is documented in the code (where the constant is actually used, so there is a little bit more context). |
||
| private static final Pattern TABLE_PARAMETER_SAFE_VALUE_PATTERN = Pattern.compile("^[a-zA-Z0-9]*$"); | ||
|
|
||
| @Inject | ||
| public ThriftHiveMetastore(MetastoreLocator metastoreLocator, ThriftHiveMetastoreConfig thriftConfig) | ||
| { | ||
|
|
@@ -208,6 +212,27 @@ public List<String> getAllTables(String databaseName) | |
| } | ||
| } | ||
|
|
||
| @Override | ||
| public List<String> getTablesWithParameter(String databaseName, String parameterKey, String parameterValue) | ||
| { | ||
| try { | ||
| return retry() | ||
| .stopOn(UnknownDBException.class) | ||
| .stopOnIllegalExceptions() | ||
| .run("getTablesWithParameter", stats.getGetTablesWithParameter().wrap( | ||
| () -> doGetTablesWithParameter(databaseName, parameterKey, parameterValue))); | ||
| } | ||
| catch (UnknownDBException e) { | ||
| return ImmutableList.of(); | ||
| } | ||
| catch (TException e) { | ||
| throw new PrestoException(HIVE_METASTORE_ERROR, e); | ||
| } | ||
| catch (Exception e) { | ||
| throw propagate(e); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public Optional<Table> getTable(String databaseName, String tableName) | ||
| { | ||
|
|
@@ -718,7 +743,7 @@ public List<String> getAllViews(String databaseName) | |
| .stopOn(UnknownDBException.class) | ||
| .stopOnIllegalExceptions() | ||
| .run("getAllViews", stats.getGetAllViews().wrap( | ||
|
electrum marked this conversation as resolved.
Outdated
|
||
| () -> doGetTablesWithParameter(databaseName, PRESTO_VIEW_FLAG, "true"))); | ||
| } | ||
| catch (UnknownDBException e) { | ||
| return ImmutableList.of(); | ||
|
|
@@ -731,16 +756,25 @@ public List<String> getAllViews(String databaseName) | |
| } | ||
| } | ||
|
|
||
|
lxynov marked this conversation as resolved.
Outdated
|
||
| private List<String> doGetTablesWithParameter(String databaseName, String parameterKey, String parameterValue) | ||
| throws TException | ||
| { | ||
| checkArgument(TABLE_PARAMETER_SAFE_KEY_PATTERN.matcher(parameterKey).matches(), "Parameter key contains invalid characters: '%s'", parameterKey); | ||
| /* | ||
| * The parameter value is restricted to have only alphanumeric characters so that it's safe | ||
| * to be used against HMS. When using with a LIKE operator, the HMS may want the parameter | ||
| * value to follow a Java regex pattern or a SQL pattern. And it's hard to predict the | ||
| * HMS's behavior from outside. Also, by restricting parameter values, we avoid the problem | ||
| * of how to quote them when passing within the filter string. | ||
| */ | ||
| checkArgument(TABLE_PARAMETER_SAFE_VALUE_PATTERN.matcher(parameterValue).matches(), "Parameter value contains invalid characters: '%s'", parameterValue); | ||
| /* | ||
| * Thrift call `get_table_names_by_filter` may be translated by Metastore to a SQL query against Metastore database. | ||
| * Hive 2.3 on some databases uses CLOB for table parameter value column and some databases disallow `=` predicate over | ||
| * CLOB values. At the same time, they allow `LIKE` predicates over them. | ||
| */ | ||
| String filterWithEquals = HIVE_FILTER_FIELD_PARAMS + parameterKey + " = \"" + parameterValue + "\""; | ||
|
findepi marked this conversation as resolved.
Outdated
|
||
| String filterWithLike = HIVE_FILTER_FIELD_PARAMS + parameterKey + " LIKE \"" + parameterValue + "\""; | ||
|
|
||
| return alternativeCall( | ||
| chosenTableParamAlternative, | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.