Skip to content

Extends flush_metadata_cache procedure #14219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Praveen2112 merged 1 commit into from
Sep 27, 2022
Merged

Extends flush_metadata_cache procedure #14219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/src/main/sphinx/connector/hive.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1079,6 +1079,11 @@ The following procedures are available:

Flush all Hive metadata caches.

* ``system.flush_metadata_cache(schema_name => ..., table_name => ...)``

Flush Hive metadata caches entries connected with selected table.
Procedure requires named parameters to be passed

* ``system.flush_metadata_cache(schema_name => ..., table_name => ..., partition_column => ARRAY[...], partition_value => ARRAY[...])``

Flush Hive metadata cache entries connected with selected partition.
Expand Down
10 changes: 8 additions & 2 deletions .../main/java/io/trino/plugin/hive/metastore/procedure/FlushHiveMetastoreCacheProcedure.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ public class FlushHiveMetastoreCacheProcedure
private static final String PROCEDURE_USAGE_EXAMPLES = format(
"Valid usages:%n" +
" - '%1$s()'%n" +
" - %1$s(%2$s => ..., %3$s => ...)" +
" - %1$s(%2$s => ..., %3$s => ..., %4$s => ARRAY['...'], %5$s => ARRAY['...'])",
PROCEDURE_NAME,
// Use lowercase parameter names per convention. In the usage example the names are not delimited.
Expand Down Expand Up @@ -113,8 +114,13 @@ private void doFlushMetadataCache(Optional<String> schemaName, Optional<String>
if (schemaName.isEmpty() && tableName.isEmpty() && partitionColumns.isEmpty()) {
cachingHiveMetastore.flushCache();
}
else if (schemaName.isPresent() && tableName.isPresent() && !partitionColumns.isEmpty()) {
cachingHiveMetastore.flushPartitionCache(schemaName.get(), tableName.get(), partitionColumns, partitionValues);
else if (schemaName.isPresent() && tableName.isPresent()) {
if (!partitionColumns.isEmpty()) {
Copy link
Copy Markdown
Member

@Praveen2112 Praveen2112 Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we could resolve a call procedure based on the name and arguments. So we could have an individual method for each configuration - instead of if-else.

cachingHiveMetastore.flushPartitionCache(schemaName.get(), tableName.get(), partitionColumns, partitionValues);
}
else {
cachingHiveMetastore.invalidateTable(schemaName.get(), tableName.get());
}
}
else {
throw new TrinoException(
Expand Down
10 changes: 10 additions & 0 deletions plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,16 @@ public void testFlushPartitionCache()
// Should return 0 rows as we left cache untouched
assertQueryReturnsEmptyResult(queryUsingPartitionCacheForValue2);

// Refresh cache for schema_name => 'dummy_schema', table_name => 'dummy_table'
getQueryRunner().execute(format(
"CALL system.flush_metadata_cache(schema_name => '%s', table_name => '%s')",
HIVE_TEST_SCHEMA,
tableName));

// Should return expected rows for all partitions
assertQuery(queryUsingPartitionCacheForValue1, expectedQueryResultForValue1);
assertQuery(queryUsingPartitionCacheForValue2, expectedQueryResultForValue2);

computeActual(format("DROP TABLE %s", fullyQualifiedTestTableName));
}

Expand Down
7 changes: 4 additions & 3 deletions ...st/java/io/trino/plugin/hive/metastore/cache/TestCachingHiveMetastoreWithQueryRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,15 +143,16 @@ public void testFlushHiveMetastoreCacheProcedureCallable()
@Test
public void testIllegalFlushHiveMetastoreCacheProcedureCalls()
{
String illegalParameterMessage = "Illegal parameter set passed. Valid usages:\n - 'flush_metadata_cache()'\n - flush_metadata_cache(schema_name => ..., table_name => ..., partition_column => ARRAY['...'], partition_value => ARRAY['...'])";
String illegalParameterMessage = "Illegal parameter set passed. Valid usages:\n" +
" - 'flush_metadata_cache()'\n" +
" - flush_metadata_cache(schema_name => ..., table_name => ...)" +
" - flush_metadata_cache(schema_name => ..., table_name => ..., partition_column => ARRAY['...'], partition_value => ARRAY['...'])";

assertThatThrownBy(() -> getQueryRunner().execute("CALL system.flush_metadata_cache('dummy_schema')"))
.hasMessageContaining("Only named arguments are allowed for this procedure");

assertThatThrownBy(() -> getQueryRunner().execute("CALL system.flush_metadata_cache(schema_name => 'dummy_schema')"))
.hasMessage(illegalParameterMessage);
assertThatThrownBy(() -> getQueryRunner().execute("CALL system.flush_metadata_cache(schema_name => 'dummy_schema', table_name => 'dummy_table')"))
.hasMessage(illegalParameterMessage);

assertThatThrownBy(() -> getQueryRunner().execute("CALL system.flush_metadata_cache(schema_name => 'dummy_schema', table_name => 'dummy_table', partition_column => ARRAY['dummy_partition'])"))
.hasMessage("Parameters partition_column and partition_value should have same length");
Expand Down