Respect hive.metastore.thrift.delete-files-on-drop config property for dropping partitions

[ebyhr]

Description

Respect hive.metastore.thrift.delete-files-on-drop config property for dropping partitions
Fixes #13545 Fixes #13821

Documentation

(x) Sufficient documentation is included in this PR.

Release notes

(x) Release notes entries required with the following suggested text:

# Hive
* Respect `hive.metastore.thrift.delete-files-on-drop` config property for dropping partitions. ({issue}`13545`)

[alexjo2144]

The Hive connector doc needs an update if we're going to do this. Specifically:

system.unregister_partition(schema_name, table_name, partition_columns, partition_values)

Unregisters given, existing partition in the metastore for the specified table. The partition data is not deleted.

[ebyhr]

I don't think we need to update unregister_partition docs since deleteData argument is false. I will add a test for the procedure though.

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/procedure/UnregisterPartitionProcedure.java

Lines 107 to 112 in bcfe05f

	metastore.dropPartition(
	session,
	table.getDatabaseName(),
	table.getTableName(),
	partition.getValues(),
	false);

[ebyhr]

Using normal test environment because somehow I can't reproduce the issue on product test environment.

[alexjo2144]

Should we wrap the file deletion in a separate try/catch?

I'm just thinking, if the partition is dropped from the metastore but the file system doesn't allow us to delete the files for some reason, the query should still succeed right?

[ebyhr]

I feel users should disable it in such cases. This feature can be controlled by config property, so exception looks better in my opinion.

[alexjo2144]

I think it depends on what the behavior is for a following select statement. If I run a drop partition and it fails, the partition should still show up in the next query. If it doesn't show up, the drop should succeed. Does that make sense?

[ebyhr]

I was missing that subsequent deleteDirRecursive already has try-catch block in the method. Do you think it still requires additional change?

[alexjo2144]

You're totally right. This is fine

[findinpath]

There are several CI failures.

[findepi]

There are several CI failures.

Added empty commit to retrigger the build after #14323 merged.

[ebyhr]

CI hit #14441

[findepi]

redundant?

[ebyhr]

It's not redundant. ThriftMetastoreConfig.setDeleteFilesOnDrop is equals to hive.metastore.thrift.delete-files-on-drop not hive.allow-register-partition-procedure.

[findepi]

else validate initialTables is empty, since it's ignored otherwise

however, a better way for this would be to call the parameter createTpchSchemas
then createTpchSchemas(false) would skip schema cretion and initialTables(empty) (default) would skip tables creation

[ebyhr]

Renamed to createTpchSchemas .

[findepi]

are we using s3.*.amazonaws.com without https://? should we add it explicitly? in secrets?

[ebyhr]

Pushed another commit to add the prefix.

[homar]

Maybe it would make sense to check that the table in fact is empty?

[ebyhr]

Included additional assertions.

[ebyhr]

Addressed comments.

[findepi]

LGTM, but the build is red

[ebyhr]

Failed jobs succeeded in #14557 (within the repo). I will take a look these failures and file issues if needed.

[ebyhr]

CI hit

• suite-1 Flaky TestHiveTableStatistics.testMixedHiveAndPrestoStatistics #14588
• suite-3 Flaky TestTrinoCli.shouldRunMultipleBatchQueriesWithStdinRedirect #14587
• suite-7-non-generic Flaky io.trino.tests.product.kudu.TestKuduConnectoKerberosSmokeTest.kerberosAuthTicketExpiryTest #14441

[sumanshusamarora]

@ebyhr Is there a similar property available for glue metastore. Upon drop table command, the table meta information is deleted from glue but the underlying data remains in azure. Anything you can suggest to allow dropping of data?