Add support to redirect from a Hive catalog to an Iceberg catalog

[lxynov]

Add catalog redirection support to SPI
Add a catalog redirection method to connector SPI. It allows a connector
to redirect to another catalog which may or may not use the same
connector.

Catalog redirection is tried whenever a table or view is accessed, e.g.,
in SELECT, INSERT, DROP TABLE, ALTER TABLE, GRANT, etc., queries. However,
it's not tried with procedures.

Multi-stop redirection without loop is allowed within a max depth.

Add support to redirect from Hive to an Iceberg catalog
When enabled, Hive Connector would redirect a table access to an Iceberg
catalog when the table is an Iceberg table.

---

Issue: #4442
Umbrella issue: #1324

Some notes:

• Access control is enforced on the eventual table.
• Information schema queries (show tables, select * from information_schema.columns) stay the same. They won't include results from other catalogs. Having said that, in the Hive/Iceberg case, a Hive catalog does include results of Iceberg tables inherently.
• I thought about adding warning messages for users. Can work on that as a follow-up if it makes sense.

[electrum]

This could be written as

return getOptionalCatalogMetadata(session, tableName.getCatalogName())
        .map(catalog -> {
            CatalogName catalogName = catalog.getConnectorId(session, tableName);
            ConnectorMetadata metadata = catalog.getMetadataFor(catalogName);
            ConnectorSession connectorSession = session.toConnectorSession(catalogName);
            return metadata.redirectCatalog(connectorSession, tableName.asSchemaTableName());
        });

[lxynov]

metadata.redirectCatalog(connectorSession, tableName.asSchemaTableName()) returns Optional<String> instead of String, making it difficult to follow the pattern here.

[dain]

Try flatMap?

[findepi]

This shouldn't be Iceberg specific. One could want to redirect to some other catalog for some other reasons.
eg HBase table to Phoenix, Hive JDBC table to respective JDBC connector, etc.

[lxynov]

@electrum Thanks for the review. I've updated the PR

[lxynov]

This shouldn't be Iceberg specific. One could want to redirect to some other catalog for some other reasons.
eg HBase table to Phoenix, Hive JDBC table to respective JDBC connector, etc.

@findepi People can implement the ConnectorMetadata:: redirectCatalog method to achieve these

[findepi]

@lxynov great! I actually mean redirecting from hive to other connectors.
This needs to be extensible. Therefore, we should not have iceberg specific
configuration properties like hive.redirecting-to-iceberg-enabled, as they
clearly prevent extensibility.

[lxynov]

@findepi oh I see your point. That makes sense. We can name it hive.catalog-redirection-enabled. @electrum what do you think?

[electrum]

I'm not sure it makes sense to be extensible at the user level. The redirection here is specific to Iceberg based on the Hive connector's knowledge that the table is an Iceberg table. If we had an HBase connector in addition to a Phoenix connector, it seems that the same level of specific knowledge would apply.

[lxynov]

@electrum Thanks for the review! PR updated. The failing test seems unrelated.

[dain]

Is there going to be a redirect Iceberg to Hive feature also?

[dain]

I would expect this to be an error. Do we have cases where we do not have a fully qualified name?

[dain]

I think this name might be too generic. I believe this only applies to the table namespace, and not procedures or functions. I suggest we make this clear, so we can add those is the future if needed.

[dain]

Try flatMap?

[dain]

Why not add this directly to Metadata and make redirectCatalog a private method on Metadata? I ask because I don't see any direct usages of redirectCatalog.

[electrum]

Replaced by #5160