[HUDI-5411] Avoid virtual key info for COW table in the input format

[codope]

Change Logs

Fetching virtual key involves reading from commit metadata or data file (TableSchemaResolver) which is a costly operation. This is only needed for schema projection in the case of MOR table (realtime splits). So, we can avoid it in case of COW table.

NOTE: This is stacked on top of #7526

Impact

Improves performance of hive-compatible query engines that depend on input format implementation in Hudi, e.g. trino-hive connector. Tested listing on a TPC-DS table with 1824 partitions.

Without this change (1.5 minutes):

trino:default> select count(*) from store_sales;
  _col0
---------
 2750311
(1 row)

Query 20221221_054403_00003_t63mx, FINISHED, 1 node
Splits: 1,832 total, 1,832 done (100.00%)
1:29 [2.75M rows, 28.5MB] [30.8K rows/s, 327KB/s]

With this change (18 seconds):

trino:default> select count(*) from store_sales;
  _col0
---------
 2750311
(1 row)

Query 20221221_055625_00002_knx5g, FINISHED, 1 node
Splits: 1,832 total, 1,832 done (100.00%)
17.30 [2.75M rows, 28.5MB] [169K rows/s, 1.75MB/s]

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[codope]

This is the main change. Earlier it used to be simply
Option<HoodieVirtualKeyInfo> virtualKeyInfoOpt = getHoodieVirtualKeyInfo(tableMetaClient);.

[alexeykudinkin]

Great find @codope! Glad we're able to identify the root-cause of that slow-down

[alexeykudinkin]

I'd suggest we take this fix one step further -- instead of fetching virtual key-info here let's push it inside createFileStatusUnchecked by pass the meta-client in there

[hudi-bot]

CI report:

• bd71935 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build