[HUDI-2763] Metadata table records - support for key deduplication based on hardcoded key field

[manojpec]

What is the purpose of the pull request

• The backing log format for the metadata table is HFile, a KeyValue type.
  Since the key field in the metadata record payload is a duplicate of the
  Key in the Cell, the redundant key field in the record can be emptied
  to save on the cost.

• HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
  with the key field emptied by default. HFile writer tries to find if
  the record has metadata payload schema field 'key' and if so it does
  the key trimming from the record payload.

• HoodieHFileReader when reading the serialized records back from disk,
  it materializes the missing keyFields if any. HFile reader tries to
  find if the record has metadata payload schema fiels 'key' and if so
  it does the key materialization in the record payload.

NOTE: There is a generic version of this PR at #4447
where the reader/writers of the log/base HFile format would say what the actual key field is
and the key deduplication and key materialization is done for the any such requested key field.

Verify this pull request

• Tests have been added to verify the default virtual keys and key
  deduplication support for the metadata table records.

• Manually verified the serialized records on the disk are trimmed
  off the key field

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[vinothchandar]

Need to find a way to remove the reference to metadatapayload reference from HFileWriter and HFileReader. lets keep pushing in this direction

[vinothchandar]

We need to avoid these direct references at all costs. I/O layer cannot reference the metadata payload class directly

[vinothchandar]

cc @prashantwason as well .

I still think we need to avoid the references to HoodieMetadataPayload directly from HFileReader and HoodieHFileDataBlock level. The issue here that these layers currently don't take in any config object.

For HFileReader - there is a hadoop configuration object that can be used to pass the field name
For HFileDataBlock - the log format writer creation could take an additional param.

Alternatively, a good Hudi citizen contributor (I just made that up), would have to plumb the interfaces more nicely to take a Properties and pass the writeConfig/commonConfig all the way to the reader/writer. This will be useful for many things going forward. This has to happen in a separate PR.

[manojpec]

@prashantwason @vinothchandar

After discussions, made HoodieHFileReader the single source of truth for all HFile schema related fields. HFileReader already tracks other fields and it is meaningful to move the key field also here. MetadataPayload and HFileDataBlock will refer to HFileReader for the key field. Passing the properties to HFileDataBlock can come in separate PR as suggested.

[vinothchandar]

Made some edits. LGTM high level, please fix the remaining naming comments. Also look at the changes I made to avoid duplicate conversion of records

[prashantwason]

LGTM. Nothing more to add to the existing open review comments.

[hudi-bot]

CI report:

• e9da09e Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[manojpec]

CI passed in the re-run - https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=5508&view=results

[nsivabalan]

LGTM

[alexeykudinkin]

Why is this changing?

[manojpec]

This is only for metadata table. Its intentional that we are disabling meta fields, that is enabling virtual keys for metadata table.