[HUDI-5642] Enable schema reconciliation by default

[codope]

Change Logs

Enable schema reconciliation by default as commit should go through with the wider schema in case of conflict.

Impact

Schema will be reconciled by default.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

@codope I think we should create a JIRA issue for it.

[codope]

The test failures are due to SchemaCompatibiltyException. One thing to note is that for an incoming schema with compatible projection of the latest table schema, the table schema is chosen but it may lose any data type promotion. See

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

Lines 566 to 574 in 3979848

	if (isCompatibleProjectionOf(tableSchema, newSchema)) {
	// Picking table schema as a writer schema we need to validate that we'd be able to
	// rewrite incoming batch's data (written in new schema) into it
	(tableSchema, isSchemaCompatible(newSchema, tableSchema))
	} else {
	// Picking new schema as a writer schema we need to validate that we'd be able to
	// rewrite table's data into it
	(newSchema, isSchemaCompatible(tableSchema, newSchema))
	}

[vinothchandar]

@codope please help me understand why this is critical priority?

[codope]

please help me understand why this is critical priority?

this was planned to be done in one of the prev releases but there are some issues. I've lowered the priority for now, but we should take it up for 0.14.0. Updated the fix version on JIRA appropriately.

[hudi-bot]

CI report:

• 67221fd Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

this was planned to be done in one of the prev releases but there are some issues. I've lowered the priority for now, but we should take it up for 0.14.0. Updated the fix version on JIRA appropriately.

That's cool, but we are not expecting to introduce any schema data type related regressions.

[yihua]

This config is deprecated after #9743, so closing this stale PR.