[HUDI-8504] Fix missing database config when building Hudi configs in Spark

[fhan688]

Change Logs

The current implementation missed database when building hudi config, which will lead to obtain the incorrect table even though the table exists. Such as the case of spark writing data

hudi/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala

Line 97 in cea81e8

val config = buildHoodieInsertConfig(catalogTable, sparkSession, isOverWritePartition, isOverWriteTable, partitionSpec, extraOptions, staticOverwritePartitionPathOpt)

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

Lines 251 to 254 in cea81e8

	val databaseName = hoodieConfig.getStringOrDefault(HoodieTableConfig.DATABASE_NAME, "")
	val tblName = hoodieConfig.getStringOrThrow(HoodieWriteConfig.TBL_NAME,
	s"'${HoodieWriteConfig.TBL_NAME.key}' must be set.").trim
	val tableIdentifier = TableIdentifier(tblName, if (databaseName.isEmpty) None else Some(databaseName))

Because the value of DATABASE_NAME.key is lost when buildHoodieInsertConfig, so databaseName is empty in tableIdentifier.

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

Lines 667 to 670 in cea81e8

	getCatalogTable(spark, tableId).map { catalogTable =>
	val (structName, namespace) = getAvroRecordNameAndNamespace(tableId.table)
	convertStructTypeToAvroSchema(catalogTable.schema, structName, namespace)
	}

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

Lines 674 to 680 in cea81e8

	private def getCatalogTable(spark: SparkSession, tableId: TableIdentifier): Option[CatalogTable] = {
	if (spark.sessionState.catalog.tableExists(tableId)) {
	Some(spark.sessionState.catalog.getTableMetadata(tableId))
	} else {
	None
	}
	}

Then the table under the databaseName cannot be obtained or the table under the default database is obtained, it is obviously inconsistent with expectation.

this PR fix this bug.

Impact

hudi-spark-common

Risk level (write none, low medium or high below)

Low

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

@fhan688 Thanks for the contribution, can you check the test failures.

[hudi-bot]

CI report:

• 75b901f Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build