[SPARK-40733][SQL] Make the contents of SERDEPROPERTIES in the result of ShowCreateTableAsSerdeCommand have a fixed order

[LuciferYang]

What changes were proposed in this pull request?

This pr add a sort operation to make the contents of SERDEPROPERTIES in the result of ShowCreateTableAsSerdeCommand have a fixed order.

Why are the changes needed?

Improve Java version compatibility, make the results of ShowCreateTableAsSerdeCommand consistent when using Java version.

Run the following command with Java 19:

mvn clean install -DskipTests -pl sql/hive -am  
mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite

There are 2 failed test:

- SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."
- SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."

The content of SERDEPROPERTIES is correct, but the order is different from result running with Java 8/11/17.

Does this PR introduce any user-facing change?

Yes, the SERDEPROPERTIES part of ShowCreateTableAsSerdeCommand results will be displayed in order by key.

How was this patch tested?

• Pass Git Hub Actions
• Manual test:

Run the following command with Java 19:

mvn clean install -DskipTests -pl sql/hive -am  
mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite

Before

- SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."
- SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."

After

Run completed in 23 seconds, 930 milliseconds.
Total number of tests run: 58
Suites: completed 2, aborted 0
Tests: succeeded 58, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

[LuciferYang]

Test first

[LuciferYang]

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

Lines 1202 to 1210 in 1cb8250

	private def showDataSourceTableOptions(metadata: CatalogTable, builder: StringBuilder): Unit = {
	// For datasource table, there is a provider there in the metadata.
	// If it is a Hive table, we already convert its metadata and fill in a provider.
	builder ++= s"USING ${metadata.provider.get}\n"
	val dataSourceOptions = conf.redactOptions(metadata.storage.properties).toSeq.sortBy(_._1).map {
	case (key, value) =>
	s"'${escapeSingleQuotedString(key)}' = '${escapeSingleQuotedString(value)}'"
	}

showDataSourceTableOptions already do the similar work in #34753

[dongjoon-hyun]

+1, LGTM. Thank you, @LuciferYang .
Merged to master for Apache Spark 3.4.0.

[LuciferYang]

Thanks @dongjoon-hyun