[SPARK-54595][SQL] Keep existing behavior of MERGE INTO without SCHEMA EVOLUTION clause

[szehon-ho]

What changes were proposed in this pull request?

Keep existing behavior for MERGE INTO without SCHEMA EVOLUTION clause for UPDATE SET * and INSERT * as well as UPDATE struct or INSERT struct, to throw exception if the source and target schemas are not exactly the same.

Why are the changes needed?

As @aokolnychyi tested this feature, he mentioned that as of Spark 4.1 the behavior is changed for MERGE INTO but without SCHEMA EVOLUTION clause.

In particular:

• Source has less columns/nested fields than target => we fill with NULL or DEFAULT for inserts, and existing value for Update. (though we disabled for nested structs by default in [SPARK-54525)]([SPARK-54525][SQL] Disable nested struct coercion in MERGE INTO under a config #53229)
• Source has more columns/fields than target => we drop the extra fields.

Initially, I thought its a good improvement of MERGE INTO and is not related to SCHEMA EVOLUTION exactly because the schema is not altered. But Anton has a good point that it may be a surprise to some user. So it may be better for now to be more conservative and keep the exact same behavior for without SCHEMA EVOLUTION clause.

Note: this behavior is still enabled if SCHEMA EVOLUTION is specified, as the user then is more explicit about the decision.

Does this PR introduce any user-facing change?

No, this keeps behavior exactly the same as 4.0 without SCHEMA EVOLUTION clause.

How was this patch tested?

Added a test and changed existing test output to expect the exception if SCHEMA EVOLUTION is not specified.

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Thank you, but this PR seems to break other test cases again, @szehon-ho .

Please make the CI happy.

[dongjoon-hyun]

Gentle ping, @szehon-ho .