[SPARK-46537][SQL] Convert NPE and asserts from commands to internal errors

[MaxGekk]

What changes were proposed in this pull request?

In the PR, I propose to handle NPE and asserts from eagerly executed commands, and convert them to internal errors.

Why are the changes needed?

To unify the approach for errors raised by Spark SQL.

Does this PR introduce any user-facing change?

Yes.

Before the changes:

Cannot invoke "org.apache.spark.sql.connector.read.colstats.ColumnStatistics.min()" because the return value of "scala.Option.get()" is null
java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.connector.read.colstats.ColumnStatistics.min()" because the return value of "scala.Option.get()" is null
	at org.apache.spark.sql.execution.datasources.v2.DescribeColumnExec.run(DescribeColumnExec.scala:63)

After:

org.apache.spark.SparkException: [INTERNAL_ERROR] Eagerly executed command failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
	at org.apache.spark.SparkException$.internalError(SparkException.scala:107)
...
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.connector.read.colstats.ColumnStatistics.min()" because the return value of "scala.Option.get()" is null
	at org.apache.spark.sql.execution.datasources.v2.DescribeColumnExec.run(DescribeColumnExec.scala:63)

How was this patch tested?

Manually, by running the test from another PR: #44524

Was this patch authored or co-authored using generative AI tooling?

No.

[MaxGekk]

@yaooqinn @LuciferYang @cloud-fan This PR fixes the issue: #44524 (comment) . Thanks to @Zouxxyy for the test.

[yaooqinn]

LGTM

[MaxGekk]

The failure of Run / Run Spark on Kubernetes Integration test is not related to the changes, I do believe.

Merging to master. Thank you, @yaooqinn and @cloud-fan for review.