[SPARK-9763][SQL] Minimize exposure of internal SQL classes.

[rxin]

There are a few changes in this pull request:

1. Moved all data sources to execution.datasources, except the public JDBC APIs.
2. In order to maintain backward compatibility from 1, added a backward compatibility translation map in data source resolution.
3. Moved ui and metric package into execution.
4. Added more documentation on some internal classes.
5. Renamed DataSourceRegister.format -> shortName.
6. Added "override" modifier on shortName.
7. Removed IntSQLMetric.

[SparkQA]

Test build #40259 has finished for PR 8056 at commit 122864a.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @zsxwing for review.

[zsxwing]

@rxin could you help remove @VisibleForTesting from SQLListener.scala because of the issue mentioned by @shivaram in #7774 (comment) ?

[SparkQA]

Test build #40263 has finished for PR 8056 at commit 8f4dc20.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

Although the SQL UI only displays the execution info now, moving the ui package into the execution package looks not a good idea. I guess we may add other information to the SQL tab later, then we may need to move the ui package back.

[rxin]

@zsxwing removed visible for testing tag.

We can move UI package around later -- it's also OK if it contains non-execution stuff. Not that big of a deal...

[SparkQA]

Test build #40275 has finished for PR 8056 at commit c3a4ba4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #40280 has finished for PR 8056 at commit 9d83ba2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

The failure is because ParquetIOSuite has a hard-code name.

[zsxwing]

Is it intentional to keep DDLSourceLoadSuite.scala and ResolvedDataSourceSuite.scala under org.apache.spark.sql.sources?

[SparkQA]

Test build #40283 has finished for PR 8056 at commit 3dfc06c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1419 has finished for PR 8056 at commit 3dfc06c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Not intentional -- but not that big of a deal for them to be there since I care more about the public API visibility here.

[SparkQA]

Test build #40306 has finished for PR 8056 at commit 9df4801.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

OK I'm going to merge this since it's from a flaky test.

[rxin]

cc @liancheng for the change also.

[rama-mullapudi]

jdbcutils scala code has a typo error in schemaString function, decimal type has extra closing braces } which is causing create table with decimal fail as create statement sent to db looks as below

CREATE TABLE foo (TKT_GID DECIMAL(10},0}) NOT NULL)

Below is the code

def schemaString(df: DataFrame, url: String): String = {
.....
case t: DecimalType => s"DECIMAL(${t.precision}},${t.scale}})"
....
}

[rxin]

@rama-mullapudi thanks. can you submit a patch to fix those?

I think I only moved the stuff around.

[jyssky]

@rxin Following the discussion on SPARK-9763, I am actually wondering why we convert the StructType with "asNullable" which set all the contained StructField to be Nullable. This will cause problem when one StructFiled is not allowed to be nullable, but the HadoopFsRelationProvider automatically sets it to be nullable. Is it because that all the fields in HadoopFsRelationProvider have to be nullable? Thanks!