[SPARK-3654][SQL] Unifies SQL and HiveQL parsers

[liancheng]

This PR is a follow up of #2590, and tries to introduce a top level SQL parser entry point for all SQL dialects supported by Spark SQL.

A top level parser SparkSQLParser is introduced to handle the syntaxes that all SQL dialects should recognize (e.g. CACHE TABLE, UNCACHE TABLE and SET, etc.). For all the syntaxes this parser doesn't recognize directly, it fallbacks to a specified function that tries to parse arbitrary input to a LogicalPlan. This function is typically another parser combinator like SqlParser. DDL syntaxes introduced in #2475 can be moved to here.

The ExtendedHiveQlParser now only handle Hive specific extensions.

Also took the chance to refactor/reformat SqlParser for better readability.

[SparkQA]

QA tests have started for PR 2698 at commit 53ae252.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2698 at commit 53ae252.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21421/Test FAILed.

[liancheng]

Although looks trickier, Option[(String, Option[String])] captures the syntax and semantics of SET command better, and simplifies parsing logic.

[liancheng]

Scala style checking doesn't like the digit.+ DSL syntax in the parser, I'm falling back to rep1.

[SparkQA]

QA tests have started for PR 2698 at commit 9ebc290.

• This patch merges cleanly.

[liancheng]

Debugging code, forgot to remove this...

[SparkQA]

QA tests have finished for PR 2698 at commit 9ebc290.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21423/Test FAILed.

[SparkQA]

QA tests have started for PR 2698 at commit cc14609.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2698 at commit cc14609.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21426/Test FAILed.

[SparkQA]

QA tests have started for PR 2698 at commit a4b0fc4.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2698 at commit a4b0fc4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21461/Test PASSed.

[liancheng]

@marmbrus Please help review. ADD JAR and ADD FILE are still Hive specific. I'd like to move them to the top level parser in another PR.

[SparkQA]

QA tests have started for PR 2698 at commit ba2c121.

• This patch merges cleanly.

[liancheng]

This PR conflicts with #2678. It would be good to merge that one first, and then I'll rebase.

[SparkQA]

QA tests have finished for PR 2698 at commit ba2c121.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21525/Test FAILed.

[AmplabJenkins]

Can one of the admins verify this patch?

[SparkQA]

QA tests have started for PR 2698 at commit ceada76.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2698 at commit ceada76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical(val keywords: Seq[String]) extends StdLexical
  • case class FloatLit(chars: String) extends Token
  • class SqlParser extends AbstractSparkSQLParser
  • case class SetCommand(kv: Option[(String, Option[String])]) extends Command
  • case class ShellCommand(cmd: String) extends Command
  • case class SourceCommand(filePath: String) extends Command
  • case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21559/Test PASSed.