Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

What changes were proposed in this pull request?

Currently, Spark supports ExternalCatalog, but the usage is limited to HiveExternalCatalog by using private string variables for class name holder. This PR aims to parameterize those variables as SQL configuration and supports user-provided ExternalCatalog and SessionState classes.

The following is an intended use case and a snippet of test case in this PR.

val session = SparkSession.builder()
  .master("local")
  .config("spark.sql.externalCatalog", "org.apache.spark.sql.MyExternalCatalog")
  .config("spark.sql.externalSessionState", "org.apache.spark.sql.MySessionState")
  .enableProvidedCatalog()
  .getOrCreate()

assert(session.sharedState.externalCatalog.isInstanceOf[MyExternalCatalog])
assert(session.sessionState.isInstanceOf[MySessionState])

How was this patch tested?

Pass the Jenkins test with a new test case.

@SparkQA
Copy link

SparkQA commented Oct 3, 2016

Test build #66280 has finished for PR 15336 at commit d2bf71c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 3, 2016

Test build #66282 has finished for PR 15336 at commit d0a1778.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

The issue is closed as 'Later'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants