Skip to content

Conversation

@dragqueen95
Copy link
Contributor

What changes were proposed in this pull request?

#44575 added a default parameter to HiveThriftServer2.startWithContext API. Although this is source-compatible, this is not binary-compatible - for eg. a binary built with new code won't be able to run with the old code. In this PR, we maintain forward and backward compatibility by keeping the API same and introducing a separate API for the additional parameter.

Why are the changes needed?

See above

Does this PR introduce any user-facing change?

only developer API change

How was this patch tested?

Existing tests

Was this patch authored or co-authored using generative AI tooling?

no

@github-actions github-actions bot added the SQL label Mar 26, 2024
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the goal of this PR, @dragqueen95 . Thank you for making a PR.

  • Could you add @Since(xxxVersionxxx) annotation for both methods to make it clear?
  • In addition, please enable GitHub Action on your repository to make CI pass. Apache Spark community uses your GitHub Action resource.

@dragqueen95 dragqueen95 marked this pull request as draft March 26, 2024 17:51
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-46575][SQL][FOLLOWUP] Maintain backward/ forward compatibility for the DeveloperApi HiveThriftServer2.startWithContext [SPARK-46575][SQL][FOLLOWUP] Add back HiveThriftServer2.startWithContext(SQLContext) method for compatibility Mar 26, 2024
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@cloud-fan cloud-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending CI

@dragqueen95 dragqueen95 marked this pull request as ready for review March 27, 2024 02:23
@Since("2.0.0")
@DeveloperApi
def startWithContext(sqlContext: SQLContext): HiveThriftServer2 = {
startWithContext(sqlContext, exitOnError)
Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, could you give a value for exitOnError, @dragqueen95 ? The value seems to be missed here.

Error:  /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala:88: not found: value exitOnError

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true seems to be fine to me.

@dongjoon-hyun
Copy link
Member

Thank you for the update, @cloud-fan .

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in f9eb3f3 Mar 27, 2024
* the call logs the error and exits the JVM with exit code -1. When false, the
* call throws an exception instead.
*/
@Since("3.5.2")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4.0.0?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I thought #44575 was backported.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To @dragqueen95 , could you make a follow-up to fix this to 4.0.0 since SPARK-46575 is not in branch-3.5?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised #45737

@dragqueen95 dragqueen95 deleted the thriftserver-back-compat branch March 27, 2024 09:50
@pan3793
Copy link
Member

pan3793 commented Jun 13, 2024

SQLContext was not recommended since 2.0.0, for new introduced Developer API in 4.0.0, maybe we can replace it with SparkSession?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants