Skip to content

[SPARK-40914][CONNECT] Mark org.apache.spark.sql.connect to be private #38392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
amaliujia wants to merge 5 commits into from
Closed

[SPARK-40914][CONNECT] Mark org.apache.spark.sql.connect to be private #38392

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
baa1e73
[SPARK-40914][CONNECT] Mark private API to be private[connect].
amaliujia Oct 25, 2022
a196b71
update
amaliujia Oct 25, 2022
dae258c
update
amaliujia Oct 25, 2022
21d2126
update
amaliujia Oct 25, 2022
25665d6
update
amaliujia Oct 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion connector/connect/src/main/scala/org/apache/spark/sql/connect/SparkConnectPlugin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.util
import scala.collection.JavaConverters._

import org.apache.spark.SparkContext
import org.apache.spark.annotation.Unstable
import org.apache.spark.annotation.{Since, Unstable}
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
import org.apache.spark.sql.connect.service.SparkConnectService

Expand All @@ -34,6 +34,7 @@ import org.apache.spark.sql.connect.service.SparkConnectService
* JAR is available in the CLASSPATH and the driver plugin is configured to load this class.
*/
@Unstable
@Since("3.4.0")
class SparkConnectPlugin extends SparkPlugin {

/**
Expand Down
9 changes: 6 additions & 3 deletions connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,14 @@ import org.apache.spark.sql.connect.planner.DataTypeProtoConverter

/**
* A collection of implicit conversions that create a DSL for constructing connect protos.
*
* All classes in connect/dsl are considered an internal API to Spark Connect and are subject to
* change between minor releases.
Copy link
Contributor Author

@amaliujia amaliujia Oct 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still keep this to match Catalyst's Dsl.

*/

package object dsl {

object expressions { // scalastyle:ignore
private[connect] object expressions { // scalastyle:ignore
Copy link
Member

@HyukjinKwon HyukjinKwon Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we document this as an internal package, I think it's fine to remove private[connect], see also SPARK-16964 and SPARK-16813

Copy link
Contributor Author

@amaliujia amaliujia Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense at least for this dsl package. Removed private[connect] in this file.

implicit class DslString(val s: String) {
def protoAttr: proto.Expression =
proto.Expression
Expand Down Expand Up @@ -133,7 +136,7 @@ package object dsl {
.build()
}

object commands { // scalastyle:ignore
private[connect] object commands { // scalastyle:ignore
implicit class DslCommands(val logicalPlan: proto.Relation) {
def write(
format: Option[String] = None,
Expand Down Expand Up @@ -172,7 +175,7 @@ package object dsl {
}
}

object plans { // scalastyle:ignore
private[connect] object plans { // scalastyle:ignore
implicit class DslLogicalPlan(val logicalPlan: proto.Relation) {
def select(exprs: proto.Expression*): proto.Relation = {
proto.Relation
Expand Down
2 changes: 1 addition & 1 deletion .../connect/src/main/scala/org/apache/spark/sql/connect/planner/DataTypeProtoConverter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import org.apache.spark.sql.types.{DataType, IntegerType, LongType, StringType,
/**
* This object offers methods to convert to/from connect proto to catalyst types.
*/
object DataTypeProtoConverter {
private[connect] object DataTypeProtoConverter {
def toCatalystType(t: proto.DataType): DataType = {
t.getKindCase match {
case proto.DataType.KindCase.I32 => IntegerType
Expand Down
2 changes: 1 addition & 1 deletion ...src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import org.apache.spark.util.Utils
* added to the GRPC server in order of their position in the list. Once the statically compiled
* interceptors are added, dynamically configured interceptors are added.
*/
object SparkConnectInterceptorRegistry {
private[connect] object SparkConnectInterceptorRegistry {

// Contains the list of configured interceptors.
private lazy val interceptorChain: Seq[InterceptorBuilder] = Seq(
Expand Down