[SPARK-40995] [CONNECT] [DOC] Defining Spark Connect Client Connection String

[grundprinzip]

What changes were proposed in this pull request?

This patch adds documentation to describe how clients should implement handling
connecting to the Spark Connect endpoint. GRPC as a protocol is well documented and has
many options. However, this does not make it easy for users to reason about how to correctly
configure GRPC to make it work for their use cases.

To overcome the issue, this document defines a client connection string that needs to be parsed
by the different language clients and can be used to properly configure the GRPC client.

Why are the changes needed?

Documentation and design specification for clients implementing the Spark Connect protocol.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Doc only.

[amaliujia]

Overall LGTM

Is the user_id (or the user session token) be relevant to this doc?

spark/python/pyspark/sql/connect/client.py

Line 107 in 8f6b185

def __init__(self, user_id: str, host: Optional[str] = None, port: int = 15002):

[grundprinzip]

Good point, I will incorporate that into the doc.

[HyukjinKwon]

The usage documentation should better be in https://github.com/apache/spark/tree/master/docs as an md file so it can be properly documented.

[HyukjinKwon]

Maybe it's better to have a JIRA. BTW, wonder if we have an e2e example for users can copy and paste to try. (e.g., like most of docs in https://spark.apache.org/docs/latest/index.html). Another decision to make is if we should document it in PySpark docs (https://spark.apache.org/docs/latest/api/python/getting_started/index.html) or Spark main page (https://spark.apache.org/docs/latest/index.html)

[grundprinzip]

@HyukjinKwon I will add a Jira this is just the starting point to align where we want to go.

My idea would be that once this is merged I will create a pr for the python client to follow this proposal and then we can update this doc with an e2e example.

But the main goal here is developer documentation not user doc. For example, scala must then implement the same model.

[HyukjinKwon]

For developer documentation, it might better be placed under sources as a comment e.g., packages.scala. e.g.) https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala#L40-L123

Or. maybe README.md at the top level of the compoenent https://github.com/apache/spark/blob/master/hadoop-cloud/README.md or https://github.com/apache/spark/blob/master/connector/docker/README.md

[grundprinzip]

What about we link to it from the top level Readme in the component?

The reason why it's not in the code is because it's client language agnostic.

[HyukjinKwon]

Maybe putting it to the top model level (connector/connect/README.md) for now could be a good idea (?). Just wanted to avoid a different structure compared to other compoenents (connector/connect/doc)

[grundprinzip]

@HyukjinKwon so my proposal is the following:

• I moved the README from the pyspark folder into the top-level connector/connect directory and made it clear that these are developer docs.
• The doc directory was moved to docs to be consistent with the project
• Linked from the top-level README to the connection string configuration doc.

WDYT?

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

Merged to master.