[SPARK-49464] Add documentations

### What changes were proposed in this pull request?

This PR includes Operator docs under `docs/` for configuration, architecture, operations, and metrics.

### Why are the changes needed?

Operator docs are necessary for users to understand the design and getting started with the operator installation

### Does this PR introduce _any_ user-facing change?

No - new release

### How was this patch tested?

CIs

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #113 from jiangzho/doc.

Authored-by: zhou-jiang <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: jiangzho

build-tools/docs-utils/build.gradle

@@ -38,3 +38,5 @@ tasks.register('generateConfPropsDoc', Exec) {
     description = "Generate config properties doc for operator"
     commandLine "java", "-classpath", sourceSets.main.runtimeClasspath.getAsPath(), javaMainClass, docsPath
 }
+
+build.finalizedBy(generateConfPropsDoc)

docs/architecture.md

@@ -0,0 +1,81 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications and clusters. The Operator can be installed on Kubernetes
+cluster(s) using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark workload in one or more managed namespaces.
+Spark Operator enables user to describe Spark application(s) or cluster(s) as 
+[Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). 
+
+The Operator continuously tracks events related to the Spark custom resources in its reconciliation 
+loops:
+
+For SparkApplications:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) and record status till app terminates
+* Operator releases all Spark-app owned resources to cluster
+
+For SparkClusters:
+
+* User submits a SparkCluster custom resource(CR) using kubectl / API
+* Operator launches master and worker(s) based on CR spec and observes their status
+* Operator releases all Spark-cluster owned resources to cluster upon failure
+
+The Operator is built with the [Java Operator SDK](https://javaoperatorsdk.io/) for
+launching Spark deployments and submitting jobs under the hood. It also uses 
+[fabric8](https://fabric8.io/) client to interact with Kubernetes API Server.
+
+## Application State Transition
+
+[<img src="resources/application_state_machine.png">](resources/application_state_machine.png)
+
+* Spark applications are expected to run from submitted to succeeded before releasing resources
+* User may configure the app CR to time-out after given threshold of time if it cannot reach healthy
+  state after given threshold. The timeout can be configured for different lifecycle stages, 
+  when driver starting and when requesting executor pods. To update the default threshold,  
+  configure `.spec.applicationTolerations.applicationTimeoutConfig` for the application.        
+* K8s resources created for an application would be deleted as the final stage of the application 
+  lifecycle by default. This is to ensure resource quota release for completed applications.  
+* It is also possible to retain the created k8s resources for debug or audit purpose. To do so,   
+  user may set `.spec.applicationTolerations.resourceRetainPolicy` to `OnFailure` to retain 
+  resources upon application failure, or set to `Always` to retain resources regardless of 
+  application final state.
+    - This controls the behavior of k8s resources created by Operator for the application, including
+      driver pod, config map, service, and PVC(if enabled). This does not apply to resources created 
+      by driver (for example, executor pods). User may configure SparkConf to
+      include `spark.kubernetes.executor.deleteOnTermination` for executor retention. Please refer 
+      [Spark docs](https://spark.apache.org/docs/latest/running-on-kubernetes.html) for details.
+    - The created k8s resources have `ownerReference` to their related `SparkApplication` custom
+      resource, such that they could be garbage collected when the `SparkApplication` is deleted.
+    - Please be advised that k8s resources would not be retained if the application is configured to
+      restart. This is to avoid resource quota usage increase unexpectedly or resource conflicts 
+      among multiple attempts.
+
+## Cluster State Transition
+
+[<img src="resources/cluster_state_machine.png">](resources/application_state_machine.png)
+
+* Spark clusters are expected to be always running after submitted.
+* Similar to Spark applications, K8s resources created for a cluster would be deleted as the final 
+  stage of the cluster lifecycle by default.

docs/config_properties.md

@@ -0,0 +1,40 @@
+[//]: # (This doc is automatically generated by gradle task, manual updates would be overridden.)
+# Spark Operator Config Properties
+ | Key | Type | Default Value | Allow Hot Reloading | Description | 
+ | --- | --- | --- | --- | --- | 
+ | spark.kubernetes.operator.name | String | spark-kubernetes-operator | false | Name of the operator. | 
+ | spark.kubernetes.operator.namespace | String | default | false | Namespace that operator is deployed within. | 
+ | spark.kubernetes.operator.watchedNamespaces | String | default | true | Comma-separated list of namespaces that the operator would be watching for Spark resources. If set to '*', operator would watch all namespaces. | 
+ | spark.kubernetes.operator.terminateOnInformerFailureEnabled | Boolean | false | false | Enable to indicate informer errors should stop operator startup. If disabled, operator startup will ignore recoverable errors, caused for example by RBAC issues and will retry periodically. | 
+ | spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer | 30 | false | Grace period for operator shutdown before reconciliation threads are killed. | 
+ | spark.kubernetes.operator.reconciler.parallelism | Integer | 50 | false | Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if set to non-positive number. | 
+ | spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds | Long | 30 | true | Timeout (in seconds) to for requests made to API server. This applies only to foreground requests. | 
+ | spark.kubernetes.operator.reconciler.intervalSeconds | Long | 120 | true | Interval (in seconds, non-negative) to reconcile Spark applications. Note that reconciliation is always expected to be triggered when app spec / status is updated. This interval controls the reconcile behavior of operator reconciliation even when there's no update on SparkApplication, e.g. to determine whether a hanging app needs to be proactively terminated. Thus this is recommended to set to above 2 minutes to avoid unnecessary no-op reconciliation. | 
+ | spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. | 
+ | spark.kubernetes.operator.reconciler.appStatusListenerClassNames | String |  | false | Comma-separated names of SparkAppStatusListener class implementations | 
+ | spark.kubernetes.operator.reconciler.clusterStatusListenerClassNames | String |  | false | Comma-separated names of SparkClusterStatusListener class implementations | 
+ | spark.kubernetes.operator.dynamicConfig.enabled | Boolean | false | false | When enabled, operator would use config map as source of truth for config property override. The config map need to be created in spark.kubernetes.operator.namespace, and labeled with operator name. | 
+ | spark.kubernetes.operator.dynamicConfig.selector | String | app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides | false | The selector str applied to dynamic config map. | 
+ | spark.kubernetes.operator.dynamicConfig.reconcilerParallelism | Integer | 1 | false | Parallelism for dynamic config reconciler. Unbounded pool would be used if set to non-positive number. | 
+ | spark.kubernetes.operator.reconciler.rateLimiter.refreshPeriodSeconds | Integer | 15 | false | Operator rate limiter refresh period(in seconds) for each resource. | 
+ | spark.kubernetes.operator.reconciler.rateLimiter.maxLoopForPeriod | Integer | 5 | false | Max number of reconcile loops triggered within the rate limiter refresh period for each resource. Setting the limit <= 0 disables the limiter. | 
+ | spark.kubernetes.operator.reconciler.retry.initialIntervalSeconds | Integer | 5 | false | Initial interval(in seconds) of retries on unhandled controller errors. | 
+ | spark.kubernetes.operator.reconciler.retry.intervalMultiplier | Double | 1.5 | false | Interval multiplier of retries on unhandled controller errors. Setting this to 1 for linear retry. | 
+ | spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds | Integer | -1 | false | Max interval(in seconds) of retries on unhandled controller errors. Set to non-positive for unlimited. | 
+ | spark.kubernetes.operator.api.retryMaxAttempts | Integer | 15 | false | Max attempts of retries on unhandled controller errors. Setting this to non-positive value means no retry. | 
+ | spark.kubernetes.operator.api.retryAttemptAfterSeconds | Long | 1 | false | Default time (in seconds) to wait till next request. This would be used if server does not set Retry-After in response. Setting this to non-positive number means immediate retry. | 
+ | spark.kubernetes.operator.api.statusPatchMaxAttempts | Long | 3 | false | Maximal number of retry attempts of requests to k8s server for resource status update. This would be performed on top of k8s client spark.kubernetes.operator.retry.maxAttempts to overcome potential conflicting update on the same SparkApplication. This should be positive number. | 
+ | spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts | Long | 3 | false | Maximal number of retry attempts of requesting secondary resource for Spark application. This would be performed on top of k8s client spark.kubernetes.operator.retry.maxAttempts to overcome potential conflicting reconcile on the same SparkApplication. This should be positive number | 
+ | spark.kubernetes.operator.metrics.josdkMetricsEnabled | Boolean | true | false | When enabled, the josdk metrics will be added in metrics source and configured for operator. | 
+ | spark.kubernetes.operator.metrics.clientMetricsEnabled | Boolean | true | false | Enable KubernetesClient metrics for measuring the HTTP traffic to the Kubernetes API Server. Since the metrics is collected via Okhttp interceptors, can be disabled when opt in customized interceptors. | 
+ | spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled | Boolean | true | false | When enabled, additional metrics group by http response code group(1xx, 2xx, 3xx, 4xx, 5xx) received from API server will be added. Users can disable it when their monitoring system can combine lower level kubernetes.client.http.response.<3-digit-response-code> metrics. | 
+ | spark.kubernetes.operator.metrics.port | Integer | 19090 | false | The port used for checking metrics | 
+ | spark.kubernetes.operator.health.probePort | Integer | 19091 | false | The port used for health/readiness check probe status. | 
+ | spark.kubernetes.operator.health.sentinelExecutorPoolSize | Integer | 3 | false | Size of executor service in Sentinel Managers to check the health of sentinel resources. | 
+ | spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds | Integer | 60 | true | Allowed max time(seconds) between spec update and reconciliation for sentinel resources. | 
+ | spark.kubernetes.operator.leaderElection.enabled | Boolean | false | false | Enable leader election for the operator to allow running standby instances. When this is disabled, only one operator instance is expected to be up and running at any time (replica = 1) to avoid race condition. | 
+ | spark.kubernetes.operator.leaderElection.leaseName | String | spark-operator-lease | false | Leader election lease name, must be unique for leases in the same namespace. | 
+ | spark.kubernetes.operator.leaderElection.leaseDurationSeconds | Integer | 180 | false | Leader election lease duration in seconds, non-negative. | 
+ | spark.kubernetes.operator.leaderElection.renewDeadlineSeconds | Integer | 120 | false | Leader election renew deadline in seconds, non-negative. This needs to be smaller than the lease duration to allow current leader renew the lease before lease expires. | 
+ | spark.kubernetes.operator.leaderElection.retryPeriodSeconds | Integer | 5 | false | Leader election retry period in seconds, non-negative. | 
+