docs(spider): Add a user guide for deploying Spider and related components with the CLP Helm chart.

docs/src/user-docs/guides-k8s-deployment.md

@@ -287,6 +287,11 @@ helm template clp . -f custom-values.yaml
 To use Presto as the query engine, see the [Using Presto][presto-guide] guide for setup
 instructions, including the Helm values file and installation steps.
 
+### Using Spider as the scheduler
+
+To use [Spider] as the compression scheduler, see the [Using Spider][spider-guide] guide for setup
+instructions, including the Helm values file and installation steps.
+
 ### Component scheduling
 
 You can control where any CLP component is scheduled using standard Kubernetes scheduling primitives
@@ -686,6 +691,7 @@ To tear down a `kubeadm` cluster:
 * [Using object storage][s3-storage]: Configuring S3 storage
 * [Configuring retention periods][retention-guide]: Setting up data retention policies
 * [Using Presto][presto-guide]: Distributed SQL queries on compressed logs
+* [Using Spider][spider-guide]: Spider-orchestrated compression
 
 [admin-tools]: reference-sbin-scripts/admin-tools.md
 [aks]: https://azure.microsoft.com/en-us/products/kubernetes-service
@@ -711,5 +717,7 @@ To tear down a `kubeadm` cluster:
 [s3-storage]: guides-using-object-storage/index
 [SeaweedFS]: https://github.com/seaweedfs/seaweedfs
 [seaweedfs-setup]: guides-docker-compose-deployment.md#setting-up-seaweedfs
+[Spider]: https://github.com/y-scope/spider
+[spider-guide]: guides-using-spider.md
 [webui-clp-json]: quick-start/clp-json.md#searching-from-the-ui
 [webui-clp-text]: quick-start/clp-text.md#searching-from-the-ui

docs/src/user-docs/guides-using-spider-scheduler.md

@@ -0,0 +1,198 @@
+# Using Spider scheduler
+
+[Spider][spider] is a distributed task scheduling and execution framework. It schedules tasks across
+configured workers and manages their execution lifecycle.
+
+CLP uses Spider as its next-generation framework for orchestrating compression and search jobs,
+providing the following benefits:
+
+* **Improved fault tolerance**
+* **Minimized scheduling overhead**
+* **Maximized utilization of compute resources**
+
+:::{note}
+In the current release, CLP uses Spider only for compression jobs coordinated by
+`compression-coordinator`. Support for orchestrating search jobs with Spider is under development
+and will be introduced in a future release.
+
+Spider is planned to become the default job execution framework for CLP, eventually replacing the
+existing Celery-based job orchestration components.
+:::
+
+## Deployment
+
+When Spider is enabled, compatible CLP coordinators are automatically deployed and submit supported
+jobs to Spider for scheduling and execution. Currently, Spider and its related CLP components can be
+enabled only in [Kubernetes deployments][k8s-deployment] using the CLP Helm chart.
+
+:::{note}
+Spider is not supported in Docker Compose deployments in the current release. Support for Docker
+Compose is planned for a future release.
+:::
+
+### Kubernetes (Helm)
+
+When deploying CLP on Kubernetes using Helm, enable Spider by setting `spider.enabled` to `true`.
+The CLP Helm chart will then automatically deploy the following components:
+
+* **Spider subchart**: Deploys all the Spider services required to schedule and execute jobs.
+* **compression-coordinator**: Coordinates supported CLP compression jobs and submits them to
+  Spider for execution.
+
+:::{note}
+In the current release, `compression-coordinator` supports only compression jobs that use certain
+CLP package configurations. See "Using compression-coordinator" for the current limitations.
+:::
+
+#### Set up
+
+1. Create a values file to enable Spider and its related CLP components.
+
+   ```{code-block} yaml
+   :caption: spider-values.yaml
+
+   spider:
+     enabled: true
+
+     # Optional Spider configuration passed directly to the Spider subchart.
+     # The values below are the defaults and can be adjusted as needed.
+     spiderConfig:
+       storage:
+         log_level: "INFO"
+         port: 50051
+       scheduler:
+         log_level: "INFO"
+       worker:
+         replicas: 4
+
+   clpConfig:
+     # Optional `compression-coordinator` configuration.
+     # The values below are the defaults and can be adjusted as needed.
+     compression_coordinator:
+       logging_level: "INFO"
+       # How often the coordinator checks for newly created compression jobs.
+       job_polling_interval_millisecs: 100
+       # Maximum number of times a failed compression task can be retried.
+       compression_task_max_retry: 1
+       # Maximum number of times a failed commit task can be retried.
+       commit_task_max_retry: 1
+
+     logs_input:
+       # In the current release, compression jobs coordinated by `compression-coordinator` require
+       # input logs to be stored in S3.
+       type: "s3"
+       aws_authentication:
+         type: "credentials"
+         credentials:
+           access_key_id: "<access-key-id>"
+           secret_access_key: "<secret-access-key>"
+
+     archive_output:
+       storage:
+         # In the current release, `compression-coordinator` supports only S3 archive output.
+         type: "s3"
+         s3_config:
+           region_code: "us-east-2"
+           bucket: "<archives-bucket>"
+           key_prefix: "archives/"
+           aws_authentication:
+             type: "credentials"
+             credentials:
+               access_key_id: "<access-key-id>"
+               secret_access_key: "<secret-access-key>"
+   ```
+
+2. Install (or upgrade) the Helm chart with the Spider values:
+
+   ```bash
+   helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml
+   ```
+
+3. Verify that `compression-coordinator` and Spider pods are running:
+
+   ```bash
+   kubectl get pods | grep -E "spider|compression-coordinator"
+   ```
+
+   The pods may take a few minutes to become ready. During startup, some Spider pods may restart
+   while the Spider database is being initialized. A small number of restarts during this period is
+   expected.
+
+   The output should include pods similar to the following:
+
+   ```text
+   NAME                                       READY   STATUS    RESTARTS   AGE
+   clp-compression-coordinator-...            1/1     Running   0          5m
+   clp-spider-database-0                      1/1     Running   0          5m
+   clp-spider-scheduler-...                   1/1     Running   2          5m
+   clp-spider-storage-...                     1/1     Running   2          5m
+   clp-spider-worker-...                      1/1     Running   2          5m
+   ```
+
+## Configure Spider
+
+Spider can be configured to meet the resource, storage, and authentication requirements of your
+workload. This section describes commonly adjusted settings for Spider in a CLP deployment, followed
+by options for more advanced performance tuning.
+
+Configure these settings in the CLP Helm values file.
+
+### Common settings
+
+The following settings are commonly adjusted when deploying Spider with CLP.
+
+:::{confval} spider.enabled
+:type: `bool`
+:default: `false`
+
+Enables Spider and deploys its related CLP components.
+:::
+
+:::{confval} spider.spiderConfig.worker.replicas
+:type: `int`
+:default: 4
+
+Sets the number of Spider worker pods to deploy. Increase this value to allow more tasks to run
+concurrently.
+:::
+
+:::{confval} spider.spiderConfig.worker.extra_volumes
+:type: `list`
+:default: Chart-defined defaults
+
+Adds volumes to the Spider worker pods.
+
+When overriding this list, retain the default entries that mount the CLP configuration and staging
+directory.
+:::
+
+:::{confval} spider.spiderConfig.worker.extra_volume_mounts
+:type: `list`
+:default: Chart-defined defaults
+
+Adds the volume mounts corresponding to {confval}`spider.spiderConfig.worker.extra_volumes` to the
+Spider worker pods.
+
+When overriding this list, retain the default entries that mount the CLP configuration and staging
+directory.
+:::
+
+:::{confval} spider.spiderConfig.worker.service_account_name
+:type: `string`
+:default: The CLP `ServiceAccount`
+
+Specifies the Kubernetes `ServiceAccount` assigned to the Spider worker pods.
+
+On Amazon EKS, configure this service account with the appropriate AWS permissions when Spider
+workers need to access AWS resources. See [this guide][aws-eks-service-accounts] for details.
+:::
+
+### Advanced tuning
+
+:::{warning}
+🚧 This section is still under construction.
+:::
+
+[aws-eks-service-accounts]: https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html
+[k8s-deployment]: guides-k8s-deployment.md
+[spider]: https://github.com/y-scope/spider