From 4e3fd4df8aaff63fb697221940eee128a658c2d1 Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Tue, 4 Aug 2026 15:07:27 -0400 Subject: [PATCH 01/14] docs(spider): Add "Using Spider with CLP" guide for Spider-orchestrated compression via the Helm chart. --- docs/src/user-docs/guides-k8s-deployment.md | 8 ++ docs/src/user-docs/guides-using-spider.md | 148 ++++++++++++++++++++ docs/src/user-docs/index.md | 8 ++ 3 files changed, 164 insertions(+) create mode 100644 docs/src/user-docs/guides-using-spider.md diff --git a/docs/src/user-docs/guides-k8s-deployment.md b/docs/src/user-docs/guides-k8s-deployment.md index 29b952db49..e2c2ab6ec4 100644 --- a/docs/src/user-docs/guides-k8s-deployment.md +++ b/docs/src/user-docs/guides-k8s-deployment.md @@ -287,6 +287,11 @@ helm template clp . -f custom-values.yaml To use Presto as the query engine, see the [Using Presto][presto-guide] guide for setup instructions, including the Helm values file and installation steps. +### Using Spider as the scheduler + +To use [Spider] as the compression scheduler, see the [Using Spider][spider-guide] guide for setup +instructions, including the Helm values file and installation steps. + ### Component scheduling You can control where any CLP component is scheduled using standard Kubernetes scheduling primitives @@ -686,6 +691,7 @@ To tear down a `kubeadm` cluster: * [Using object storage][s3-storage]: Configuring S3 storage * [Configuring retention periods][retention-guide]: Setting up data retention policies * [Using Presto][presto-guide]: Distributed SQL queries on compressed logs +* [Using Spider][spider-guide]: Spider-orchestrated compression [admin-tools]: reference-sbin-scripts/admin-tools.md [aks]: https://azure.microsoft.com/en-us/products/kubernetes-service @@ -711,5 +717,7 @@ To tear down a `kubeadm` cluster: [s3-storage]: guides-using-object-storage/index [SeaweedFS]: https://github.com/seaweedfs/seaweedfs [seaweedfs-setup]: guides-docker-compose-deployment.md#setting-up-seaweedfs +[Spider]: https://github.com/y-scope/spider +[spider-guide]: guides-using-spider.md [webui-clp-json]: quick-start/clp-json.md#searching-from-the-ui [webui-clp-text]: quick-start/clp-text.md#searching-from-the-ui diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md new file mode 100644 index 0000000000..73ffc1056f --- /dev/null +++ b/docs/src/user-docs/guides-using-spider.md @@ -0,0 +1,148 @@ +# Using Spider with CLP + +[Spider] is a distributed system for executing user-defined tasks. CLP uses it to orchestrate compression and search +jobs. This guide describes how to set up and use Spider with CLP. + +:::{warning} +Currently, only the [clp-json](quick-start/clp-json.md) flavor of CLP supports Spider-orchestrated +jobs, and only for logs ingested from S3 through [`log-ingestor`][log-ingestor-guide]. +::: + +:::{note} +The Spider integration currently covers compression only. Search jobs remain unaffected; they +continue to use Celery regardless of whether Spider is enabled. +::: + +## Deployment options + +Spider-orchestrated compression is currently only supported through the +[Kubernetes (Helm)][k8s-deployment] deployment method. + +## Kubernetes (Helm) + +When deploying CLP on Kubernetes using Helm, Spider can be enabled by setting `spider.enabled` to +`true`. When enabled, the chart additionally deploys: + +* The **compression coordinator**, a CLP component that polls for compression jobs and drives them + through Spider. +* The **Spider subchart**: Spider's storage service, scheduler, workers, and a bundled MariaDB for + Spider's job state. The workers run a CLP-specific image + (`ghcr.io/y-scope/clp/clp-spider-worker`) that bundles CLP's task library and core binaries, so + they can execute CLP compression tasks. + +Spider runs alongside the existing Celery-based compression pipeline; enabling it doesn't replace +the Celery components. + +### Requirements + +* A running CLP Kubernetes deployment (see the [Kubernetes deployment guide][k8s-deployment]) with + the following configuration: + * **S3 for both input and output**: `clpConfig.logs_input.type` and + `clpConfig.archive_output.storage.type` must both be `"s3"` (see the + [object storage guide][object-storage-guide]). The compression coordinator validates this at + startup and will fail to start otherwise. + * **clp-s storage engine**: `clpConfig.package.storage_engine` must be `"clp-s"` (the default). + +### Set up + +1. Create a values file to enable Spider: + + ```{code-block} yaml + :caption: spider-values.yaml + + spider: + enabled: true + + # Optional: Spider component settings (passed through to the Spider subchart), shown with + # their default values. Adjust them to suit your usage patterns. + spiderConfig: + storage: + log_level: "INFO" + port: 50051 + scheduler: + log_level: "INFO" + worker: + replicas: 4 + + clpConfig: + # Optional: compression coordinator settings, shown with their default values. Adjust them + # to suit your usage patterns. + compression_coordinator: + job_polling_interval_millisecs: 100 + logging_level: "INFO" + + logs_input: + # NOTE: Currently, only S3 input is supported. Support for ingestion from local + # filesystems is planned for a future release. + type: "s3" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + + archive_output: + storage: + # NOTE: Currently, only S3 storage is supported. Support for filesystem compression is + # planned for a future release. + type: "s3" + s3_config: + region_code: "us-east-2" + bucket: "" + key_prefix: "archives/" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + ``` + +2. Install (or upgrade) the Helm chart with the Spider values: + + ```bash + helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml + ``` + +3. Verify that the compression coordinator and Spider pods are running. Pods may take a few minutes + to become ready, and Spider pods restarting a few times while Spider's database comes up is + normal: + + ```bash + kubectl get pods | grep -E "spider|compression" + ``` + + ```text + NAME READY STATUS RESTARTS AGE + clp-compression-coordinator-... 1/1 Running 0 5m + clp-compression-scheduler-... 1/1 Running 0 5m + clp-compression-worker-... 1/1 Running 0 5m + clp-spider-database-0 1/1 Running 0 5m + clp-spider-scheduler-... 1/1 Running 2 5m + clp-spider-storage-... 1/1 Running 2 5m + clp-spider-worker-... 1/1 Running 2 5m + ``` + +Once the pods are ready, compression jobs submitted through [`log-ingestor`][log-ingestor-guide] +will be orchestrated by Spider. + +### Configuration + +The values below are the most commonly adjusted settings. Keys under `spider.spiderConfig` that +are specific to the Kubernetes deployment are listed individually; all other keys map directly to +Spider's own configuration. + +| Value | Default | Description | +|---|---|---| +| `spider.enabled` | `false` | Deploys Spider and the compression coordinator. | +| `spider.spiderConfig.worker.replicas` | `4` | Number of Spider worker pods. | +| `spider.spiderConfig.worker.extra_envs` | See `values.yaml` | Additional environment variables for the Spider worker pods. Overriding this **replaces** the chart's default list (CLP's config path and database credentials), so restate the defaults when adding entries. | +| `spider.spiderConfig.worker.extra_volumes` / `.extra_volume_mounts` | See `values.yaml` | Additional volumes and mounts for the Spider worker pods. The chart's defaults mount CLP's config and a staging directory; restate them when adding entries. | +| `spider.spiderConfig.worker.service_account_name` | CLP's ServiceAccount | The ServiceAccount that the Spider worker pods run as. | +| All other keys under `spider.spiderConfig` | See the Spider user guide (**null**) | Spider's own configuration; each key maps to the corresponding setting of the matching Spider component — e.g., `spiderConfig.storage.log_level` maps to the `log_level` setting under Spider's storage component. | + + + +[k8s-deployment]: guides-k8s-deployment.md +[log-ingestor-guide]: guides-using-log-ingestor.md +[object-storage-guide]: guides-using-object-storage/index.md +[Spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index f10656eafe..ba0ff33c3b 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -69,6 +69,13 @@ Using `log-ingestor` Use `log-ingestor` to continuously ingest logs. ::: +:::{grid-item-card} +:link: guides-using-spider +Using Spider +^^^ +Use Spider to orchestrate compression jobs on Kubernetes. +::: + :::{grid-item-card} :link: guides-external-database External database setup @@ -246,6 +253,7 @@ guides-k8s-deployment guides-using-object-storage/index guides-using-log-ingestor +guides-using-spider guides-external-database guides-retention ::: From a4139dd13e2b491ccde427d7dd31488cc929ab62 Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Wed, 5 Aug 2026 12:34:21 -0400 Subject: [PATCH 02/14] docs(spider): Link the compression-coordinator guide from the Spider guide. --- docs/src/user-docs/guides-using-spider.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md index 73ffc1056f..d1ce14e326 100644 --- a/docs/src/user-docs/guides-using-spider.md +++ b/docs/src/user-docs/guides-using-spider.md @@ -23,8 +23,8 @@ Spider-orchestrated compression is currently only supported through the When deploying CLP on Kubernetes using Helm, Spider can be enabled by setting `spider.enabled` to `true`. When enabled, the chart additionally deploys: -* The **compression coordinator**, a CLP component that polls for compression jobs and drives them - through Spider. +* The [**compression coordinator**][compression-coordinator-guide], a CLP component that polls for + compression jobs and drives them through Spider. * The **Spider subchart**: Spider's storage service, scheduler, workers, and a bundled MariaDB for Spider's job state. The workers run a CLP-specific image (`ghcr.io/y-scope/clp/clp-spider-worker`) that bundles CLP's task library and core binaries, so @@ -142,6 +142,7 @@ Spider's own configuration. +[compression-coordinator-guide]: guides-using-compression-coordinator.md [k8s-deployment]: guides-k8s-deployment.md [log-ingestor-guide]: guides-using-log-ingestor.md [object-storage-guide]: guides-using-object-storage/index.md From 5809bc41b1e6c523bca0df5f33095096a2eec54f Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Wed, 5 Aug 2026 15:11:45 -0400 Subject: [PATCH 03/14] docs(spider): Reference the compression-coordinator guide from the coordinator config comment. --- docs/src/user-docs/guides-using-spider.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md index d1ce14e326..cc8a032aa0 100644 --- a/docs/src/user-docs/guides-using-spider.md +++ b/docs/src/user-docs/guides-using-spider.md @@ -66,7 +66,8 @@ the Celery components. clpConfig: # Optional: compression coordinator settings, shown with their default values. Adjust them - # to suit your usage patterns. + # to suit your usage patterns. For details about the coordinator, see the "Using + # compression-coordinator" guide. compression_coordinator: job_polling_interval_millisecs: 100 logging_level: "INFO" From cfb5edcbc2d7e22f4b8ad68777c4db5c6269c31a Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Wed, 5 Aug 2026 15:15:19 -0400 Subject: [PATCH 04/14] docs(spider): Add a section for compressing logs from S3 through Spider via log-ingestor. --- docs/src/user-docs/guides-using-spider.md | 46 ++++++++++++++++++++++- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md index cc8a032aa0..791c35d5db 100644 --- a/docs/src/user-docs/guides-using-spider.md +++ b/docs/src/user-docs/guides-using-spider.md @@ -123,8 +123,8 @@ the Celery components. clp-spider-worker-... 1/1 Running 2 5m ``` -Once the pods are ready, compression jobs submitted through [`log-ingestor`][log-ingestor-guide] -will be orchestrated by Spider. +Once the pods are ready, you can +[compress your logs from S3 through Spider](#compressing-your-logs-from-s3-through-spider). ### Configuration @@ -143,6 +143,48 @@ Spider's own configuration. +## Compressing your logs from S3 through Spider + +With Spider enabled, compression jobs are submitted through [`log-ingestor`][log-ingestor-guide], +which continuously scans an S3 bucket and groups the discovered objects into compression jobs. + +1. Forward the `log-ingestor` Service and submit an ingestion job (see the + [`log-ingestor` guide][log-ingestor-guide] for the full API): + + ```bash + kubectl port-forward service/clp-log-ingestor 3002:3002 + ``` + + ```bash + curl -X POST http://localhost:3002/s3_scanner \ + -H "Content-Type: application/json" \ + -d '{ + "bucket_name": "", + "key_prefix": "", + "scanning_interval_sec": 5, + "buffer_config": {"timeout_sec": 10} + }' + ``` + +2. Verify that the job was executed through Spider by checking the `spider_id` column of the + `compression_jobs` table in CLP's database: + + ```bash + kubectl exec clp-database-0 -- mariadb --table -u clp-user -p"" clp-db \ + -e "SELECT id, status, spider_id FROM compression_jobs;" + ``` + + ```text + +----+--------+-----------+ + | id | status | spider_id | + +----+--------+-----------+ + | 1 | 2 | 1 | + +----+--------+-----------+ + ``` + + A status of `2` means the job succeeded, and a non-`NULL` `spider_id` means the job was + executed through Spider (`NULL` means it ran through the Celery pipeline). + [compression-coordinator-guide]: guides-using-compression-coordinator.md [k8s-deployment]: guides-k8s-deployment.md [log-ingestor-guide]: guides-using-log-ingestor.md From fbb81b2461b3b2d057c3fc2f4c2206c0f94f046c Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Wed, 5 Aug 2026 15:17:30 -0400 Subject: [PATCH 05/14] docs(spider): Add a TODO to refine the exposed configuration options. --- docs/src/user-docs/guides-using-spider.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md index 791c35d5db..959477f0b6 100644 --- a/docs/src/user-docs/guides-using-spider.md +++ b/docs/src/user-docs/guides-using-spider.md @@ -128,6 +128,9 @@ Once the pods are ready, you can ### Configuration +TODO: This section needs further refinement — decide which of these options must be exposed to +users and which should stay internal. + The values below are the most commonly adjusted settings. Keys under `spider.spiderConfig` that are specific to the Kubernetes deployment are listed individually; all other keys map directly to Spider's own configuration. From fdce5d3af5d1bcfe36f4e42b91df77ff0c14d8e2 Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Thu, 6 Aug 2026 17:19:51 -0400 Subject: [PATCH 06/14] Rewrite. --- .../guides-using-spider-scheduler.md | 198 ++++++++++++++++++ docs/src/user-docs/guides-using-spider.md | 1 - docs/src/user-docs/index.md | 25 +++ 3 files changed, 223 insertions(+), 1 deletion(-) create mode 100644 docs/src/user-docs/guides-using-spider-scheduler.md diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md new file mode 100644 index 0000000000..b7198f5a98 --- /dev/null +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -0,0 +1,198 @@ +# Using Spider scheduler + +[Spider][spider] is a distributed task scheduling and execution framework. It schedules tasks across +configured workers and manages their execution lifecycle. + +CLP uses Spider as its next-generation framework for orchestrating compression and search jobs, +providing the following benefits: + +* **Improved fault tolerance** +* **Minimized scheduling overhead** +* **Maximized utilization of compute resources** + +:::{note} +In the current release, CLP uses Spider only for compression jobs coordinated by +`compression-coordinator`. Support for orchestrating search jobs with Spider is under development +and will be introduced in a future release. + +Spider is planned to become the default job execution framework for CLP, eventually replacing the +existing Celery-based job orchestration components. +::: + +## Deployment + +When Spider is enabled, compatible CLP coordinators are automatically deployed and submit supported +jobs to Spider for scheduling and execution. Currently, Spider and its related CLP components can be +enabled only in [Kubernetes deployments][k8s-deployment] using the CLP Helm chart. + +:::{note} +Spider is not supported in Docker Compose deployments in the current release. Support for Docker +Compose is planned for a future release. +::: + +### Kubernetes (Helm) + +When deploying CLP on Kubernetes using Helm, enable Spider by setting `spider.enabled` to `true`. +The CLP Helm chart will then automatically deploy the following components: + +* **Spider subchart**: Deploys all the Spider services required to schedule and execute jobs. +* **compression-coordinator**: Coordinates supported CLP compression jobs and submits them to + Spider for execution. + +:::{note} +In the current release, `compression-coordinator` supports only compression jobs that use certain +CLP package configurations. See "Using compression-coordinator" for the current limitations. +::: + +#### Set up + +1. Create a values file to enable Spider and its related CLP components. + + ```{code-block} yaml + :caption: spider-values.yaml + + spider: + enabled: true + + # Optional Spider configuration passed directly to the Spider subchart. + # The values below are the defaults and can be adjusted as needed. + spiderConfig: + storage: + log_level: "INFO" + port: 50051 + scheduler: + log_level: "INFO" + worker: + replicas: 4 + + clpConfig: + # Optional `compression-coordinator` configuration. + # The values below are the defaults and can be adjusted as needed. + compression_coordinator: + logging_level: "INFO" + # How often the coordinator checks for newly created compression jobs. + job_polling_interval_millisecs: 100 + # Maximum number of times a failed compression task can be retried. + compression_task_max_retry: 1 + # Maximum number of times a failed commit task can be retried. + commit_task_max_retry: 1 + + logs_input: + # In the current release, compression jobs coordinated by `compression-coordinator` require + # input logs to be stored in S3. + type: "s3" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + + archive_output: + storage: + # In the current release, `compression-coordinator` supports only S3 archive output. + type: "s3" + s3_config: + region_code: "us-east-2" + bucket: "" + key_prefix: "archives/" + aws_authentication: + type: "credentials" + credentials: + access_key_id: "" + secret_access_key: "" + ``` + +2. Install (or upgrade) the Helm chart with the Spider values: + + ```bash + helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml + ``` + +3. Verify that `compression-coordinator` and Spider pods are running: + + ```bash + kubectl get pods | grep -E "spider|compression-coordinator" + ``` + + The pods may take a few minutes to become ready. During startup, some Spider pods may restart + while the Spider database is being initialized. A small number of restarts during this period is + expected. + + The output should include pods similar to the following: + + ```text + NAME READY STATUS RESTARTS AGE + clp-compression-coordinator-... 1/1 Running 0 5m + clp-spider-database-0 1/1 Running 0 5m + clp-spider-scheduler-... 1/1 Running 2 5m + clp-spider-storage-... 1/1 Running 2 5m + clp-spider-worker-... 1/1 Running 2 5m + ``` + +## Configure Spider + +Spider can be configured to meet the resource, storage, and authentication requirements of your +workload. This section describes commonly adjusted settings for Spider in a CLP deployment, followed +by options for more advanced performance tuning. + +Configure these settings in the CLP Helm values file. + +### Common settings + +The following settings are commonly adjusted when deploying Spider with CLP. + +:::{confval} spider.enabled +:type: `bool` +:default: `false` + +Enables Spider and deploys its related CLP components. +::: + +:::{confval} spider.spiderConfig.worker.replicas +:type: `int` +:default: 4 + +Sets the number of Spider worker pods to deploy. Increase this value to allow more tasks to run +concurrently, provided that the Kubernetes cluster has sufficient compute resources. +::: + +:::{confval} spider.spiderConfig.worker.extra_volumes +:type: `list` +:default: Chart-defined defaults + +Adds volumes to the Spider worker pods. + +When overriding this list, retain the default entries that mount the CLP configuration and staging +directory. +::: + +:::{confval} spider.spiderConfig.worker.extra_volume_mounts +:type: `list` +:default: Chart-defined defaults + +Adds the volume mounts corresponding to {confval}`spider.spiderConfig.worker.extra_volumes` to the +Spider worker pods. + +When overriding this list, retain the default entries that mount the CLP configuration and staging +directory. +::: + +:::{confval} spider.spiderConfig.worker.service_account_name +:type: `string` +:default: The CLP `ServiceAccount` + +Specifies the Kubernetes `ServiceAccount` assigned to the Spider worker pods. + +On Amazon EKS, configure this service account with the appropriate AWS permissions when Spider +workers need to access AWS resources. See [this guide][aws-eks-service-accounts] for details. +::: + +### Advanced tuning + +:::{warning} +🚧 This section is still under construction. +::: + +[aws-eks-service-accounts]: https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html +[k8s-deployment]: guides-k8s-deployment.md +[spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md index 959477f0b6..d370880d44 100644 --- a/docs/src/user-docs/guides-using-spider.md +++ b/docs/src/user-docs/guides-using-spider.md @@ -188,7 +188,6 @@ which continuously scans an S3 bucket and groups the discovered objects into com A status of `2` means the job succeeded, and a non-`NULL` `spider_id` means the job was executed through Spider (`NULL` means it ran through the Celery pipeline). -[compression-coordinator-guide]: guides-using-compression-coordinator.md [k8s-deployment]: guides-k8s-deployment.md [log-ingestor-guide]: guides-using-log-ingestor.md [object-storage-guide]: guides-using-object-storage/index.md diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index ba0ff33c3b..e2c500f203 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -125,6 +125,24 @@ Use Presto for distributed SQL queries on compressed logs. --- +## Scheduling & performance + +Configure how CLP schedules jobs and uses compute resources. + +::::{grid} 1 1 2 2 +:gutter: 2 + +:::{grid-item-card} +:link: guides-using-spider-scheduler +Using Spider scheduler +^^^ +Use Spider to schedule and execute supported CLP jobs across distributed workers. +::: + +:::: + +--- + ## Core Try CLP's core binaries directly, without deploying the full CLP package. @@ -267,6 +285,13 @@ guides-mcp-server/index guides-using-presto ::: +:::{toctree} +:hidden: +:caption: Scheduling & performance + +guides-using-spider-scheduler +::: + :::{toctree} :hidden: :caption: Core From 48ee98674ddf0c85b3206acfc7ac69c29e11f0cb Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Thu, 6 Aug 2026 17:37:24 -0400 Subject: [PATCH 07/14] In progress. --- docs/src/user-docs/guides-using-spider-scheduler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index b7198f5a98..d1a11459da 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -153,7 +153,7 @@ Enables Spider and deploys its related CLP components. :default: 4 Sets the number of Spider worker pods to deploy. Increase this value to allow more tasks to run -concurrently, provided that the Kubernetes cluster has sufficient compute resources. +concurrently. ::: :::{confval} spider.spiderConfig.worker.extra_volumes From 79286efff67a1bdb341a5f2a7886c7b214c1c85b Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Thu, 6 Aug 2026 22:15:22 -0400 Subject: [PATCH 08/14] docs(spider): Remove the superseded Spider guide in favor of the Spider scheduler guide. --- docs/src/user-docs/guides-k8s-deployment.md | 8 +- docs/src/user-docs/guides-using-spider.md | 194 -------------------- docs/src/user-docs/index.md | 8 - 3 files changed, 4 insertions(+), 206 deletions(-) delete mode 100644 docs/src/user-docs/guides-using-spider.md diff --git a/docs/src/user-docs/guides-k8s-deployment.md b/docs/src/user-docs/guides-k8s-deployment.md index e2c2ab6ec4..90555894f0 100644 --- a/docs/src/user-docs/guides-k8s-deployment.md +++ b/docs/src/user-docs/guides-k8s-deployment.md @@ -289,8 +289,8 @@ instructions, including the Helm values file and installation steps. ### Using Spider as the scheduler -To use [Spider] as the compression scheduler, see the [Using Spider][spider-guide] guide for setup -instructions, including the Helm values file and installation steps. +To use [Spider] as the compression scheduler, see the [Using Spider scheduler][spider-guide] guide +for setup instructions, including the Helm values file and installation steps. ### Component scheduling @@ -691,7 +691,7 @@ To tear down a `kubeadm` cluster: * [Using object storage][s3-storage]: Configuring S3 storage * [Configuring retention periods][retention-guide]: Setting up data retention policies * [Using Presto][presto-guide]: Distributed SQL queries on compressed logs -* [Using Spider][spider-guide]: Spider-orchestrated compression +* [Using Spider scheduler][spider-guide]: Spider-orchestrated compression [admin-tools]: reference-sbin-scripts/admin-tools.md [aks]: https://azure.microsoft.com/en-us/products/kubernetes-service @@ -718,6 +718,6 @@ To tear down a `kubeadm` cluster: [SeaweedFS]: https://github.com/seaweedfs/seaweedfs [seaweedfs-setup]: guides-docker-compose-deployment.md#setting-up-seaweedfs [Spider]: https://github.com/y-scope/spider -[spider-guide]: guides-using-spider.md +[spider-guide]: guides-using-spider-scheduler.md [webui-clp-json]: quick-start/clp-json.md#searching-from-the-ui [webui-clp-text]: quick-start/clp-text.md#searching-from-the-ui diff --git a/docs/src/user-docs/guides-using-spider.md b/docs/src/user-docs/guides-using-spider.md deleted file mode 100644 index d370880d44..0000000000 --- a/docs/src/user-docs/guides-using-spider.md +++ /dev/null @@ -1,194 +0,0 @@ -# Using Spider with CLP - -[Spider] is a distributed system for executing user-defined tasks. CLP uses it to orchestrate compression and search -jobs. This guide describes how to set up and use Spider with CLP. - -:::{warning} -Currently, only the [clp-json](quick-start/clp-json.md) flavor of CLP supports Spider-orchestrated -jobs, and only for logs ingested from S3 through [`log-ingestor`][log-ingestor-guide]. -::: - -:::{note} -The Spider integration currently covers compression only. Search jobs remain unaffected; they -continue to use Celery regardless of whether Spider is enabled. -::: - -## Deployment options - -Spider-orchestrated compression is currently only supported through the -[Kubernetes (Helm)][k8s-deployment] deployment method. - -## Kubernetes (Helm) - -When deploying CLP on Kubernetes using Helm, Spider can be enabled by setting `spider.enabled` to -`true`. When enabled, the chart additionally deploys: - -* The [**compression coordinator**][compression-coordinator-guide], a CLP component that polls for - compression jobs and drives them through Spider. -* The **Spider subchart**: Spider's storage service, scheduler, workers, and a bundled MariaDB for - Spider's job state. The workers run a CLP-specific image - (`ghcr.io/y-scope/clp/clp-spider-worker`) that bundles CLP's task library and core binaries, so - they can execute CLP compression tasks. - -Spider runs alongside the existing Celery-based compression pipeline; enabling it doesn't replace -the Celery components. - -### Requirements - -* A running CLP Kubernetes deployment (see the [Kubernetes deployment guide][k8s-deployment]) with - the following configuration: - * **S3 for both input and output**: `clpConfig.logs_input.type` and - `clpConfig.archive_output.storage.type` must both be `"s3"` (see the - [object storage guide][object-storage-guide]). The compression coordinator validates this at - startup and will fail to start otherwise. - * **clp-s storage engine**: `clpConfig.package.storage_engine` must be `"clp-s"` (the default). - -### Set up - -1. Create a values file to enable Spider: - - ```{code-block} yaml - :caption: spider-values.yaml - - spider: - enabled: true - - # Optional: Spider component settings (passed through to the Spider subchart), shown with - # their default values. Adjust them to suit your usage patterns. - spiderConfig: - storage: - log_level: "INFO" - port: 50051 - scheduler: - log_level: "INFO" - worker: - replicas: 4 - - clpConfig: - # Optional: compression coordinator settings, shown with their default values. Adjust them - # to suit your usage patterns. For details about the coordinator, see the "Using - # compression-coordinator" guide. - compression_coordinator: - job_polling_interval_millisecs: 100 - logging_level: "INFO" - - logs_input: - # NOTE: Currently, only S3 input is supported. Support for ingestion from local - # filesystems is planned for a future release. - type: "s3" - aws_authentication: - type: "credentials" - credentials: - access_key_id: "" - secret_access_key: "" - - archive_output: - storage: - # NOTE: Currently, only S3 storage is supported. Support for filesystem compression is - # planned for a future release. - type: "s3" - s3_config: - region_code: "us-east-2" - bucket: "" - key_prefix: "archives/" - aws_authentication: - type: "credentials" - credentials: - access_key_id: "" - secret_access_key: "" - ``` - -2. Install (or upgrade) the Helm chart with the Spider values: - - ```bash - helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f spider-values.yaml - ``` - -3. Verify that the compression coordinator and Spider pods are running. Pods may take a few minutes - to become ready, and Spider pods restarting a few times while Spider's database comes up is - normal: - - ```bash - kubectl get pods | grep -E "spider|compression" - ``` - - ```text - NAME READY STATUS RESTARTS AGE - clp-compression-coordinator-... 1/1 Running 0 5m - clp-compression-scheduler-... 1/1 Running 0 5m - clp-compression-worker-... 1/1 Running 0 5m - clp-spider-database-0 1/1 Running 0 5m - clp-spider-scheduler-... 1/1 Running 2 5m - clp-spider-storage-... 1/1 Running 2 5m - clp-spider-worker-... 1/1 Running 2 5m - ``` - -Once the pods are ready, you can -[compress your logs from S3 through Spider](#compressing-your-logs-from-s3-through-spider). - -### Configuration - -TODO: This section needs further refinement — decide which of these options must be exposed to -users and which should stay internal. - -The values below are the most commonly adjusted settings. Keys under `spider.spiderConfig` that -are specific to the Kubernetes deployment are listed individually; all other keys map directly to -Spider's own configuration. - -| Value | Default | Description | -|---|---|---| -| `spider.enabled` | `false` | Deploys Spider and the compression coordinator. | -| `spider.spiderConfig.worker.replicas` | `4` | Number of Spider worker pods. | -| `spider.spiderConfig.worker.extra_envs` | See `values.yaml` | Additional environment variables for the Spider worker pods. Overriding this **replaces** the chart's default list (CLP's config path and database credentials), so restate the defaults when adding entries. | -| `spider.spiderConfig.worker.extra_volumes` / `.extra_volume_mounts` | See `values.yaml` | Additional volumes and mounts for the Spider worker pods. The chart's defaults mount CLP's config and a staging directory; restate them when adding entries. | -| `spider.spiderConfig.worker.service_account_name` | CLP's ServiceAccount | The ServiceAccount that the Spider worker pods run as. | -| All other keys under `spider.spiderConfig` | See the Spider user guide (**null**) | Spider's own configuration; each key maps to the corresponding setting of the matching Spider component — e.g., `spiderConfig.storage.log_level` maps to the `log_level` setting under Spider's storage component. | - - - -## Compressing your logs from S3 through Spider - -With Spider enabled, compression jobs are submitted through [`log-ingestor`][log-ingestor-guide], -which continuously scans an S3 bucket and groups the discovered objects into compression jobs. - -1. Forward the `log-ingestor` Service and submit an ingestion job (see the - [`log-ingestor` guide][log-ingestor-guide] for the full API): - - ```bash - kubectl port-forward service/clp-log-ingestor 3002:3002 - ``` - - ```bash - curl -X POST http://localhost:3002/s3_scanner \ - -H "Content-Type: application/json" \ - -d '{ - "bucket_name": "", - "key_prefix": "", - "scanning_interval_sec": 5, - "buffer_config": {"timeout_sec": 10} - }' - ``` - -2. Verify that the job was executed through Spider by checking the `spider_id` column of the - `compression_jobs` table in CLP's database: - - ```bash - kubectl exec clp-database-0 -- mariadb --table -u clp-user -p"" clp-db \ - -e "SELECT id, status, spider_id FROM compression_jobs;" - ``` - - ```text - +----+--------+-----------+ - | id | status | spider_id | - +----+--------+-----------+ - | 1 | 2 | 1 | - +----+--------+-----------+ - ``` - - A status of `2` means the job succeeded, and a non-`NULL` `spider_id` means the job was - executed through Spider (`NULL` means it ran through the Celery pipeline). - -[k8s-deployment]: guides-k8s-deployment.md -[log-ingestor-guide]: guides-using-log-ingestor.md -[object-storage-guide]: guides-using-object-storage/index.md -[Spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index e2c500f203..aeace35656 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -69,13 +69,6 @@ Using `log-ingestor` Use `log-ingestor` to continuously ingest logs. ::: -:::{grid-item-card} -:link: guides-using-spider -Using Spider -^^^ -Use Spider to orchestrate compression jobs on Kubernetes. -::: - :::{grid-item-card} :link: guides-external-database External database setup @@ -271,7 +264,6 @@ guides-k8s-deployment guides-using-object-storage/index guides-using-log-ingestor -guides-using-spider guides-external-database guides-retention ::: From ecab327a12aaa110a776d8c2dbdac4b35ffb4666 Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Thu, 6 Aug 2026 22:17:46 -0400 Subject: [PATCH 09/14] docs(spider): Remove the extra_volumes and extra_volume_mounts settings from the Spider scheduler guide. --- .../guides-using-spider-scheduler.md | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index d1a11459da..71fe82936e 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -156,27 +156,6 @@ Sets the number of Spider worker pods to deploy. Increase this value to allow mo concurrently. ::: -:::{confval} spider.spiderConfig.worker.extra_volumes -:type: `list` -:default: Chart-defined defaults - -Adds volumes to the Spider worker pods. - -When overriding this list, retain the default entries that mount the CLP configuration and staging -directory. -::: - -:::{confval} spider.spiderConfig.worker.extra_volume_mounts -:type: `list` -:default: Chart-defined defaults - -Adds the volume mounts corresponding to {confval}`spider.spiderConfig.worker.extra_volumes` to the -Spider worker pods. - -When overriding this list, retain the default entries that mount the CLP configuration and staging -directory. -::: - :::{confval} spider.spiderConfig.worker.service_account_name :type: `string` :default: The CLP `ServiceAccount` From 395ef2984ed6610e4ac4538a45559faaf55733a6 Mon Sep 17 00:00:00 2001 From: ChenXing Yang <20001020ycx@gmail.com> Date: Thu, 6 Aug 2026 22:23:51 -0400 Subject: [PATCH 10/14] docs(spider): Remove the service_account_name setting from the Spider scheduler guide's common settings. --- docs/src/user-docs/guides-using-spider-scheduler.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index 71fe82936e..029af22a61 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -156,22 +156,11 @@ Sets the number of Spider worker pods to deploy. Increase this value to allow mo concurrently. ::: -:::{confval} spider.spiderConfig.worker.service_account_name -:type: `string` -:default: The CLP `ServiceAccount` - -Specifies the Kubernetes `ServiceAccount` assigned to the Spider worker pods. - -On Amazon EKS, configure this service account with the appropriate AWS permissions when Spider -workers need to access AWS resources. See [this guide][aws-eks-service-accounts] for details. -::: - ### Advanced tuning :::{warning} 🚧 This section is still under construction. ::: -[aws-eks-service-accounts]: https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html [k8s-deployment]: guides-k8s-deployment.md [spider]: https://github.com/y-scope/spider From f97535c1d2879518017def92c5f20aee9a33483c Mon Sep 17 00:00:00 2001 From: ChenXing Yang <60459812+20001020ycx@users.noreply.github.com> Date: Thu, 6 Aug 2026 22:27:08 -0400 Subject: [PATCH 11/14] Apply suggestion from @20001020ycx --- docs/src/user-docs/guides-using-spider-scheduler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index 029af22a61..c78a5bf866 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -118,7 +118,7 @@ CLP package configurations. See "Using compression-coordinator" for the current while the Spider database is being initialized. A small number of restarts during this period is expected. - The output should include pods similar to the following: + The output should eventually include entries similar to the following: ```text NAME READY STATUS RESTARTS AGE From 6e8016f31858d0676581d1c58391f1f07b25f3c5 Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Fri, 7 Aug 2026 16:18:06 -0400 Subject: [PATCH 12/14] Add section. --- .../guides-using-spider-scheduler.md | 53 ++++++++++++++++++- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index c78a5bf866..793c10ded1 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -158,8 +158,57 @@ concurrently. ### Advanced tuning -:::{warning} -🚧 This section is still under construction. +The following settings provide additional control over Spider's runtime behavior and performance. +Adjust them as needed to suit your workload and deployment environment. + +:::{confval} spider.spiderConfig.scheduler.config.active_job_queue_size +:type: `int` +:default: 8 + +Sets the maximum number of jobs that can actively make progress in Spider at the same time. Jobs +beyond this limit remain queued until an active slot becomes available. Currently, queued jobs are +admitted in first-in, first-out (FIFO) order. + +Active jobs share the available Spider workers in a round-robin fashion. As a result, this setting +controls the trade-off between job-level concurrency and the amount of compute capacity available +to each active job. + +In the current release, Spider executes only compression jobs, so this setting effectively limits +the number of compression jobs that can make progress concurrently. + +Consider increasing this value when you want more jobs to make progress concurrently and sufficient +worker capacity is available. Decrease it when you prefer to concentrate compute resources on fewer +jobs at a time. +::: + +:::{confval} spider.spiderConfig.scheduler.config.dispatch_queue_capacity +:type: `int` +:default: 16 + +Sets the capacity of the scheduler's task dispatch queue. This queue buffers tasks that are ready to +be dispatched to Spider workers. + +As a general guideline, configure this value to approximately two to four times the number of +Spider workers. If you increase the number of workers, consider increasing this value as well so +that the scheduler can keep enough tasks available for dispatch and avoid underutilizing workers. +::: + +:::{confval} spider.spiderConfig.scheduler.config.ready_task_capacity +:type: `int` +:default: 1048576 + +Sets the maximum number of ready tasks that the Spider scheduler can buffer for scheduling. + +A larger capacity allows the scheduler to consider more ready tasks at once, which can improve +fairness across active jobs and better accommodate workloads with high task submission rates. +However, buffering more tasks also increases the scheduler's memory usage. + +Consider: + +* Increasing this value for workloads that produce large numbers of ready tasks or have high task + submission rates. +* Decreasing this value if scheduler memory usage is a concern and the workload does not require a + large ready-task buffer. ::: [k8s-deployment]: guides-k8s-deployment.md From 24f40c0d73aa939e7d17c8409b8d9050c02b80a8 Mon Sep 17 00:00:00 2001 From: ChenXing Yang <60459812+20001020ycx@users.noreply.github.com> Date: Fri, 7 Aug 2026 17:30:31 -0400 Subject: [PATCH 13/14] Apply suggestion from @20001020ycx --- docs/src/user-docs/guides-k8s-deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/user-docs/guides-k8s-deployment.md b/docs/src/user-docs/guides-k8s-deployment.md index 90555894f0..6396a31d42 100644 --- a/docs/src/user-docs/guides-k8s-deployment.md +++ b/docs/src/user-docs/guides-k8s-deployment.md @@ -289,7 +289,7 @@ instructions, including the Helm values file and installation steps. ### Using Spider as the scheduler -To use [Spider] as the compression scheduler, see the [Using Spider scheduler][spider-guide] guide +To use [Spider] as the scheduler for compression and search jobs, see the [Using Spider scheduler][spider-guide] guide for setup instructions, including the Helm values file and installation steps. ### Component scheduling From a392dacf669a840bf79380398abb700ee471b174 Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Sun, 23 Aug 2026 20:56:14 -0400 Subject: [PATCH 14/14] Polish before approve. --- .../guides-using-spider-scheduler.md | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/src/user-docs/guides-using-spider-scheduler.md b/docs/src/user-docs/guides-using-spider-scheduler.md index 793c10ded1..cdfcc6408c 100644 --- a/docs/src/user-docs/guides-using-spider-scheduler.md +++ b/docs/src/user-docs/guides-using-spider-scheduler.md @@ -12,8 +12,8 @@ providing the following benefits: :::{note} In the current release, CLP uses Spider only for compression jobs coordinated by -`compression-coordinator`. Support for orchestrating search jobs with Spider is under development -and will be introduced in a future release. +`compression-coordinator`. Support for orchestrating search jobs with Spider is planned for a future +release. Spider is planned to become the default job execution framework for CLP, eventually replacing the existing Celery-based job orchestration components. @@ -36,13 +36,13 @@ When deploying CLP on Kubernetes using Helm, enable Spider by setting `spider.en The CLP Helm chart will then automatically deploy the following components: * **Spider subchart**: Deploys all the Spider services required to schedule and execute jobs. -* **compression-coordinator**: Coordinates supported CLP compression jobs and submits them to - Spider for execution. +* **compression-coordinator**: Coordinates CLP compression jobs and submits them to Spider for + execution. -:::{note} -In the current release, `compression-coordinator` supports only compression jobs that use certain -CLP package configurations. See "Using compression-coordinator" for the current limitations. -::: + :::{note} + In the current release, `compression-coordinator` supports only compression jobs that use certain + CLP package configurations. See "Using compression-coordinator" for the current limitations. + ::: #### Set up @@ -76,7 +76,9 @@ CLP package configurations. See "Using compression-coordinator" for the current compression_task_max_retry: 1 # Maximum number of times a failed commit task can be retried. commit_task_max_retry: 1 - + # Maximum number of compression jobs that can be submitted to Spider concurrently. + max_concurrent_jobs: 1000 + logs_input: # In the current release, compression jobs coordinated by `compression-coordinator` require # input logs to be stored in S3. @@ -200,12 +202,12 @@ that the scheduler can keep enough tasks available for dispatch and avoid underu Sets the maximum number of ready tasks that the Spider scheduler can buffer for scheduling. A larger capacity allows the scheduler to consider more ready tasks at once, which can improve -fairness across active jobs and better accommodate workloads with high task submission rates. +fairness across active jobs and better accommodate workloads with high job submission rates. However, buffering more tasks also increases the scheduler's memory usage. Consider: -* Increasing this value for workloads that produce large numbers of ready tasks or have high task +* Increasing this value for workloads that produce large numbers of ready tasks or have high job submission rates. * Decreasing this value if scheduler memory usage is a concern and the workload does not require a large ready-task buffer.